How to pass the Outlier assessment.

What the Outlier assessment actually measures

Outlier's screening isn't a knowledge test. It measures whether you can read a rubric under ambiguity, apply it consistently, and make defensible judgment calls. Three signals reliably predict passing:

• 01
  Rubric adherence under ambiguity. Real screening tasks include deliberately underspecified rubrics. Passers ask the right clarifying question or make their assumption explicit. Failers guess confidently and move on.
• 02
  Inter-rater agreement (implicit). Your ratings are compared to a golden set. High agreement on the "easy" items is table stakes; the differentiator is agreement on the ambiguous items where reasonable graders disagree.
• 03
  Consistency across the sample. Grading item 3 the same way you'd grade item 30 (given the same rubric) matters more than being globally "right" on a single item.

Task-type specific strategies

Coding Evaluation

You're evaluating two code responses to the same prompt. Common failure modes: (1) picking the more elegant code when the rubric asks about correctness, (2) missing edge cases the rubric implicitly requires, (3) inconsistent treatment of style vs. functionality. Tip: read the entire rubric before looking at either code sample. Force yourself to name the criterion the sample is being graded on before deciding.

Math Reasoning

You're verifying whether a model's proof or calculation is correct. Common failure modes: (1) accepting a plausible-looking wrong answer because the algebra "looks right," (2) rejecting a correct but unusual approach. Tip: reconstruct the problem yourself before evaluating the response. If your reconstruction doesn't match, the model's is probably wrong — but not always.

RLHF Preference Ranking

You're picking between two model responses. Common failure modes: (1) favoring longer/more detailed responses when the rubric doesn't require it, (2) letting your personal preferences override the specified evaluation criteria. Tip: the rubric always wins over your gut. If the rubric says "prioritize factual accuracy" and Response A is more polished but slightly less accurate, A loses.

Creative Writing

You're evaluating story quality, tone, or fiction craft. Most subjective task type — but Outlier's screening still has "correct" answers in the sense of measurable inter-rater agreement. Tip: read the rubric's stated priorities (voice? plot? technical craft?) and treat those as fixed. Personal taste is not the standard.

Common rejection reasons (evidence-based)

• ✗
  Inconsistent grading across the sample. Not necessarily wrong on any single item — but grading item 3 as "great" and item 30 as "poor" when the rubric applied the same way should produce the same rating.
• ✗
  Not reading the rubric fully before grading. A ~15% additional pass rate is attributable, per public post-mortems, to reading the entire rubric before starting the first item.
• ✗
  Over-confidence on ambiguous items. Marking a genuinely ambiguous item as "definitely A" when the rubric would allow "probably A with reservations" costs points on ambiguity handling.
• ✗
  Ignoring the "why" field. Most Outlier assessments include a free-text explanation field. Skipping it or writing single-word justifications is a strong negative signal.

If you fail — what's next?

Rejection is not permanent on Outlier. You can typically retry the same domain after 30-90 days, or apply to a different domain immediately. If you consistently fail coding evaluation but have strong math credentials, apply to math reasoning instead. Many Outlier trainers succeed in one domain after failing another.

Common questions.