Language
Name the language, locale, reading level, writing requirement, and any bilingual comparison the task requires. Separate conversational fluency from the ability to apply technical instructions in that language.
For employers
AI workforce
Prepare a workforce brief around observable qualifications, language and domain context, calibration, and the people who will review difficult decisions.
Define the work before the role title
A title such as evaluator, trainer, or subject-matter expert does not explain the judgment a contributor must make. Start with representative tasks. State what a person will read, write, compare, classify, or decide; which reference material they can use; and which decisions should be escalated to a reviewer.
Build a candidate qualification map
Domain
Describe the knowledge needed to judge the material and the evidence that would demonstrate it: work samples, credentials, a structured exercise, or experience with a specific kind of decision.
Task judgment
List the observable actions behind the role. Examples include checking a claim against a source, comparing two responses to a rubric, explaining a domain error, or recording why evidence is insufficient.
Operating context
Record time-zone overlap, tool access, data-handling constraints, expected task cadence, and the route for questions. Treat these as planning inputs rather than assumptions attached to a title.
Plan calibration before volume
Prepare a small set of representative examples with expected reasoning, including clear passes, clear failures, and boundary cases. Ask candidates or contributors to apply the same instructions. Compare where judgments diverge, then revise the instructions or examples before expanding the work.
Calibration should also reveal whether the qualification map is useful. If experienced reviewers disagree for understandable reasons, the rubric or reference material may need work before screening criteria become stricter.
Assign reviewer responsibilities
- Task owner: explains the intended use and decides when the work has answered the business or technical question.
- Domain reviewer: resolves subject-matter errors and documents the evidence behind difficult calls.
- Language reviewer: handles locale, fluency, and meaning questions when those are central to the task.
- Calibration lead: tracks disagreement patterns, instruction changes, and which examples belong to each version.
Questions for an acceptance brief
What demonstrates qualification? Which mistakes can be corrected with clearer instruction, and which require a different reviewer? How will the team sample work, route disagreement, and update calibration examples? Answering those questions creates a practical brief for a workforce discussion without assuming a delivery model in advance.
Bring the brief, not just a title
Share the task context, required judgment, source material, reviewer needs, and the decision your team is preparing to make.