This concise reference uses general AI-evaluation terminology. Definitions can help readers interpret task descriptions, but their application varies by project.
Core evaluation terms
RLHF (Reinforcement Learning from Human Feedback): Reinforcement-learning methods that use human feedback to shape a reward signal guiding model behavior.
Review role details and application requirements before you apply.
Evaluation (eval): A process for examining model outputs against a stated task or criteria.
Rubric: Written criteria used to make an evaluation more explicit and repeatable.
Calibration: An alignment activity where reviewers discuss or apply a rubric; how it is run varies by project.
Inter-rater agreement: A measure of how often independent reviewers reach similar conclusions. It measures consistency, not correctness by itself.
People and artifacts
Annotation: Adding labels, classifications, or notes to data or model outputs for a defined task.
Domain expert: A reviewer with subject-matter knowledge relevant to a task.
Reference answer: An example response or decision used to illustrate how a task may be judged; its role varies by project.
Systems and safety
AI agent: A system that can use model outputs with tools or steps to pursue a task.
Red teaming: Structured attempts to identify unsafe, unreliable, or unintended behavior in a system.
Further reading
These sources illustrate specific approaches to human feedback, agreement, rubric-based evaluation, and red teaming. They do not define every project or term above.
- OpenAI: Learning from human preferences describes a human-preference learning approach.
- Google Cloud: side-by-side evaluation discusses Cohen’s kappa as an agreement measure; agreement does not by itself establish correctness.
- Microsoft Research: RUBICON discusses rubric-based evaluation and notes that high agreement can still be subjective or error-prone.
- Anthropic: Red teaming language models describes red teaming used to discover and measure harmful behavior.