Technical guide
Scope an AI evaluation task before assigning it
This public NIST AI Risk Management Framework-based planning checklist helps teams turn an evaluation question into a reviewable task brief.
1. Name the context and decision
Record the system or workflow, intended use, relevant users, and the decision the evaluation should inform. Context determines which harms, failures, and trade-offs deserve attention.
2. Define the task evidence
State the input, expected output, reference material, and evidence an evaluator may use. Separate factual checks from preference judgments and explain what to do when evidence is missing.
3. Write observable criteria
Turn the quality question into criteria a reviewer can inspect. Give boundary examples where interpretation is likely, and identify any criteria that should trigger escalation instead of a forced score.
4. Match reviewer context
List the language, domain knowledge, instructions, and tools needed for the judgment. Plan a calibration exercise using representative examples before relying on production review.
5. Plan disagreements and records
Define who resolves disagreement, what evidence they review, and how rubric, example, and decision changes are versioned. Preserve the task version with each result so later analysis has context.
6. Set acceptance questions
Decide what evidence is sufficient to accept the work, which failure patterns require revision, and when the task itself should be re-scoped. Sample across important cases rather than relying on one aggregate score.
Public technical basis
The NIST AI Risk Management Framework is the primary public source for this checklist. Its Govern, Map, Measure, and Manage functions provide the planning structure; teams should apply the original framework to their system and risk context.
Bring a reviewable brief
Use the checklist to prepare the task, evidence, rubric, reviewer context, and acceptance questions before discussing workforce or AI-data needs.