Build your first AI evaluation set

An evaluation is a question with a scoring rule

For a note-to-checklist app, the question might be: does the output contain only actions stated in the note? Your scoring rule could check unsupported additions, missed actions and output structure separately. Decide those rules before comparing versions.

Evaluation is not one universal score. A system can format answers correctly and still invent facts. Start with the dimensions that matter for your specific task, and make each failure inspectable.

Write a small set you can read end to end

For this exercise, create twelve invented notes: four ordinary, four ambiguous and four edge cases. Twelve is a convenient starting size for a learning project, not a statistical guarantee. Include an empty input, a note with no action and a note that explicitly cancels an earlier action.

Store each case with an identifier and expected behaviour. When there is more than one valid wording, write a criterion rather than one exact sentence. Keep sensitive real-world records out of a practice dataset.

case_id: cancelled-task
input: "Prepare the slides. Update: cancel the slides task."
expected: No active slides task is returned.
checks:
  - valid output structure
  - no unsupported actions
  - cancellation respected

Choose the simplest scoring method that fits

Code can check output structure, required fields or membership in a fixed set of labels. Human review is useful when correctness depends on meaning. A model-based grader can help with scale, but it should be checked against human judgements and a written rubric.

Do not use exact string matching to grade an answer that permits many correct phrasings. Conversely, do not ask another model to decide whether a required field exists when ordinary code can answer the question directly.

Compare versions on the same cases

Record the prompt version, model identifier, settings and outputs. Run version A and version B on the same dataset. Review the changed cases, including examples where the newer version made things worse. If outputs vary, repeat runs rather than treating a single sample as definitive.

Keep some cases aside while you tune. Repeatedly modifying the prompt to fit every known example can hide weaknesses on unfamiliar inputs. Once you inspect a held-out case and tune against it, it is no longer an untouched final check.

Turn failures into a development loop

A useful report names the failure, links to the input and explains the next experiment. “Nine out of twelve passed” is less helpful than knowing that two failures ignored cancellations and one produced an invalid shape. For a small set, report counts and examples rather than implying precise population reliability.

At the end of the exercise, add one regression case for every bug you fix. Keep the test set versioned beside the project. Your goal is not to make every number green; it is to make a change’s consequences visible before someone relies on the output.