Evaluation is a decision process. The goal is not a single leaderboard number. It is evidence that a model is fit for a defined job.
1. Define the job
Write the user, the task, the risk, and what "good" means. A coding assistant, a document summarizer, and a customer agent need different criteria. If you cannot name the decision, you cannot evaluate it.
2. Choose the sample
Use real outputs or realistic prompts. Include ordinary cases and the edge cases that would be costly in production. A convenience sample of easy prompts will flatter every model.
3. Match the expertise
Reviewers should be qualified for the content. Domain errors are invisible to people outside the field. For high-stakes items, use more than one independent review.
4. Collect structured judgments
Score or rank against the written criteria. Require a short explanation when an item fails. Capture disagreement instead of forcing a false consensus on the first pass.
5. Decide, then keep the loop
Compare models on the same set. Record why a model was chosen. Re-run the set when prompts, tools, or the model version change. Evaluation that cannot be repeated is not a baseline.