Volume labeling still has a place. It is not enough for work that depends on professional judgment. High-signal data comes from comparisons, corrections, and explanations that a model can learn from.
Start from the failure you want to fix
Do not collect "more data" in the abstract. Collect examples of the errors that matter: bad rankings, missed safety issues, weak reasoning, or incorrect domain facts. The dataset should be able to teach those distinctions.
Use expert tasks, not only labels
- Compare two outputs and say which is better, and why.
- Correct an output instead of only marking it wrong.
- Write a short explanation that a later model or reviewer can use.
- Create edge cases that are rare in logs but costly in production.
Control quality as you go
Gold examples, overlapping review, and batch rejection keep the set clean. If reviewers disagree, the task definition is unfinished. Fix the definition before you scale the queue.
Keep the trail
Every dataset version should record source, schema, reviewer requirements, and quality status. Training data without that trail is hard to trust and harder to improve.