Ask a team how good their work is and you'll usually get a number. Ask exactly what that number measures, and the answer starts to change shape.
Part of the reason is that a reviewer does two different things. They judge the work as it arrives, and they often correct it. Those are two facts about your operation, and they belong in two numbers.
The share of items that were right as submitted.
It tells you about annotators, training and guidelines.
The share that is right after review, meaning items right as submitted plus items corrected.
It tells you what the client is likely to receive, and how well review catches errors.
Delivered accuracy is usually the higher of the two, and the gap between them is how much repair work your reviewers are quietly doing.
Think of a kitchen. If every dish that's sent back and remade counts as “right the first time”, you've learned something about the remake and nothing about the cooking.
It's the same trap as counting every mismatch as a mistake, or every closed fix as a correction: a tidy number standing in for what actually happened.
Where the Signal Gets Muddied
Almost never on purpose. A reviewer edits the row and overwrites the original answer, so the first-pass error disappears. Corrected items get counted as passes. Fixed and unfixed work get pooled into one table. An item is marked “fixed” without anyone noting whether anything changed, so a fix that left the data identical counts as a success.
Then there's the sample. In many pipelines, QC audits a sample of what has already passed review. That means the score describes the output after review, not the annotator, and it's an estimate rather than a count of the whole batch. That's easy to forget, and the number then gets read as a verdict on people.
We kept the two apart on a recent project. Once QC verdicts and the fix file sat side by side, we could see which failed items had genuinely changed and which had passed through untouched. A single blended score would have counted both as handled.
Blending also changes behaviour. When a number becomes the target, people start working towards the number instead of the thing it was meant to measure (Strathern, 1997, p. 308). If the headline score rewards corrections, effort goes into closing items instead of preventing errors, and annotators end up judged on a figure that is partly the reviewers' work. And whatever the metric, one score is not a verdict on annotation quality (James, 2026).
A clean first-pass number also shows something less comfortable: sometimes the annotators are part of the problem. Guidelines get updated. Recurring problem items get shared, with notes on how to handle them. And the same mistakes still come back, partly because some annotators rarely open the material they've been given. A blended score hides this, because review corrections cover for it. First-pass accuracy, tracked by annotator before and after each guideline change, shows whether the change actually landed, and whether the next conversation should be about the guideline or about the person.
Questions to Ask of Any Quality Number
- Was it measured before or after review?
- Is it from a sample, and how was the sample chosen?
- Does it separate first-pass work from corrected work?
- When it moved, which part moved?
To make those answerable, keep the reviewer's verdict on the original item and store corrections in a separate field. Tag every fix as changed, checked or flagged. Judge annotators on first-pass quality, and the review process on delivered quality.
A quality number is only useful if you know what it measures.
Want to know what your quality number really measures?
Talk to Our TeamJames, J. (2026). Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation. arXiv:2603.06865. arxiv.org/abs/2603.06865
Strathern, M. (1997). “Improving ratings”: Audit in the British university system. European Review, 5(3), 305–321.