A QC check fails an item. Where does it go next?
If your answer is “back to whoever did it”, you're in good company. It feels right. They know the task, they have the context, and it seems fair that the person who made the mistake cleans it up.
It's also the quickest way to get the same error back with a tick next to it.
Think about what you're really asking. You're asking someone to look again at a decision they made a short while ago, using the same understanding of the rule that produced it. Anyone who has proofread their own work knows how that goes: you read what you meant to write, not what's on the page. People also tend to look for evidence that their first answer was right, not that it was wrong (Nickerson, 1998). So a second pass by the same person is often a re-confirmation dressed up as a review.
If the error came from a misread guideline, the misreading goes back with the item. Nothing about their understanding has changed, so what they see hasn't either.
Then there's the shortcut. “This field failed” is a very easy instruction to obey. Flip the field, tick the box, move on. The item passes. Why it failed is untouched, and it will be back in the next batch, along with the same error.
We learned this the practical way. On our projects, rework goes to someone other than the original annotator, because when people fix their own items, very little changes.
A different fixer isn't a cure on its own, though. On a recent project, we compared QC's verdicts with the fix file, and some items QC had failed came out of the fix stage exactly as they went in. Marked done, counted, closed, with nothing corrected. A tidy way to look busy, and not much else.
That's easy to miss at volume. A queue can show every item as finished while the errors underneath are still there. Who does the fixing, and whether they have time to look properly, makes a real difference.
So What Actually Works?
Two things, in this order.
Ideally someone with the guideline open, the reason for the failure in front of them, and no first answer to defend. Independent judgements stop one person's reading from being the only reading, and collecting more than one independent label on uncertain items is a well-studied way to improve data quality (Sheng, Provost & Ipeirotis, 2008).
This is the step most teams skip. Track every fix by what it did: changed, checked and left as it was, or flagged as a guideline conflict.
An unchanged fix isn't automatically a failure. Sometimes the item was right and the QC fail was wrong, and that's worth knowing too. But every unchanged fix should be one of two things, a missed fix or a wrong fail, and someone should be able to say which. A team that can answer that has a fix process it can stand behind.
Report the quality of fixed work separately from first-pass work, or a weak fix process will hide behind a strong first pass.
Don't cut the original annotator out entirely. They shouldn't be the one repairing the item, but they should still hear about it, with feedback that addresses the misunderstanding behind the error, not just the wrong answer. And if the same rule keeps coming back after that, stop reworking and look at the cause. It may be a gap in the guideline, not a person.
Treat It as a Process, Not a Switch
A second person is a starting point, not a guarantee. It matters who that person is: someone who knows the rule well and has time to look at the item properly rather than rush through it. Get either wrong and a different fixer can still hand back an unchanged item.
That's why we treat rework as continuous improvement. We look at whose fixes change what QC flagged and whose don't, how long a fix takes, and whether the people we route to have what they need to do it well. Then we adjust who gets what and look again. What works on one project may not work on the next, and the only way to know is to keep checking.
What If Only One Person Knows the Task?
It happens, especially on small or specialist projects. If rework has to go back to the author, borrow the independence from somewhere else: have a second person review a sample of the reworked items without seeing the first fix, and compare. It costs a little time. Skipping it costs you the same error, delivered twice.
Rework should add a second, independent look. Sending it back to the same person only gives you the first look twice.
Want a second look at how your rework is handled?
Talk to Our TeamNickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? Improving data quality and data mining using multiple, noisy labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.