In our last piece, “Keep Your QC Signal Clean”, we argued for reading two numbers separately: how accurate an annotator is on the first pass, and how accurate the delivered work is after review. Both come from sampling, and sampling takes time. Pace does not. Every submitted item arrives with a time on it, so it is tempting to treat that as the cheap shortcut. Whoever goes fastest is cutting corners, right?
We tested that on a large batch from one of our most recent projects. As a rule for everyone, it does not hold. As a clue about a few people, it is useful.
What the batch showed
We took the 76 annotators who had done 20 or more whole videos start to finish and split them into four equal groups by pace. The fastest group needed about 34 minutes of work per minute of video. The slowest needed about 129.
If speed meant carelessness, the fastest group should have needed the most corrections. It did not. QC fixes per video were 2.3, 2.2, 2.4 and 2.9, from fastest to slowest. Videos sent back for redo did not follow pace either: 4.9% for the fastest group, then 3.6%, 3.6% and 4.4%. Some of the fastest people on the batch were also among the cleanest. They were not rushing. They just knew the task, the way a fast typist is not a careless one.
That does not mean nobody was fast and wrong. A few were, and reviewers were correcting their work on the QC layer. Among the busiest annotators, three of the fastest, at 24, 28 and 37 minutes per minute of video, needed 3.8, 3.5 and 3.1 QC fixes per video. Others at almost the same pace needed about two or fewer. Same speed, very different quality. The group average had hidden them.
Why pace misleads
Fast is often skilled. When you rank by speed alone, you end up penalizing the people you most want to keep on your team.
Slow is not safe. The slowest group had slightly more fixes per video, not fewer. Staring at an item is not the same as checking it.
Pace depends on the work. Compare people on a similar mix of item types and lengths, or the ranking shows who was dealt the harder items. This is a tougher balancing act.
The clock can be wrong. On this batch, a timer that tracked working hours had no time matched to 11 annotators, two of them among our top four by volume, and most of the “missed shift” flags raised from it were false. Pace should come from the console and claim times, which record the work itself.
Fix counts are noisy too. Reviewers differ in how much they correct, as we saw last time, so a fix count is a clue and not a verdict. That is why we tag each fix as changed, checked or flagged.
What to do instead
Treat pace as a screen. It tells you where to look first. It does not tell you what you will find.
Set a floor, not a ranking. Work out the least effort an item can plausibly take, measured from console and claim times. Someone working well below it cannot have done the item properly, however good the output looks. For this batch, our review set the line at 15 minutes of work per minute of video. Another task needs its own number.
Then read the quality signals. For anyone under the floor, check first-pass and delivered accuracy, as in the last article. If both are healthy, the floor is wrong or the person is simply good. If not, you found a real problem early.
Look at both ends, and say what the floor is. Someone who is stuck needs training. Someone who is rushing needs a conversation. Tell annotators the floor in advance, because a floor that is known changes behaviour and a secret one is a trap.
Reward the fast and accurate. They are your best case for the next batch. Give them the first claims and let them carry the newer people.
Pace points, accuracy decides
Last time the point was to keep the first-pass signal clean. This time it is to keep pace in its place: a way to decide where to look.
A speedometer tells you how fast you are going. It does not tell you whether you took the right road. Accuracy decides.
Want to know what your pace data is really telling you?
Talk to Our Team