Asking whether a search result is good sounds simple. So does asking which of two AI-written answers is better. Both turn out to be hard, and for the same reason: there is no answer key, only a careful reader with good judgement and clear guidelines.
Between January and October 2025, a Yenda team in Zambia worked with Tasq.ai on two connected evaluation projects for one of the world's largest online community platforms. The first rated search relevance. The second evaluated answers from the platform's new AI search feature, which builds each answer out of quotes from community posts and comments. The same trained people moved from one project into the next. This is what the work involved and what it taught us.
Search Relevance Evaluation Is a Judgement About Intent
For every search query and post, a rater decided how well that post would satisfy a real member of the community. The scale ran from very satisfactory to very unsatisfactory, and each rating needed a reason: mismatched intent, outdated, too broad, too narrow, not safe for work, or the wrong language.
The scale was the easy part. Three questions sat behind every rating:
- What do most people mean by this search, and what do only a few mean?
- Does this person want something recent, or something that doesn't date?
- How is searching a community platform different from searching the open web?
The team delivered 85,809 query-and-post ratings in six batches. An annotator rated each one, a reviewer checked it, and team leads spot-checked samples before delivery. The client sent feedback after every batch, and we used it to retrain the team before the next one opened.
AI Answer Evaluation Needs a Clear Order of Priorities
The second project was a side-by-side evaluation. Raters read two AI-generated answers to the same question, picked the better one and explained why. They scored four things, in this order:
- Topicality: does it answer the question that was asked?
- Accuracy: is it correct and current?
- Presentation: is it clear and well organised?
- Hallucination: does every quote match its source?
On sensitive topics such as health, politics or religion, safety and bias came first. Every quote was checked against the post it came from, and every judgement carried a written explanation graded by severity. Each of the 1,797 answer pairs received three independent judgements.
Lesson 1: A Low First Score Is Often a Guideline Problem
Our first sample came back below target. When we went through the errors, most traced back to how the team had read the guidelines, not to a lack of care.
So we sent questions. What should a rater do with a quote where a typo has been corrected? With a link that leads off the platform? How do you judge whether the surrounding posts are relevant context? Those questions, and the feedback that followed, led the client to add severity levels to its guidelines. We have written before about why disagreement is a signal worth reading, and this was a clear case of it.
Lesson 2: Split a Long Rubric Between Specialist Layers
One person holding four criteria, a safety override and a severity scale in their head will be slow and inconsistent. We proposed splitting the work:
- Layer 1 checks hallucination and presentation.
- Layer 2 checks accuracy, topicality and safety.
- Team leads check the work before delivery and log every fix in a quality tracker.
A common-mistakes guide, built from client feedback, trained each new member. The work became both more accurate and faster.
Lesson 3: Hallucination Detection Is the Hardest Skill to Find
Checking a quote against its source sounds mechanical. In practice it was the skill our team found hardest, so we now screen for it during training instead of hoping it develops on the job. Training runs in two stages: five items, feedback from a lead, then five more.
Lesson 4: Keep the Team That Already Knows the Content
Raters who had spent months on search relevance already knew the platform's content and Tasq.ai's tooling, and they ramped up faster on AI answer evaluation than people starting cold. That depth also bought speed. In August 2025 the client added a new task, asking raters for a short, honest first reaction to each answer pair, as a real user would give. We sourced, trained and launched that team in three days.
If You Are Evaluating AI Search
- Rank your criteria, and say what overrides the ranking.
- Define severity before the first batch, not after it.
- Treat your evaluators' questions as data about your guidelines.
Evaluating search results or AI-generated answers?
Talk to Our Team