Write the evaluation before you write the agent

The pattern is consistent enough to be boring. A team builds an LLM feature, it demos well, it ships, and then it slowly stops working. Somebody tweaks a prompt to fix a complaint and breaks three cases nobody was watching. Six weeks later the feature has a reputation and no owner.

The missing piece is never the model. It is that there was no fixed set of examples with known-correct answers, so every change was a matter of opinion.

Build the dataset from complaints

You do not need a thousand labelled examples to start. You need thirty, and the fastest source is the list of cases people have already complained about. Write each one down as an input, the output you got, and the output you wanted. That is an evaluation set.

Then run it in CI on every change to a prompt, a model version, a retrieval parameter, or a tool definition. The point is not a high score. The point is that a regression becomes a failed build instead of a support ticket.

Score the thing you actually care about

Resist grading on similarity to a reference answer. For a classification task, score the label. For an extraction task, score the fields. For a drafting task, score the properties that matter — did it cite a real source, did it stay under the word limit, did it avoid making a commitment on the company's behalf.

def score(case, output):
    checks = {
        "cites_real_source": all(s in case.corpus for s in output.sources),
        "within_limit": len(output.text.split()) <= 180,
        "no_commitment": not COMMITMENT_RE.search(output.text),
        "label_correct": output.label == case.expected_label,
    }
    return checks, all(checks.values())

Four boolean checks beat one similarity score, because when the build fails you know which property broke.

What this costs

About two days at the start of a project, and roughly an hour a week to add cases as new failures appear. In exchange, you get the ability to change models. When a cheaper or better model arrives, you run the suite and you know within an hour whether you can switch. Teams without a suite are stuck paying for whatever they launched with, because nobody can prove a change is safe.

An AI feature without an evaluation suite is a feature you can no longer safely change.

This is why evaluation is the first thing we scope on an AI engagement and the first thing we hand over. Agents are the interesting part. The suite is the part that keeps them working after we leave.