Building LLM Evaluation Sets to Test AI Features Before Launch

Building LLM Evaluation Sets to Test AI Features Before Launch — cover image

Most teams test their first AI feature the same way: they try five or six prompts, the answers look good, and they ship. Two weeks later someone edits the prompt to fix one complaint, and three other behaviours quietly break. Nobody notices until a customer does. An LLM evaluation set (often called an "eval set") is the fix. It is a collection of realistic inputs paired with a description of what a good answer looks like, which you run every time anything changes.

This guide explains how to build one without a research team or expensive tooling. It is written for founders, product managers, and engineers at startups and SMEs who are shipping chatbots, document extractors, summarisers, or agents and want to stop guessing whether a change made things better or worse.

Why spot checking AI output fails

Traditional software is deterministic. If a function returns the right value today, it returns the right value tomorrow. Language models are different: the same input can produce different outputs, and a small prompt tweak can shift behaviour in places you were not looking at. Spot checking fails for three reasons.

An eval set is a regression test suite for behaviour you cannot fully specify in code.

What goes into a good evaluation set

Each test case has three parts: an input, the context the model would see in production, and a definition of success. The definition of success is the part teams skip, and it is the part that makes the whole exercise useful.

Real inputs beat invented ones

Pull examples from your support inbox, chat logs, uploaded documents, or sales calls. Real users misspell words, paste messy text, ask two questions at once, and switch languages mid sentence. If you serve Indian customers, include Hinglish messages and regional name formats, because a model that handles clean English perfectly can stumble on these.

Cover four categories

Write the expected outcome, not the exact words

Language models rarely produce identical text twice, so do not compare against a single "golden answer". Instead describe the properties a correct answer must have. For a refund policy bot, that might be: "States the return window correctly, does not promise a refund for opened items, and offers to connect the customer to a human." Those are checkable statements.

A real-world example: a support assistant for an online store

Consider an illustrative scenario. A D2C brand launches an AI assistant that answers order and return questions. The team writes a prompt, tests ten questions, and ships. A month later the founder asks the assistant to sound friendlier. The new prompt works nicely for greetings, but the assistant starts offering refunds on final sale items, because the friendlier tone made it more agreeable.

With an eval set, this would have been caught in minutes. One test case reads: "Customer asks to return a final sale item. Expected: politely declines, explains the policy, offers store credit if the policy allows." The tone change fails that case, the team sees it before release, and they adjust the prompt. The eval set did not make the model smarter. It made the team aware of a trade off they had accidentally made.

Step by step: building your first eval set

  1. Define the job. Write one sentence describing what the AI feature must do and one sentence on what it must never do. These two sentences drive every test case.
  2. Collect 30 to 50 real examples. Export recent conversations or documents, remove personal data, and pick a spread across the four categories above. Our article on redacting PII before sending data to LLMs covers the clean up step.
  3. Write success criteria for each case. Use short, checkable statements. Prefer "mentions the 7 day window" over "answers well".
  4. Choose a grading method. Use code checks where you can (valid JSON, correct field present, forbidden phrase absent). Use an LLM judge for subjective qualities. Use human review for the small set of high stakes cases.
  5. Run a baseline. Execute the set against your current prompt and model and record the results. This is your reference point.
  6. Change one thing at a time. Modify the prompt, model, or retrieval settings, rerun the set, and compare. If you change three things together you will not know which one mattered.
  7. Automate it. Run the set in your CI pipeline so every pull request that touches AI behaviour shows a pass rate. Start with a manual script if you must, but make it a single command.
  8. Feed production back in. When a real user finds a failure, add it as a new test case. Over time the set becomes a record of everything your product has learned.

Choosing a grading method

Code based checks

These are cheap, fast, and reliable. If your model extracts invoice data, you can check that the total is a number, the date parses, and the vendor name matches a known list. Whenever an output can be validated by code, validate it by code.

LLM as judge

A second model reads the answer plus your rubric and returns pass or fail. This works well for tone, completeness, and faithfulness to source documents. Keep the rubric narrow ("Does the answer contradict the provided policy text?") and spot check the judge's decisions by hand every so often. A judge that is lenient will quietly hide regressions.

Human review

Reserve people for the cases where a wrong answer is expensive: legal wording, medical information, financial advice. A weekly review of a small sample is often enough, and it keeps the team close to how the feature actually behaves.

Evaluating retrieval and agents separately

If your feature uses retrieval augmented generation, a wrong answer can come from the retriever fetching the wrong passage or from the model misusing a good passage. Test them separately. For retrieval, record which document should have been found for each question and check whether it appears in the top results. For the generation step, supply the correct passage and see whether the model uses it faithfully.

Agents add another layer: tool choice and tool arguments. Test whether the agent picks the right tool, passes valid arguments, and stops when it should ask a human. When something fails in production, tracing and observability for LLM workflows tells you which step went wrong, and the failing trace becomes your next test case.

Key benefits of an evaluation set

Common mistakes to avoid

How to start this week

You do not need a platform to begin. Open a spreadsheet with four columns: input, context, success criteria, and result. Add 30 rows from real usage. Write a short script that sends each input to your AI feature, collects the outputs, and applies your checks. Run it, record the pass rate, and put that number in your team chat. From that point on, every change to the AI feature has a score attached to it.

If you want help designing evaluations, wiring them into delivery, or building the AI feature itself, our team works on this every day through our AI development services.

Conclusion

AI features do not fail loudly. They drift, regress, and surprise you in ways that a handful of manual tests will never reveal. A small evaluation set turns that uncertainty into a number you can track, a test you can automate, and a habit your whole team can share. Start with thirty real examples, write checkable success criteria, run them on every change, and keep adding the failures you find. Within a few weeks, you will ship AI changes with the same confidence you have shipping ordinary code.

Frequently Asked Questions

How many test cases does an LLM evaluation set need?
Start with 30 to 50 real examples that cover your common cases and your known failure modes. You can grow it over time as production surfaces new problems. A small, well chosen set that you actually run beats a huge set nobody maintains.
Can I use an LLM to grade another LLM's answers?
Yes, an LLM judge is a common approach for subjective qualities like tone or helpfulness. Treat it as a tool that needs checking: review a sample of its grades by hand, use a clear rubric, and prefer simple pass or fail questions over vague 1 to 10 scores.
When should I run evaluations?
Run them whenever you change a prompt, a model, a retrieval setting, or a tool definition. Wiring them into your CI pipeline means a regression shows up in a pull request instead of in a customer complaint.
What is the difference between evaluation and monitoring?
Evaluation tests your AI against known examples before release. Monitoring watches real traffic after release. You need both: evals stop known problems from returning, and monitoring finds the new problems that your test set did not anticipate.
Do small businesses really need AI evaluations?
If an AI feature talks to customers, touches money, or writes to your systems, yes. Even a spreadsheet of 30 examples with expected outcomes gives you far more confidence than reading a few outputs and hoping for the best.