Building an evaluation set for an internal LLM workflow before it goes live: sampling real cases, writing grading rubrics, and running regression checks after every prompt change

Here's the trap, and I've watched at least four different teams fall into it. You build a summarization tool, or a ticket triage bot, or whatever internal workflow your team decided Claude or GPT-4 should handle now. You run five examples through it, they look clean, and you ship it.

Three weeks later somebody tweaks the system prompt to fix an unrelated formatting complaint, and now the model is inventing customer names in roughly one out of every ten summaries. Nobody notices for two weeks because nobody was watching that number, mostly because nobody was tracking a number at all.

I think about this the way I think about a mechanic who tightens your brakes while changing the oil, except he doesn't actually check the brakes afterward. Both jobs happened under the hood, but only one got tested before the car left the lot.

An eval set is how you avoid being that mechanic. It's a fixed pile of real test cases plus a grading rubric, and the workflow has to pass it every time someone touches the prompt, swaps the model, or fiddles with retrieval. No pass, no deploy.

Step One: Pull Real Cases, Not Cases You Made Up on a Tuesday

The biggest mistake I see, over and over, is teams writing hypothetical test cases based on what they assume users will type. That's a bit like designing a driving exam around streets you've personally never driven on. You end up testing your imagination rather than the road.

Go grab actual inputs from wherever this thing will live. Ticket classifier? Pull real tickets, including the unhinged ones. I once inherited an eval set that had exactly zero tickets written in all lowercase with no punctuation, and it turned out those were 15% of real traffic, usually from the angriest customers. The model choked on every single one, and we only found out because someone finally bothered to look.

Same logic applies to document summarizers. Pull the actual PDFs people upload, tables and all, including the ones where the table got exported as a wall of tab characters that makes every parser want to quit its job.

A couple things worth keeping in mind while you sample:

• Grab edge cases on purpose: the empty input, the 10,000-word monster, the message that switches languages halfway through.
• Include easy cases too. You need a floor to stand on, not just a horror show.

Volume matters here, and it's not a fixed number; it depends on how varied your traffic actually is. Dozens works for a narrow task, while low hundreds is safer if inputs are all over the place. Skip the sampling step and your eval set just mirrors whatever you already believed about the problem, limiting it to confirming assumptions you already held.

Step Two: Write the Rubric Before You Look at a Single Output

Most people get impatient here and jump straight to "does this look right?" Don't. Write the rubric before anything gets run through the model, or you'll end up grading based on whatever the model happened to produce that day. That amounts to writing the answer key after the test's already been graded, which means you're just agreeing with yourself rather than evaluating anything.

A workable rubric usually covers four things, and I want to be upfront that it's four and not the tidier-sounding three, because the fourth one is the one people forget until it bites them.

Did it get the core task right? Did it avoid making things up, since hallucination gets its own line because it hides well and shows up in outputs that otherwise read fine? Did it follow the format you asked for, whether that's JSON, a word count, or a tone? Did it handle weird inputs gracefully instead of confidently producing nonsense?

Score each one plainly. Pass or fail is fine, and a 1-to-5 scale works if you need finer grain. What doesn't work is vibes-based scoring, where two people look at the same output and land in completely different places because nobody defined what "good" actually meant going in. I've sat in review meetings that turned into forty-minute arguments over a single summary, purely because the rubric didn't exist yet.

Step Three: Treat Regression Checks Like Muscle Memory

Once you've got real cases and a rubric, the eval set is your regression suite. Change the prompt, swap the model version, adjust retrieval, run the full set again before anything ships. Software teams have run tests before every deploy for decades now; LLM workflows need the same habit, just with rubrics doing the work assertions used to do.

Here's a version of a story I've heard, with small variations, from at least three different teams. Someone fixes a bug where the assistant rambled too much, and the fix is to add "be concise" to the system prompt, ship it, and feel good about it. Except now the assistant is trimming legally required disclaimers because it's optimizing for brevity above literally everything else, including the thing it's legally supposed to say. Nobody catches it until a customer asks why the refund policy summary doesn't mention the actual refund policy.

An eval set catches this in the format column, assuming "include the disclaimer" was written into the rubric on day one. That's the whole reason the rubric exists before you run anything.

The Uncomfortable Part: This Takes Longer Than You Want It To

Sampling real cases, writing rubrics, scoring outputs by hand at first (yes, by hand, at least initially) all feels slow next to just shipping the thing and seeing what happens. Teams that skip this step are usually under a deadline, and evaluation feels like homework nobody assigned them.

Skipping it doesn't make the work disappear, though, it just moves the work later and makes it meaner. An afternoon writing rubrics turns into a Slack message at 11pm titled "why is the bot lying to customers," which is a considerably worse way to spend your evening than writing a spreadsheet.

What It Looks Like Once It's Actually Running

Once the eval set exists, prompt changes stop being nerve-wracking guesses. Someone proposes a tweak, runs it against the set, and gets a scorecard back. Did accuracy hold? Did hallucinations creep up? Did format compliance slip because the new prompt got a little too "creative" for its own good?

LLM development stays probabilistic and occasionally weird even with all this in place, but it becomes measurable, and measurable beats vibes every time. The question stops being "does this feel better" and turns into "did it pass the same 80 cases it passed last week, graded the same way it was graded then."

Not glamorous, and not the kind of thing you put in a demo, but this is the gap between a workflow you can actually trust and one you're just hoping doesn't embarrass you in front of a customer at 2pm on a Tuesday.