Evaluating LLM agents without going broke

There's a tier of LLM evaluation work that costs five figures a month: thousands of synthetic conversations against rubric-graded judges, A/B-tested model swaps, full RAGAS pipelines, the whole apparatus. If you're at a frontier lab, that's the right amount. If you're a single engineer shipping an AI feature on the side of a product, it's not.

Here's the playbook I use instead. It works on a portfolio budget and catches the regressions that actually matter.

Tier 1: a golden set you write by hand.

For any LLM feature, the first eval I build is a flat file — JSONL is fine — of fifteen to fifty handwritten input → expected-output pairs. For /ask on this site it's question → expected-source-file. For a customer-facing chatbot it might be intent → expected-tool-call. For a code-review agent it might be diff → expected-flag.

Rules of thumb:

• Write them yourself. Generated golden sets compound your model's biases.
• Cover the failure modes you're most afraid of, not the happy paths. The happy paths work; the failure modes are why you're here.
• Keep it under fifty. Past that, the maintenance cost (every prompt change invalidates the rationale for half of them) outweighs the signal.

Cost: a couple of hours of your time. Catches: most of the regressions that actually bite — more than any other single hour I spend. I won't pretend I have a clean denominator to turn that into a percentage.

Tier 2: deterministic checks on the retrieval / structure.

Most LLM systems have a non-LLM substrate — retrieval, classification, parsing, tool selection. That substrate is deterministic enough to test like normal code.

For /ask, that means recall@5 over the golden set. If the agent generates the wrong answer, the first thing I look at is whether retrieval gave it the right chunks. If the retrieval was right and the generation was wrong, that's a prompt problem. If the retrieval was wrong, that's a chunking or embedding problem. Separating these is the biggest debugging speedup I know.

For other agents: did it pick the right tool? Did it parse the user input into the right schema? Did the structured output validate?

Cost: trivial — these are unit tests with extra steps. Catches: silent regressions when you tweak chunking, embedding model, or retrieval parameters.

Tier 3: an LLM judge for output quality.

This is the tier most people start with and shouldn't. Using an LLM to grade an LLM gets expensive fast and is the noisiest of the three tiers.

When I do use a judge, I use it sparingly:

• Pin the judge to a stronger model than the one being evaluated (Claude Opus judging Claude Sonnet, GPT-4o judging GPT-4o-mini).
• Give it a short, specific rubric — three to five binary criteria, not a numeric score. "Does the answer cite a source from the corpus? Yes/No."
• Run it on a small sample (10–20 outputs) per prompt change, not on every commit.

Cost: $0.05–0.50 per evaluation run, depending on sample size. Catches: quality regressions that don't show up in retrieval metrics.

What I deliberately don't do.

• No bigger-is-better leaderboards. MMLU and GSM8K tell me nothing about whether the agent on my site refuses gracefully when a user asks about my dating life.
• No production telemetry as the primary eval. Telemetry is great for finding problems you didn't predict. It's terrible for verifying that the change you made five minutes ago didn't break the thing you cared about.
• No prompt-tuning by vibes. Two evenings of "this feels better" is two evenings you could have spent building the golden set. The golden set will save you the same amount of time on every subsequent change.

A reasonable budget.

For an AI feature in a small product:

• Golden set: 30–50 examples, refreshed quarterly. Costs your time, not your budget.
• Deterministic retrieval/structure tests: free, run on every PR.
• LLM judge: 20 examples, run on prompt changes. ~$1/run.
• Production telemetry: log inputs, retrieval results, outputs, cost. Sample for review weekly.

Total monthly cost at the team level: under $50. Total benefit: you stop shipping silent regressions.

The temptation is to skip evals because they're "not the fun part." The fun part is also the part that lies to you. Evals don't.

— Owen