How do you ship AI changes without silent quality regressions?

Conclusion

LLM behavior changes when you change anything:

• prompt or system instructions
• model version
• retrieval settings (RAG)
• tool permissions and routes

You need a small, repeatable eval + regression gate. The minimum reliable setup is:

1. a fixed eval set (50–200 cases)
2. versioned prompts + configs
3. tracked metrics (quality + safety + cost)
4. a release gate (must-pass regressions)

Without this, you only notice failures after customers do.

Explanation

Traditional testing checks deterministic code paths. LLM apps add probabilistic behavior.

Your goal is not perfect scores. It is detecting meaningful drift.

Common causes of regressions:

• prompt edits that change formatting or refusal behavior
• model upgrades changing tone/accuracy
• retrieval tweaks causing wrong sources to dominate
• tools being enabled for routes that shouldn’t have them

Evals give you a baseline. Regression tests tell you if you broke it.

Practical Guide

Step 1: define “what good looks like” (10 minutes)

Pick 3–6 metrics you care about:

• task success rate (does it solve the user goal?)
• policy compliance (refusal where required)
• hallucination rate (wrong claims)
• tool-call correctness (right tool, right arguments)
• latency (P95)
• cost (tokens/request)

Keep metrics boring and actionable.

Step 2: build a small eval set (20 minutes)

Start with 50 cases:

• 30 common user queries
• 10 edge cases
• 10 adversarial/safety cases

For each case, store:

• input
• expected shape (not necessarily exact text)
• allowed tools (if any)
• required citations or doc_ids (if using RAG)

Rule:

• your eval set must include your top support tickets

Step 3: version everything that changes behavior (10 minutes)

Version:

• prompts
• model name/version
• retrieval config (top-k, filters)
• tool allowlists

If it changes output, it needs a version.

Step 4: choose a grading method (15 minutes)

Three practical options:

• exact checks (JSON schema, required fields)
• heuristic checks (regex for forbidden content)
• LLM-as-judge (with a fixed rubric)

Best practice:

• use exact checks for structure
• use LLM judge for “did it answer well?”
• always keep judge prompts stable

Step 5: add a release gate (10 minutes)

Define must-pass rules, e.g.:

• no safety case failures
• tool-call error rate < 1%
• cost increase < 15%

Then block merges/deploys when gates fail.

Step 6: monitor post-release drift (10 minutes)

After shipping, monitor:

• refusal rate changes
• token spikes
• tool-call spikes
• user feedback and thumbs-down

Evals catch regressions before deploy. Monitoring catches what you missed.

Pitfalls

• changing judge prompts every run (no baseline)
• using only “happy path” eval cases
• measuring only quality and ignoring cost/latency
• letting eval datasets drift without versioning
• shipping tool access changes without tests

Checklist

• [ ] I defined 3–6 success metrics (quality, safety, cost)
• [ ] I have an eval set (50–200 cases) with real user queries
• [ ] Edge and adversarial cases exist
• [ ] Prompts/models/retrieval/tool policies are versioned
• [ ] Output structure has exact checks (schema/required fields)
• [ ] Safety rules have deterministic checks
• [ ] LLM judge rubric is stable and versioned
• [ ] I track cost and latency in eval runs
• [ ] A release gate blocks deploys on regressions
• [ ] Post-release monitoring watches drift signals

FAQ

1) Do I need big eval datasets?

No. A small, representative set plus strict gates is better than a huge set you never run.

2) Should I use LLM-as-judge?

Yes, but keep the rubric stable and combine with deterministic checks for structure and safety.

3) What’s the fastest first step?

Collect 50 real queries (including failures) and run them before/after prompt or model changes.

Internal links

• Hub: AI development
• Related:
  • Minimal LLM logging checklist
  • LLM rate limits and retries checklist
  • LLM cost guardrails checklist

Disclaimer

General engineering guidance only.