Regression testing before model/prompt changes ship

Build a frozen eval set of 200-2000 representative tasks with ground-truth labels (human or LLM-curated). Before each model/prompt/RAG corpus change ships, run all tasks through the candidate system, compute (1) automated metrics (RAGAS-style faithfulness, answer-relevance, task success rate); (2) LLM-as-judge pairwise comparison against the previous version; (3) cost/latency regression checks. Maintain a canary set: ~50 high-stakes queries you monitor even harder. Store results in CI with pass/fail thresholds per metric. Senior nuance: regression test suites must be diverse enough to catch subtle regressions: "5 hand-written queries" is worse than useless (overfits to individual prompt phrasings). Real practice: maintain a rolling 5000-query set that mixes gold labels with paired A/B trajectories.