A/B testing LLM products

Best practice 2025-2026: (1) interleaved A/B testing: randomly assign user requests to candidate A or B but interleave to control for time-of-day effects; (2) primary metric: task success rate (binary label, LLM-judge calibrated against humans) or user thumbs/retention if available; (3) guardrail metrics: latency p99, hallucination rate, refusal rate, cost-per-task. Stop after pre-set sample size (don't peek). For subjective quality without binary labels: pairwise comparison via LLM judge; measure win-rate with statistical confidence intervals. Senior view: power analysis for win-rate is tricky: small absolute differences (e.g., 51% vs 49%) require tens of thousands of comparisons to call. Always run interleaved, never switch all-at-once: you cannot disentangle "model change" from "user adaptation".