Monitoring RAG in production

Production RAG monitoring has five pillars: (1) Logs: capture (query, retrieved_chunks, generated_answer, citations_used, latency, model_version); (2) Retrieval: track Recall@k on a rolling labeled set, ANN-search-index freshness, embedding-model version drift; (3) Generation: sample 1-5% of traffic to LLM-as-judge (faithfulness, answer-relevance) and human spot-audit; (4) Cost/latency: p50/p99 by query, exploded by retrieval + LLM cost; (5) User feedback: thumbs up/down, follow-up rates. Most common regression patterns: doc corpus silently updated, embedding version pinning lost, top-k too small after a corpus growth. Senior nuance: monitor context relevance and groundedness separately; if context relevance drops first, fix retrieval; if generation drops while context relevance holds, fix prompts or model.