Detecting distribution shift in production LLM traffic

Three layers: (1) embedding-based topic shift: sample incoming queries, embed, cluster, compare cluster distribution over rolling window (e.g., 1 day vs 7 days); alert when top-K cluster centroids move > X distance. (2) lexical shift: watch token distribution, language mix, average length, special-character ratio for prompt injection markers. (3) judge metric drift: sample 1% of outputs, run LLM-as-judge, alert when judge scores fall > Y or variance grows. Tools: Arize Phoenix, whylogs, Evidently, custom Prometheus dashboards. Senior nuance: input drift often precedes model-quality degradation by days: early warning reduces incidents. Beware of false positives during scheduled feature changes (e.g., new UI fields).