LLM observability vs classical ML observability

LLM observability requires: (1) full trajectory logs: prompt, retrieved context, model version, raw output, tool calls, cost, latency, user feedback; (2) eval pipelines running on sampled production data; (3) drift detection: both input drift (embedding-based topic shifts) and output drift (semantic embedding of responses, LLM-judge scores over time); (4) regression testing on a frozen task bank before every model or prompt change; (5) user feedback collection (thumbs, edits, follow-ups). Unlike classical ML observability: (a) outputs are open-ended (no fixed label schema); (b) the system is non-deterministic: sampling temperature matters; (c) prompts change frequently and silently drift; (d) "failures" are nuanced (hallucinations, factual errors, refraining too much) and not label-able as binary. Senior view: track cost-per-task not per-call, alert on both eval metric degradation and traffic shifts.