Prompt caching and KV cache prefix sharing

Many production prompts have large static prefixes (system prompt, tool definitions, retrieved context, persona instructions). If two requests share the first N tokens of the prompt, KV computations for those tokens are identical and need to be done only once. Prompt caching stores the KV cache of a prefix so subsequent matching requests reuse it. Implementations: OpenAI prompt caching (auto-prefix-match), vLLM automatic prefix caching (APC), SGLang RadixAttention (more aggressive). Cost math: a 5000-token system prompt at $1/1M input tokens = $0.005 per request if uncached, vs ~$0 reused if cached. For an app with stable system prompts, this is a 50-90% cost reduction. Senior nuance: only prefix-EXACT matches cache; usually resolve this by routing same system-prompt apps through the same instance and standardizing prompt format.