Scaling LLM inference for traffic spikes

Strategy stack 2025-2026: (1) autoscaling GPU pools (Kubernetes + custom metrics); (2) request batching and queueing: absorb spikes with throttle/defer queues; (3) prompt caching: identical system prompts become one cached prefix, massive TTFT and cost wins; (4) tiered model routing: small/cheap model for easy queries, large model only when the small one is uncertain; (5) KV cache reuse across turns of the same conversation; (6) speculative prefill: start generating the next response while the model still finishes the previous one (pipelining). Senior nuance: smooth overload via graceful degradation is better than sharp 503s: cap concurrent decodes per user, prioritize short responses when latency budget is tight.