Latency vs throughput vs cost: choosing batch sizes

Three objectives are typically in tension: (1) p50/p99 latency (TTFT + TPOT) for user-facing apps; (2) throughput (tokens/sec across all users); (3) cost ($/M tokens served). As batch size increases, throughput rises (better GPU utilization), but per-request latency also rises (each step takes longer for everyone). The right batch is where the marginal latency cost equals the marginal throughput benefit: typically mid-load for chat apps, maxed-out for batch processing. For the LLM-specific tradeoff: chat has strict TTFT (<300ms ideal) and TPOT (<50ms ideal); one can saturate decode memory bandwidth at batch >= 8 typically. Cost models: input tokens are 5-10x cheaper than output tokens (because prefill is amortized), so prompt caching is huge. Senior nuance: continuous batching + speculative decoding + quantization is the modern serving stack: each gives ~2x wins and the combination is ~5-10x.