Context windows and their practical limits

A context window is the maximum total tokens (input + output) a model can attend to in a single forward pass. Modern frontier models advertise 128k-2M tokens but effective usable length is usually shorter because of (1) attention cost: O(n^2) memory in vanilla attention, driving per-token latency and memory; (2) the lost-in-the-middle degradation; (3) degradation when the bulk of training data was at short contexts; (4) cost: input tokens still cost real money. Practical limits: ~32k-64k tokens is the sweet spot for most RAG workloads, model "advertised" length is asymptotic, and Pinecone/Weaviate-grade retrieval plus reranking usually beats simply stuffing more context. Senior nuance: techniques like FlashAttention-2 make 100k+ feasible on a single GPU; long-context training (RoPE scaling, YaRN) and post-training specifically on long-context tasks (e.g., ANELLA, NoLiMa) materially change effective limits. Don't conflate marketing max tokens with reliable retrieval recall.