The model never sees text — it sees token ids from a byte-level BPE vocabulary. How merges are learned, why 50,257 is not arbitrary, why “strawberry” and arithmetic break at the tokenizer (not the network), the 4:1 chars-to-tokens economics, and the tokenizer-level failure modes that no fine-tune can fix.

• →Tokens and Generation
• →Transformer Architecture

Scaled dot-product attention, derived. Why Q/K/V are three projections of the same input, why we divide by √d_k (the exact variance argument), what multi-head actually is (a shape split, not extra compute), the exact tensor shapes through one head, and the quadratic cost that defines everything downstream.

• →Transformer Architecture
• →Attention and Training Stability

What the embedding vector actually encodes, how encoders pool tokens into one vector, and the position problem: attention is permutation-invariant, so order must be injected. Sinusoidal vs learned-absolute vs RoPE — the rotation identity that makes RoPE relative — and how the positional scheme caps your context length.

• →Embeddings and Position
• →Transformer Architecture

How attention and the FFN are wired into a block, and why the wiring is the part that determines whether a 70B model trains at all. Residual connections, LayerNorm vs RMSNorm, pre- vs post-norm (the gradient-flow argument), the 4× FFN and SwiGLU, and why stacking depth works.

• →Transformer Architecture
• →Attention and Training Stability

What actually happens when you call the model: the autoregressive loop, the prefill/decode split and why their costs are inverted, KV-cache mechanics and the exact memory formula, why decode is bandwidth-bound, and how temperature/top-k/top-p shape the output — with real per-token byte counts.

• →Efficient Transformer Serving
• →Tokens and Generation

The production techniques that make transformers servable: MQA/GQA (cut KV bytes at the source), FlashAttention 1/2/3 (IO-aware exact attention), sliding-window/sparse attention, PagedAttention + continuous batching, and quantization. Real measured impact, the tradeoffs, and how vLLM/TensorRT-LLM/SGLang differ.

• →Efficient Transformer Serving
• →Transformer Architecture

Assemble the whole track into one exercise: trace tensor shapes and cost through a mini-GPT from token ids to logits, then size a real serving deployment — KV-cache memory, what fits on which GPU, and the levers to hit a latency/throughput target. The canonical LLM-engineer internals interview, rehearsed end to end.

• →Transformer Architecture
• →Efficient Transformer Serving