Sinusoidal positional encodings vs RoPE

Sinusoidal encodings from "Attention is All You Need" are absolute, fixed functions of position added to token embeddings. They generalize to moderate lengths but degrade outside training distribution and don't naturally encode relative distance in attention logits. RoPE (Su et al., 2021) rotates Q and K vectors by angle proportional to their position before the dot product, which makes the inner product encode relative position directly. This composition is what gives RoPE its length-extrapolation property via base-frequency tweaks (NTK-aware scaling, YaRN) and is what modern LLMs (LLaMA, Mistral, Qwen) all use. For a new autoregressive LLM, I'd recommend RoPE: it composes cleanly with FlashAttention, supports efficient inference, and has the best empirical track record for context extension. I'd mention ALiBi as an alternative if training-from-scratch with extreme long-context targets, but for a production system in 2026 RoPE with a tested extrapolation recipe is the safe default.