Why do transformers need positional embeddings at all?

Self-attention is permutation-equivariant: it treats the input as a set of token vectors with no built-in notion of order. Without positional information, "the dog bit the man" and "the man bit the dog" produce identical internal representations, only differing in the final projection of the [CLS]-style pooled output. Adding positional embeddings breaks that symmetry by injecting a position-dependent term into Q,K or directly into token vectors, so the model can encode word order and relative distance. Bag-of-words style approaches work for some tasks but lose the ability to model long-range order-sensitive syntax and coreference. If you removed positional embeddings entirely you'd see the model collapse to set-based reasoning: useful for retrieval-style scoring, useless for translation or code. Modern alternatives include RoPE, ALiBi, NoPE (no positional encoding at all, works for some tasks once attention sinks are learned).