Understanding Positional Embeddings in Modern LLMs
vismit2000
225 points
27 comments
November 15, 2026
Related Discussions
Found 5 related stories in 126.2ms across 12,728 title embeddings via pgvector HNSW
- The architecture of open-weight LLMs: MoE, hybrid layers, ternary weights silasyl · 14 pts · November 05, 2026 · 55% similar
- Learning Programming in an Age of LLMs moneroloop2018 · 236 pts · September 16, 2026 · 53% similar
- Memorilla: Latent Semantic Memory for LLMs handfuloflight · 101 pts · November 06, 2026 · 53% similar
- Jev handed our LLM the wrong page a tenth as often as our embeddings Bocajmai · 12 pts · October 13, 2026 · 51% similar
- Yes I Understand How LLMs Work AndrewKemendo · 12 pts · October 12, 2026 · 50% similar
Discussion Highlights (1 comments)
programjames
This article is only about understanding the implementation of positional embeddings, but to really understand positional embeddings you have to understand the theory. Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples: 1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters. 2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant). 3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.