Latent Lattice Language Models
E-Reverance
45 points
5 comments
November 17, 2026
Related Discussions
Found 5 related stories in 179.3ms across 13,334 title embeddings via pgvector HNSW
- Context Language Models emersonmacro · 133 pts · October 01, 2026 · 62% similar
- Memorilla: Latent Semantic Memory for LLMs handfuloflight · 101 pts · November 06, 2026 · 58% similar
- How to build a diffusion language model volodia · 45 pts · August 30, 2026 · 56% similar
- Show HN: Language Model Builder (an app to learn about and build models) felixrieseberg · 14 pts · July 21, 2026 · 56% similar
- Clojure in the age of language models silcoon · 36 pts · October 08, 2026 · 56% similar
Discussion Highlights (5 comments)
augment_me
Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens. It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state. It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark. In my head this should fail because the EMA cannot carry this granular information
royal__
How significant is a .98 decrease in perplexity. I'm a researcher in this field and I'm having trouble interpreting the implication of this.
tkslucas
Interesting approach to scale compute, never thought of using the per layer state like that
bionhoward
I am so excited for RNNs to be cool again
amitport
Isn't that "Infini-attention"? https://arxiv.org/abs/2404.07143 (2024)