You could have invented transformers from memory

craftkiller

This article does two things which make it unhelpful for me: 1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters? 2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix? To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings: > the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset. So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.

sailfast

This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it. I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!

andai

See also: NNCP: Lossless Data Compression with Neural Networks (2019) https://news.ycombinator.com/item?id=27244004 https://bellard.org/nncp/

yoz-y

Would be nice if articles defined the requirements to be a “you”.

vatsachak

Great article. My favorite fact about "transformers as a hopfield network" is that transformers are so good at memorizing that they only need one step to find a suitable memory as opposed to multiple steps for a vanilla hopfield network

kooi

Goodness, someone posts a great article which IMO accessible math, and people get upset. If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general. I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.

dullcrisp

For anyone here who understands this stuff, help me out because the article dropped me at the end when drawing the connection from memory to LLMs. For an LLM, the keys are the tokens in your context, and the values are … what exactly?

hum3hum3

The animated bird, the kea is absolutely fabulous. I appreciate the effort and detail that went into this.

femto

"interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).

ketothekingdom

Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46

sixdimensional

This sparks a thought that I have not seen revisited in the LLM age - memristors [1]. [1] https://en.wikipedia.org/wiki/Memristor