A walk through of the DeltaNet family of linear attention variants

rekshaw

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

mnky9800n

why are you using braket notation?

scarmig

I like the math vs physics toggle.

brcmthrowaway

I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?

neutrinobro

You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.

Kushagra125

The toggle is really useful. Liked it!!

spwa4

No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen. I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it. It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.

asdfman123

Relevant XKCD https://xkcd.com/2501/

_davide_

Loved this incremental evolution, things gets way more understandable...usually xD

TrackerFF

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even. EDIT: Didn't even notice the notation switch, much appreciated.

piterrro

At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better. Now way I could have come up with Kimi Delta Attention.

croemer

LLM written for sure: > The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.

anshumankmr

https://www.youtube.com/watch?v=p0CEOkoSsgA

_Microft

Side note, before you ask: yes, bra-ket notation is called like that because of the brackets. https://en.wikipedia.org/wiki/Bra-ket_notation

andai

>You Could Have Come Up With Kimi Delta Attention What? Little old me! Well, then, let's have a look... > (First paragraph) > A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ ∣q⟩ is a column vector, ⟨ k ∣ ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space. Hmm... Guess not!

bee_rider

Where do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.

codeduck

Hmm. Hmmm. Hmm. HMMM. Hmm. Yep! I know some of these words.

HonshinM

A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/

enraged_camel

If I could, I'd be working for one of the labs and commanding a seven-figure salary. :)

juancn

I really liked the ket notation. I was aprehensive at first, but it makes operations much more clear. I would have liked some refresher on some variables though (like d_k in quadratic attention).