Continuous Diffusion Language Models (CDLM's)

amelius

"Attention is all you need" should be renamed into "Attention is sufficient but not necessary".

NickNaraghi

I wonder if we’ll get something like CDLMs for automated harness engineering, sort of piloting the LLM underneath.

janalsncm

> [in 2020/2021] the dominance of autoregression was not as well-established as it is today: GPT-3 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022 I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked. (The only exception I will make is encoder-decoder models which now are often done by decoder-only.) But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass. Before ChatGPT there really wasn’t much of a concept of pre-training and post-training. It was all pre-training. Post training was what made the bots conversational and not just “continuing the thing you wrote to them”. So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.

p1esk

It’s refreshing to read something not AI generated.

ovin_dal

Diffusion models for language felt inevitable. Imagine the creative potential once these mature beyond current limits.

vatsachak

I feel like there is still low hanging fruit on the auto regressive LLMs; the encoder

Marchant_hq

CDLMs sound promising for smoother, more coherent text generation. Excited to see how they tackle the token-level discontinuities.

ainch

A great read - as with all of Sander's diffusion posts.

2001zhaozhao

I would love to see models that can think at different rates and also output a thinking scratchpad alongside output text instead of before all output. Right now models need to rely on less legible compressed CoT to get high intelligence per token/step, but with diffusion they would just need to output more tokens per step instead.