Astra can read and write weights to improve Qwen

jstummbillig

That seems both entirely obvious in hindsight and potentially super important. I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

famouswaffles

This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

ForHackernews

Possibly related to https://already-clean.goodfire.com/

lifty

Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.

vessenes

This is insane. Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x. I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions. No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

kristiandupont

I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster

Jtarii

Slightly terrifying.

jefb

This is shocking.

Starlevel004

I cannot say this fills me with confidence about alignment.

ernubufiei

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails. Absolutely incredible regardless.

6thbit

Wow, in a way it's like CRISPR for ML models!

gunalx

How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?

_boffin_

This is straight black magic. llama 70b, let’s see what we can push you to

vkgcr

This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...

mynti

I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising

rpcope1

So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?

porridgeraisin

a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today. Edit: tweets from one of the coauthor providing more context: https://x.com/a1zhang/status/2117328657161322810 https://x.com/a1zhang/status/2117330975122235442

skybrian

Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable? (When they say "Writing a Go algorithm," they mean the board game, not the language.)

adeptima

Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase

kingstnap

Friendship ended with AdamW. Astra take the wheel. But for real this is insane. Okay I need to get on this stat.