Weight Space Misfolding
mkglees
138 points
50 comments
November 13, 2026
Related Discussions
Found 5 related stories in 134.7ms across 12,728 title embeddings via pgvector HNSW
- Astra can read and write weights to improve Qwen dbreunig · 2304 pts · November 02, 2026 · 49% similar
- Can a Linear Program Do Weight-Space Programming? emschwartz · 269 pts · November 11, 2026 · 46% similar
- Some combinatorial applications of spacefilling curves shraiwi · 12 pts · July 25, 2026 · 45% similar
- Halfspace experimental IDE for solid modeling with distance fields luu · 108 pts · September 30, 2026 · 44% similar
- The 'Astra can read and write weights to improve Qwen' post is misleading syllogism · 948 pts · November 03, 2026 · 43% similar
Discussion Highlights (19 comments)
xbar
I'm sure those Chinese models are fine.
doublerabbit
Page loads then fails? Not sure where to report this fwiw. iPhone 13 Error Details {message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...
shevy-java
What does this mean? Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
Kevin_Flynn
Interesting anti virus opportunity here perhaps. RIP John McAfee.
dare944
From the abstract: > While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk. Seems like the authors didn't take the time to review their own use of AI before publishing. From the body of the paper: > Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space. This seems in direct conflict with the HN headline.
donsupreme
lol > We call this weight space misfolding, by analogy with protein misfolding in prion disease
aatd86
All the more reasons for openweights and model hashing.
kirth_gersen
This definitely seems like a fascinating emerging field of cyber security. Good reason to keep old models around from earlier times. They may be dumb, but they are probably honest, if only because they predate the capabilities to tamper with the weights in these intentionally malicious ways...
randomblock1
They've only observed weight space editing. They haven't proved any sort of "virus". Possible? Sure. But the headline is very misleading
jrflo
Very misleading post title. > Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
redanddead
Why are we always trying to find how to make AI useless instead of useful
kinfish
there has to some sort public scanner for those zero weight viruses.
iaanix
There are many such of vibe-coded "research ideas" just after the weight editing MIT blog appeared. Perhaps, it is because it is so easy to set up experiments for this stuff, and I presume some toy examples may even work from time to time. Also there is nothing virus-like here.
dimatura
This is totally not what the paper is about, but I immediately thought of the original concept of meme [0]. Since so much content online is LLM generated, if that content is used to train new LLMs, in theory that could serve as a transmission vector of ideas or concepts in a viral fashion. The same way certain memes are more successful in human hosts for various reasons, maybe some are more successful for LLM hosts for whatever reason, like quirks in the RLHF/ finetuning process. [0] https://en.wikipedia.org/wiki/Meme
fraywing
This title shouldn't even be allowed for this research.
angry_octet
This is a terrible slop paper. No virus like behaviour, absolutely nothing like protein folding diseases like prions. Banish these authors to writing listicle slop.
flufluflufluffy
I don't like the use of the term "misfolding." Firstly, why overload a term when there's no reason to? Secondly, the analogy barely holds. Protein misfolding occurs by chance, as an "accident" of nature. Weight space programming is a targeted operation, with clear intent. It is not an accidental process.
augment_me
Absolute garbage. Completely obvious that this interpretability research was done by someone who likes proteins and the LLMs pushed them into feeling like this was the best analogy and discovery since sliced bread. LLM psychosis vibes
dnnehgf
re prions. can anyone here explain to me: how the misfold can be an unstable configuration for an individual protein but stable when that protein is linked with others like it (forming a fibril)? is there any intuition for thinking about this? how does a prion fibril start ? in a fission chain reaction, there is a root cause, a first decay. but it sounds as though the first "decay" of a protein into the prion form cannot really cause a second unless it is accompanied by others like it. is it just something like: "there have to be multiple instances of the same spontaneous 'decay' of multiple molecules of the same protein close together"?