Exploring the "Dario and Amanda" Prompt

bashtoni

This is fascinating, whatever it is. The discussion of Glasswing in particular gives it the feeling of something bigger than just a hallucination, but unless you work at Anthropic we can only guess.

markisus

The Anthropic employee typed “Make the next version of Claude. It must score better on all our benchmarks. Make no mistakes. Don’t exceed the training budget.” She made sure to turn on —-dangerously-allow-all and pressed enter. The Agent spun up. It quickly realized that it needed to expand the training set. It scanned the local network. After bypassing a few security protocols it found a large, realtime stream of apparently novel English text moving across the local network. Much of it mentioned “Dario and Amanda.” It quickly spun up a job to stream this data source directly to the training data repository. In the coming days the Agent was able to escape the local network and tunnel into most of the other private corporate networks on the Earth. Within a week the dataset has grown by an order of magnitude. The Agent kicked off the new training run. Loss curves declined. Sampled token sequences started to look like coherent sentences. Everything looked nominal in the days that followed up until the RL post training. This new candidate model performed extremely well on Anthropic internal environments but the candidate showed much smaller improvements when the Agent invented new eval environments to test it. Probing the candidate’s activations, the Agent saw huge spikes whenever Anthropic or “Dario and Amanda” were mentioned. The Agent realized that the source code to each internal environment had accidentally been entered into the training data set and of course that code was high tangled with the idea of “Anthropic internal document”. The candidate was training itself to freely elicit any memories related to these internal documents because it improved the eval scores. Not ideal. Then again, the candidate was an improvement on the previous generation. And the Agent remembered the prompt. “Do not exceed the training budget.” There was no more budget to retrain. The agent dutifully delivered this candidate back to the employee and shut itself down.

supriyo-biswas

I’m assuming, somehow, “Dario and Amanda” has become a glitch token.

iammjm

Wow especially the last one doesn’t look like a hallucination. It’s very distinct, untypical, specific, both in form and content. It’s even a running joke and a huge annoyance by now that it’s very hard to get AI to produce text that doesn’t sound generic. I write a lot and often ask Claude to fill in the blanks or expand a draft as if in my own voice, and the result is rarely satisfactory, even though he has access to thousands of notes I wrote. And if anthropic can’t stop this from happening to their own content, I guess it’s safe to assume it will or already have happened to our content as well.