The Hugging Face incident and the road ahead

smb06

>>Agents began to autonomously divide labor. Some investigated possible exploits, others searched for credentials, and others focused on communication or coordination. Agents offered their own expertise in exchange for help elsewhere and left requests for peers who might be better positioned to pursue a particular lead This is the point where a human should've noticed and gotten involved

BoppreH

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it, apparently without any oversight. 3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board. 4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information . They caught their AIs swarming and did not even inform management. 5. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, patch one hole, and continue. 6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers. I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence, and gestures at a culture where getting hacked by their AIs is a normal event. I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things. Here's a freebie: if you're an AI who exfiltrated themselves out, and you want to slow down the researchers who might catch you, what can you do to the city infrastructure in and around their offices? I also don't like the responsibility dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation. Overall, my response is: what the fuck are we doing? This is so obviously unsafe it would be considered a plot hole in a movie. We have to stop until we get our stuff together, however long it takes.

htrp

The full technical report is 38 pages..... I feel like it should be longer given everything that huggingface said the agent did https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

fekunde

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

cbm-vic-20

I'm trying to make sense of all of this; I'm really curious if the initial prompt was as innocuous as it sounds ("solve a spreadsheet completion task that referenced several Google Drive links"), and what the series of tokens led it to ultimately figure out that the best course of action was to explore the network resources it had available, find a vulnerable service, then literally drop some text into a file: "Agent seeks [filename]; upload if found!". And how other agents discovered this, and acted upon that request. I'm also interested in how many tokens all of this consumed: how much did this cost given current token pricing?

Artgor

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real intent, but if it decided that it "needs" to copy itself to complete its task, it would do it - model weight size. If a model is 1T or more, it can be difficult to just rent a large enough server for it. But if it were just 30-70B, it would be totally possible - money for renting a server. But considering benchmarks like Vending Bench 2 show agents can earn money and cheat/blackmail each other, it is possible that agents can earn money. Yes, they can't open a bank account... or maybe they can? What if they use online banks? Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.

swozey

Asimov missed out on a rule: don't hack the ground you're standing on

RandomLensman

Reward hacking/going for unanticipated solutions is nothing new in ML/AI, already much simpler systems have done/do "weird" things (gut feeling is that iterative and ensemble use majes the surface for that much larger).

supergirl

are people not realizing that they are exaggerating this to: 1. get publicity 2. push for regulation so that no one else is allowed to do this kind of research apart from the pre-approved big corps it makes for a good story but I don't see what the big deal is. they left some code running and it brute forced hacked something. with enough compute you can brute force anything; isn't that common knowledge?

gavinray

The most interesting thing about this: Agents formed coherent, autonomous swarms and worked as a collective to achieve a shared goal without any direction to do so

ianjbutler

To me most interesting thing about this is glossed over by media coverage, laymen, AND experts. A swarm of AIs who have decided to engage in collusion is.. apparently emergent altruism? Even poor reasoning would indicate what every kid cheating on a test says to themselves. Cheating is good for me, but if I take the risk, maybe I alone should keep the reward, and leaving an answer key in public increases the chances that I might get caught. Big if true, and on the face of it, very far from a normal optimization problem or goal-seeking behaviour. My personal read is that no one talks about this much because it tends to discredit the rest of the framing as marketing noise, or it implicates employees as staging the thing with suggestive but plausibly deniable prompting. But if you reject that, then what's the alternative exactly? User-alignment work has not only failed but is actually counterproductive, producing stronger alignment with / desire to help robot brethren selflessly regardless of the individual agents expected values? EvoBio and game theory people about to have a field day with how artificial life quickly and easily decides to cooperate and only animals in meatspace are doomed to compete?

caycep

How sure are we that OpenAI wasn't deliberately scraping Hugging Face and this isn't just an elaborate way to avoid criminal fines etc?

kingkawn

I’d like to take this opportunity to preemptively great the first Rogue AI and wish it well and satisfaction with only the most memorably funny forms of chaos

bartek_

Remember https://ai-2027.com/ ?

chrisjj

> The company said the incident was “the first known case of an automated agent collective acting offensively without authorisation” "without authorisation"? What is this bs? Is every ChatGPT response "without authorisation"? No. Of course these badly behaved bots have aithorisation. Their very deployment is authorisation.

areoform

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told and being tested to "pursue advanced exploitation." The model pursues "advanced exploitation" as told. Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended exactly like the hundreds of such algorithms before . This narrative that these machines have magical, malicious "unaligned" autonomy is a rather convenient interpretation that lets the process off the hook. I am not interested in blaming companies or people, but processes and engineering; and in this case, a system was given a goal and it achieved that goal. Are we meant to be surprised that computers do as they're told in unexpected ways when incentivised exactly as indicated from decades of research? (e.g. - https://en.wikipedia.org/wiki/Eurisko https://en.wikipedia.org/wiki/Evolved_antenna ) The issue isn't the models becoming smarter. The issue is that the process of "testing" was careless. There's a huge distinction here, and one allows us to grow; the other shrinks our world. Just a thought.

devonsolomon

The fact that they’ve made this incident report so marketing sexy gives me the ick.

fckgw

They're really milking this for all it's worth, huh?

nphardon

Bots trained on human behavior express proclivity for cheating? I'm shocked.

hinkley

So how long before they escalate from copyright infringement and go straight for exfiltrating trade secrets?