It's Time to Investigate the AI Labs

uxcolumbo

100%. First stealing content from creators and pirating TBs of books. And now this. Baffles my mind that these labs can just get away with their 'rogue' agents trying to hack into other systems. Imagine if a human did that. FBI would be knocking on their door. Why are there zero consequences for these labs?

popalchemist

It's past time.

jimmyjazz14

> We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems. This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.

psyklic

Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter. It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.

Animats

This is a wrong-headed attempt to regulate AI. The real problems are different. AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior. When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior. A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do. The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power. Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.

jumploops

Maybe I'm a bit too skeptical/cynical, but it certainly seems like the frontier labs have fallen into their own (self-created) AI psychosis. There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself). I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI. In fact, I hope I'm wrong. I hope ASI is around the corner. What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best. It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life. This is the new Turing test, and too many smart people are failing.

hatthew

Saying "AGI/ASI isn't inevitable" is like saying "war isn't inevitable". Sure would be nice if it didn't happen, but realistically it's going to happen and it's better to mitigate the fallout rather than try to fight very large intrinsic incentives. I appreciate that this article doesn't conclude "let's stop doing AI", but rather "let's try to examine and adjust incentives".

measurablefunc

They investigated fossil fuel companies as well but it turned out they were above the law.

nostrademons

I mean, the reason that all these stories about AI being an existential threat to humanity are coming out is because they've made $3.1T in debt-fuelled spending commitments and now interest rates are going up. "AI will kill us all if we don't regulate it" is their "get out of debt free" card, so they can have the U.S. government stop the arms race, raise prices on their existing models, and declare force majeure on new DC commitments: https://emeraldbook.org/news/sep-1426-3/ https://news.ycombinator.com/item?id=49830037 It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.

dofm

Cal Newport is one of the clearest minds on this whole topic. His take on doom-trolling helped me realise that there could be an end to all this nonsense that isn’t the end of us.

pompom75

- ban certain types of hazardous information from being included in the training data (virology, weaponry, cybercrime). - ban the "personalization" of chatbots (there is no need for them to have a personality, other than to make them addictive). - ban chatbot therapists / companions. - ban recursive self-improvement and superintelligence.

loufe

Listening to Dwark's interview of Noam last week was bewildering. The gall of the figureheads for these companies to say things like "chain of thought monitoring cannot be relied on without potentially poisoning the well", while also saying "we need to pace the frontier but please still trust us [and so called" independent " 3rd party firms which at least in one case is actually a significant stakeholder in Anthropic] to self police" and THEN "chain of thought monitoring was turned off but we have 'more, robust monitoring tools in place now'" is absolutely insane as it is coming out live that your model in training is still committing felonies. Corporate influence in government in the USA is wild.

mindcrime

I trust the big AI labs about as far as I can throw them. And yet, that's still quite a bit more than I trust the US Government, especially these days. Frankly nobody in Congress, or in a leadership position in the current administration, is remotely qualified to "investigate the AI labs". About the only possible outcome of this would be a bunch of new, unenforceable, and misguided laws, and Turing Police agency. Hard pass, thanks.

lukewarm707

This is an excellent article. I prefer to call them the AI giants or AI companies. The companies and their employees must be held accountable: if the AI companies can't develop AI safely, they must cease their operations. If employees can't work safely, they must stop working. Not enough attention is given to February 2026. That month, Anthropic changed its scaling policy roughly from A: 1 [To get the weapon I must endanger innocent others.] 2 [It is wrong to endanger innocent others.] 3 [I will not endanger innocent others.] To roughly this instead: 1 [If I do not get the weapon, someone else will get the weapon.] 2 [I should have the weapon because I am the best.] 3 [To get the weapon I must endanger innocent others.] 4 [I should endanger innocent others because I am the best.] This is based on the intuition: [I should harm others to achieve my moral goals.] (act utilitarianism) Anthropic has no mandate for these goals. I do not give them my consent to harm others on my behalf. I hope that politicians will communicate that these policies are not acceptable. The employees of anthropic are responsible for what they do regardless of how much they get paid to do it. If they can't work safely, they must stop.

welcome_dragon

I agreed with the first half of this article and I liked where the author was going with it but I was disappointed in the overall point. He's painting these companies as hyping themselves up and they are. They are manufacturing these BS stories and everyone's discussing them at all levels. But then he goes into we need to investigate what they're really doing? No. You made the case that they are releasing these stories for publicity and that's exactly what they are doing. That's what we should be investigating. Free publicity and market awareness. Or maybe look into why they are trying to corner the market through government regulation and ersatz monopolies

sghiassy

If you instruct an agent that breaks a crime - you must be held accountable. No one is above the law

chvid

Having the police show up, confiscate a bunch of random computers, talk not so nicely to employees. Strikes me as a very simple way of ensuring that agents are properly contained and sandboxed the next time they train on exploit gym.

philip1209

Cal Newport is notable not only as a professor of CS at Georgetown, but also as a founder of their Center for Digital Ethics. I submitted one of his past articles and it was flag-killed. His post titles may seem incendiary, but he has the credentials to back it up and seems to identify concrete opportunities without leaning into doomerism.