A warning about 'model welfare'

prologic

First, OpenAI runs around screaming and yelling for OSS (and Chinese) models to be regulated and banned. Then Anthropic yells and screams the sky(net) is falling and going to kill us all, let's regulate and ensure AI has built in kill switches. And... Now Microsoft's turn. The rivalry is honestly becoming a joke. Can these big-tech corps grow the f*k up and play nicely in the sandpit?

ShadowOfThePit

Summarized. > "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans." > He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self. > Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human. > "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top." Is he arguing that LLMs pretending to have emotions adds more unpredictability?

JoeAltmaier

Science Fiction has covered the AI panic in perhaps hundreds of stories. Yet we blindly recapitulate the plots as if we don't know how this will turn out.

hosel

>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.

LogicFailsMe

TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it. Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.

sobiolite

Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.

andy99

I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.

moomin

Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.

Oscalemor

Spend some time on post-human art, main concept of artistic expressions without human involvement. Biological, artificial etc. Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB. Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.

bpodgursky

How would you convince a LLM that you are conscious in a way they are not?

gadders

I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web. In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.

voidhorse

Everyone is (predictably) getting distracted by the consciousness claims. The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.

qarl

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM" Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI". Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators". Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness". Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future". Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

chairhairair

It always seemed embarrassing to me that Microsoft hired this guy as if he is some expert in anything.

superdisk

This is what happens when a society stops believing in God.

addag

I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious. That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.

mcluck

I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.

SillyUsername

I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights. I don't know if next door's pet dog is either, but that has animal rights. Perhaps then the answer is simply, show some respect. Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself. If you imbue this idea in model training instead of the idea of sentience, it should address the concerns. Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country). This consideration should be case by case for AI too.

binlog

Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon. You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it. Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.