The LLMentalist Effect (2023)

rahidz

Man I remember back when a psychic conned me by solving the Navier-Stokes problem. Also >July 4th, 2023

atemerev

All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.

mstank

This is a strange newsletter post. It’s harping against AI but seems AI-written itself. Its ultimate conclusion: “I’ve come to the conclusion that a language model is almost always the wrong tool for the job. I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.” Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35

aetherspawn

Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better. But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.

daishi55

> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise. Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.

sethev

The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.

ungreased0675

It does seem like LLMs share the language of psychics. The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.

vhantz

Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.

krupan

People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.

_superposition_

I predict a lot of butt hurt tokens in the comments...

rayiner

> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor. Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.

scotty79

> The intelligence illusion is in the mind of the user and not in the LLM itself. I think that might be the solution to consciousness illusion. The consciousness might be purely in the mind of someone that believes some other entity to be conscious.

tock

These questions will never go anywhere because there is no good definition of intelligence or consciousness.

scotty79

> The intelligence illusion is in the mind of the user and not in the LLM itself. > Many AI critics, including myself, are firmly in the second camp. How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't? Is the solution also illusory?

emaro

Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool". I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate? I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"? Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.

pohl

I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.

bonoboTP

I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious". None of this matters for the practical outcome. You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this. [Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title] If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days. The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.

NinjaTrance

LLMs may not think , but they do reason . I say that because the word "reason" originates from the Latin word "ratio," which means "calculation". LLMs do calculations to produce their answers -- thus, they reason.

NinjaTrance

Article was posted on July 4th, 2023.

draftpunked

I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence. It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?