Don't be fooled–LLMs don't reason

rkagerer

https://archive.ph/ZyCoe

coreyh14444

Jet planes don't fly by flapping their wings...

dist-epoch

It's wild when you think about it, you don't need to reason to solve the hardest math problems that humans failed to solve for decades.

lordnacho

I thought I saw a paper recently explaining that LLMs have a global workspace. Is that not like having the internal state that he's talking about?

zer00eyz

This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought. Stop trying to compare either system to a human and look at it for what it is - A prediction engine that runs fast enough to brute force problems. In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions. In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path). There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).

waynecochran

<unbearable web page to read>

f6v

> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs. I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from? I'm especially confused about a prior statement as a scientist: > Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize). How does a set of beliefs help with reasoning? > Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another. We also often do the same as humans.

cyanydeez

I think, "contexting" is apropos. I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion. Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors. And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem. But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.

gavmor

Yes, this is true—there's no "logic" in the sense of deductive rigor. It's a wonder we animals are capable of it.

leourbina

Don't be fooled, submarines don't swim.

adverbly

Neither do humans! At least reasoning is not guaranteed. Would sure be nice though...

dataviz1000

I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.

RIMR

We can't even say what human reasoning is. Who could really say what isn't reasoning?

reliablereason

Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created. The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.

rkagerer

I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting. E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection. Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."

m3kw9

99% of the people treat it as a black box. It gives better answers, you can reverse reason and IDGAF

vmg12

People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.

commandlinefan

But neither do the vast majority of humans.

cmiles8

Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there. Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images. Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.

LordHumungous

LLMs lack the holy ghost