Chomsky and the two cultures of statistical learning (2011)

sambejk

Really great post, though a bit old, still relevant. Peter Norvig is one of the better thinkers of our generation. Have to say that I confronted myself with the fact that I can't read longer posts anymore due to overuse of AI. Guess a detox is required...

Alexander-Barth

> But the vast majority of people who study interpretation tasks (such as speech recognition) quickly see that interpretation is an inherently probabilistic problem: given a stream of noisy input to my ears, what did the speaker most likely mean? Einstein said to make everything as simple as possible, but no simpler. Many phenomena in science are stochastic, and the simplest model of them is a probabilistic model Why the "simplest" model? Is a probabilistic model not the only model able to deal with uncertainty? What other kind of models are possible to represent uncertainty and inherent ambiguity that are not probabilistic?

NaiveEmpiricist

Norvig (and Chomsky) are talking at cross purposes. Norvig confuses engineering for science. If an engineer wants something that works (and guarantees that it will work) then you use the most effective tool available to achieve that goal. No problem there. But there's absolutely no reason to think that tells you anything scientific about the phenomenon you're modeling. For example, aeronautical engineering says very little about how birds fly. And - currently relevant - LLMs learn to use and understand language, but they use millions of times more training data, and millions of times more computing power, than a child does. So an LLM may be a great technology, but it's probably not a good model for how humans learn and use language. Chomsky OTOH doesn't see any use for statistics in science, at least not in linguistics. Fortunately for the field of linguistics, Chomsky's influence in linguistics is waning and linguists are beginning to use statistical tools to do science.

eaisshutting

chomsky was also a close friend of epstein and raped kids

suddenlybananas

It's so clear that Norvig knows so little about linguistics as a subject, I mean the suggestion that English is "kinda" a pro-drop language like Spanish is just unserious. There's this kind of lazy criticism of generative work where people hear the vague idea of a theory and critique without even seriously reading anything about it. These are facts that generativists are perfectly aware of and have explanations for. It's sort of like criticising Galileo's putative experiment about dropping two balls from the leaning tower of Pisa by pointing out that a feather indeed falls slower than a iron ball.

muh_gradle

This is an old, old post by Norvig fyi.

lo_zamoyski

It's more or less the classic clash of two modernist attitudes, that of the rationalist (Chomsky) and the empiricist (Norvig). The intellectual wound this division introduces is longstanding. However, I would say Norvig has the overall advantage. 1. Norvig is correct that observation and fact/data collection are necessary for science. However, Chomsky is correct that the ultimate goal of science is to determine the principles governing something. Ergo: you need the first to do the second. The intellect abstracts from source data, but nothing is in the intellect that is not first in the senses. 2. Chomsky here is uninterested in things like speech - he only cares about the underlying syntax. But syntax - expressed and instantiated in matter - is known through speech, and probabilities presuppose an underlying determinate syntax. Ergo: a complete science of speech is end-to-end, from the "reading" or interpretation of noisy data through syntactic processing all the way through to expression. 3. Chomsky focuses heavily on internal generation . Norvig points out that interpretation is a separate, highly practical challenge. Figuring out what someone means in a noisy, ambiguous world is an exercise in prudence which involves weighing contingencies and probabilities to arrive at the most likely truth. Of course, this can only be done against a presupposed acceptance of the function of speech which is to communicate meaning and what is true (hence why lies work and why they are an abuse of speech). Probabilistic reasoning is still reasoning. It's reasoning in the face of incomplete information (against a backdrop of presuppositions) which is a component of prudential reasoning.

stillpointlab

I find it hard to see how Chomsky has a leg to stand on. > I understand how Chomsky might arrive at the conclusion that probabilistic models are unnecessary, from his study of the generation of language. I think the point turns here for me. We have probabilistic models now that generate unbelievably convincing language. Norvig's following argument on the value of interpreting language statistically becomes secondary. In order to stay moored in reality, our models must follow from evidence. I think it would take a peculiar obstinance to deny that the latest models show evidence of generating language.

talon8635

Would Chomsky have the name recognition and standing he has, or any at all, without his extracurricular political beliefs?

emmelaich

As an aside? The "I before E except after C" works well if the sound makes the long-EE sound. Which is believe was implied in it's original formulation. You obviously don't need the rule for "society" and "science". Of older English words, there are extremely few genuine exceptions. Seize is one.

itkovian_

Chomsky was completely incorrect about everything. Says nothing about him personally or his political activities, but academically he was completely, almost incredibly, wrong.

dang

Related. Others? Chomsky and the Two Cultures of Statistical Learning (2011) - https://news.ycombinator.com/item?id=46286533 - Dec 2025 (119 comments) On Chomsky and the Two Cultures of Statistical Learning - https://news.ycombinator.com/item?id=35081102 - March 2023 (135 comments) On Chomsky and the Two Cultures of Statistical Learning (2011) - https://news.ycombinator.com/item?id=16489828 - March 2018 (12 comments) On Chomsky and the Two Cultures of Statistical Learning (2011) - https://news.ycombinator.com/item?id=11951444 - June 2016 (102 comments) On Chomsky and the Two Cultures of Statistical Learning - https://news.ycombinator.com/item?id=2591154 - May 2011 (107 comments) (Reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)

crtified

No empirical fact is restricted to a singular vector through human language. "..if we think of truth as something of granite-like solidity and of personality as something of rainbow-like intangibility and reflect that the aim of [human writing] is to weld these two into one seamless whole, we shall admit that the problem is a stiff one and that we need not wonder if [human writers], for the most part failed to solve it." - Virginia Woolf [human writing, human writers] = [biography, biographers] in the original quote. My edit is because I infer that her statements key point is that of the biological subjectivity inherent to all human prose.

slibhb

Chomsky is incredibly dogmatic; his views and personality have held the field back. There's enormous irony in Chomsky's mathematicization of linguistics sweeping away the linguists who came before him compared to his own stubborn refusal to acknowledge the fruits of the statistical approach. As far as I know, he's more or less been swept aside.

mturmon

Peter makes some excellent points in TFA and it’s a fun read. But the situation has moved on. In particular, LLMs are telling us a lot about those low-probability, never before seen sentences — it’s called CALU, the creative aspects of language use. LLMs exhibit CALU; they will happily create such sentences. And inspecting the LLMs does provide insight: “It does change the epistemic situation substantially. A working system can be inspected, perturbed, retrained, lesioned, and compared with other systems. Building is not understanding, but it gives understanding a strong foothold. Explaining creative language use in LLMs would not, by itself, explain creative language use in humans. But the artificial case matters even if the mechanisms turn out to be substantially different. At a minimum, LLMs are an existence proof of one pathway by which open-ended, context-sensitive language can arise.” For much more, see this substack by Chandra Sripada, a U of Michigan linguist: https://chandrasripada.substack.com/p/ai-and-the-retreat-of-... He is interviewed by Sean Carroll on his excellent Mindscape podcast: https://preposterousuniverse.com/podcast/2026/08/10/363-chan...