Chomsky and the Two Cultures of Statistical Learning (2011)

Actually, there have been a number of rounds of paradigm shift (or attempted paradigm shift) in linguistics over the past 70 years. The unique thing about linguistics is that they've been compressed into the lifetime of the same central figure.

Also, I'll mention that there is a lot of empirical knowledge about natural language syntax in generative grammar, even if the details aren't quite right about their theoretical underpinnings. And a lot of them were "low-hanging fruit" waiting for someone to point out how to find them and characterize them. That Chomsky did that is the best argument for his intellectual value - permitting all that "normal science" to happen.

Science doesn't guarantee technological applications, whether or not that science is correct.

Finally, just saying: use of a word like "Chomskyism" makes you sound like one of those creationists who call evolution "Darwinism".

> You can pose and answer questions that make sense within in the system.

Yes this is one of my critiques of the academic system (I have finished a PhD and will hopefully never work at a university ever again). Many papers in the introduction start with explaining how the study builds on plenty of previous research, but that alone doesn’t prove that something is a good direction to investigate. If academics were completely honest, many papers would write “We researched topic X because we got funding for it. But since we actually don’t care about X, we just wrote a proposal about our favorite niche topic that only one of the co-authors cares about, namely topic Y, and wrote it such that it sounds like X. Next we show that Y is actually very cool in the hope to convince others that it is cool plus to convince others of the conclusion that our most senior professor already found during the PhD which was 30 years ago. Apart from the professor, we don’t believe in the main argument of the paper but since he got the grant we’ll just go with it in the hope to one day be able to push our own ideas forward”.

Indeed. Who'd have thought that mere statistics could do that? Hat's off to them - they made the darn thing work.

There's pretty interesting (I think) research in how effectively it's possible to scale down LLMs. If you're interested, take a look at the BabyLM conferences:

https://babylm.github.io/

(I have no connection to them.) The work there demonstrates that gap between what we can do now with human-relevant amounts of training text, and what remains to be discovered. Quite a bit, it looks like.

> stochastic generation of language yields a form of recognizable intelligence.

Can you use a human language well enough and it not sound intelligent? And no, I don't mean just syntactically correct because semantics is a part of human languages too. I think the astounding thing is that they managed to model a language this well. It's like trying to learn English without a functional understanding of the word "Please".

My take is that a lot of the intelligence we natively ascribe to humans exist in human languages and just like the nutrients in your food, by harnessing a language that possesses latent intelligence, you too can behave like an intelligent person. And how did the nutrients of intellect got in there in the first place? Just like languages evolve nowadays: via contributions from speakers over time. Not unlike soil fertilisation.

The existence of LLMs is telling us something, surely it means something. I'm not sure we've quite figured it out yet but its probably something along the lines of 'intelligence might not be as complicated as we thought' ... a fourth narcissistic blow to humanity after Copernicus, Darwin and Freud.

> but from all models, it's the closest

Care to back this up? What evidence do we have to suggest that back propagation, self attention, and byte pair encoding are mechanistically similar to human cognition? Similarity of the outputs says nothing about similarity of the machinery underneath. Turboprop engines tell us nothing about how birds fly - doesn't matter that they successfully transport millions of people around the world.

I would say that when it comes to higher level cognition, logic engines are more human like even if they only have a tiny fraction of the practical utility of LLMs.

LLMs behave like they're closest, but only during execution. During training, obviously, they don't resemble the human brain at all. No human reads a million books before they begin to speak.

And I'd be hard pressed to identify many cognitive characteristics of LLMs during execution that resemble human thinking, aside from those that are readily parrotable.

As a model of human cognition, LLMs deep suck.

What reason to we have to believe that LLMs are the closest model we have for how humans learn language.

We considered the hypothesis that humans are a general intelligence that figures language out by exposure. The evidence has consistently come back that that is not how humans actually work. A model where humans are hardwired with a lot of knowledge about language, then fill in the gaps based on exposure still appears to be far more accurate.

The only thing LLMs show is that it is possible to learn language de-novo.

If you are counting by bytes and including all sensory input, sure. In terms of amount of text, it's not close in the other direction. It's very difficult to make an apples to apples comparison here.

The BabyLM challenge has shown that multimodal models don't seem to learn in a more data-efficient way with respect to tokens. Adding an unlimited amount of images doesn't help you learn syntactic islands faster.

That is true, but to me it seems like ternary logic is like probabilities in disguise (false, true and unknown would be probability of 0, 1, and "between 0 and 1"). For most application, the unknown class is too broad, thus we need actual probabilities. But indeed, ternary logic is simpler than working with probabilities.

To me uncertainty on a given quantity, is represented by a probability distribution function giving how probable the value of a given quantity is. Loosely, it can also be represented by a confidence interval. It is not intrinsic to the quantity, but how much we know about this quantity.

For the domain I am working in, a model is considered deterministic if for a set of input it always provide the same output and the input are assumed to be perfectly known. A model become probabilistic, once the input or the model itself are considered as uncertain. As an example: previously weather models always provided singe prediction for tomorrow, for example rain or no rain. But now, such models are probabilistic in the sense that an ensemble of model forecast are made (with slightly different input representing our uncertainties) to provide an estimate of the probabilities. Indeed such probabilistic model is not the simplest, but to most complete and so far the only practical way to handle with uncertainty for system like the weather as far as I am aware of.

I haven't seen any allegation that Chomsky raped anyone, let alone kids. But Chomsky was quite close to Epstein and some of their leaked conversations tarnish his reputation quite a bit, irreversibly IMO.

I agree there is no proof, but merely calling it misjudged is too forgiving. I have had a lot of respect for Chomsky and his moral stance in life, so this relationship to Epstein was gravely disappointing. He tried to justify it a long time ago by saying Epstein was out of prison and had paid his dues, and that it was intellectual curiosity or something like that, and on paper I agree with this, but it was clear Epstein got off easy and in hindsight it should've been clear to the people around him he had never stopped. I don't think Chomsky raped anyone for what it's worth. But it's a big stain on his legacy.

There's also communications of his where he talks with Epstein about the allegations being ridiculous. Obviously, they weren't, and Epstein knew that, but I think the odds are good that Chomsky actually did think they were fabricated. Epstein knew so many people, it's highly unlikely that he would start chatting about child sexual abuse with everyone. Chomsky probably didn't get that version of Epstein - he just got the Epstein who liked to splash money around, help out famous friends with introductions, etc.

Chomsky is also somewhat unique in that he would communicate with literally anyone who wanted to talk to him. I remember seeing Manufacturing Consent for the first time in 9th grade in 1999, and I sent him an email, and he responded within the hour.

His Wikipedia article [0] gives a good summary of the connection they had.

[0] https://en.wikipedia.org/wiki/Noam_Chomsky#Friendship_with_J...