“Next-token predictor” is the wrong mental model for LLMs

Both systems have emergent behavior that goes well beyond what would naively be expected.

I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.

When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.

It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.

But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.

It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.

> but at least I would still call it a "next token predictor"

We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.

If you're saying it's predicting, then each result should be falsifiable.

The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.

A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.

There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.

Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.

Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.

It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).

What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.

So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.

And I think effectively the same distinction applies here.

I disagree that this accurately describes TFA.

Thank you.

1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.

2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.

3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.

Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.

To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?

I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.

It's not minimising to describe even a complex prediction process as prediction.

For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?

Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge

Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor

The deep realization is that if you can predict the next token well enough, you can do things like this:

<paste the first 10 chapters of a mystery novel>. And it turned out the killer was

And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.

I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.

That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.

The technology will be relegated to the trash bin of history, just like crypto.

The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.

The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.

But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.

Recursively invoked.

With carefully selected context.

And massive investment in RL to tune token selection.

And the ability to use cli tools on other folks' machines.

That's a powerful system built around a conceptually simple technology: Next token predictors.

It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.

The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).

FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.

This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?

That seems more outrageous a claim than the one you're dismissing.

Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)

No amount of cope and anthropomorphizing is gonna change that cold, hard fact.

P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.

Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function.

> Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas.

https://www.quantamagazine.org/the-new-math-of-how-large-sca...

None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.

There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.

The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.

More concretely, in reinforcement learning there are no ground-truth next tokens to predict.

In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.

But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.

Does it complete the sentence?

If not then it’s not a next token predictor. Or at least not a good one.

It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.

That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.

If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.

During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":

https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK

(Start at 15:30 for the LLM specific parts)

It's not correct or not, it's a gradient based on the reward signal.

For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"

It would respond with something like:

"The quick fox jumps over the lazy, brown [next word]"

(Assume all the words were previously guessed correctly at this point)

The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:

1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)

The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.

Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.

However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.

My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.

I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.

Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).

Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.

People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.

I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.

RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.

A base model is predicting training sample continuations (copying).

A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.

The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.