Attention Is Not What You Need (2025)

For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).

For a given t, this produces up to m pairs:

(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….

We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.

So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.

Compute, data, and parameter count are the three logarithmic values that correspond to model performance.

So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

If you find this fascinating and want to dive more, I encourage checking out nanochat (solid GPT-2 repro; minimal LoC); and trying to make architectural tweaks with a coding agent.

“Learning through experimentation” has never been easier or more fun in my opinion. If you have a GPU, even an older 8GB card (or a few bucks for Runpod), you can pretrain your own small language model and start experimenting / tweaking it.

The neocortex would be akin to something like a 100T neural network, and that’s with the maybe overly reductionist assumption that only synapses are doing anything when there’s a lot of biology happening in there. Every cell is filled with its own computation and state. It could be closer to a 10E model if all that is in play.

The neocortex is nowhere near the whole brain.

Architecturally the brain could be described as a slow but ultra-wide machine with colocated memory and thus an effectively enormous memory bandwidth. That lets it evaluate its whole parameter set continuously.

I keep thinking the brain must be algorithmically doing something super clever but maybe not. Maybe old mother nature just found the “bitter lesson.” Maybe it’s algorithmically inefficient and messy but there’s just a whole lot of it.

(?) We actually don't know that but it is very likely.

human intuition is supported by embodied experience, perception, and years of interacting with the physical and social world. An AI model's learned pattern recognition is analogous in some respects, but it isn't the same process.

intelligence may depend less on continuously scrutinizing everything than on having a good internal model, recognizing when that model is likely to fail, and knowing where to investigate.

It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).

Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.

Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.

We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.

In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.

Retractions are non-existent, even with clearly proven academic malpractice.

It's his/her prerogative to keep an online footprint or not.

Agreed.

I say: “if you want a human response, then give human effort”