How concerned should we be about Astra's recurrent architecture?

Posted by yurivish 5 days ago

Counter151Comment128OpenOriginal

Comments

Comment by samrus 5 days ago

I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best

Comment by sznio 5 days ago

it is what I do to solve hard problems through.

easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.

Comment by dgellow 5 days ago

One thing about the reasoning is that models are trained to generate a chain of thoughts, but it doesn’t have to be correct, accurate, or reflect the underlying logic of the LLM. It’s the same problem we have with the output, it is something plausible, but not that reliable

Comment by anon291 4 days ago

I do the same thing in my head. There is no underlying logic to an llm. Logic is an external construct alien to human like forms of reasoning.

Comment by trhway 4 days ago

>but it doesn’t have to be correct, accurate, or ...

why we don't do GAN here, ie. second model verifying correctness/accuracy/etc. ?

Comment by naasking 5 days ago

Yes, both the output should be "milestones" of sorts, like lemmas and theorems in math. Important plateaus that serve as a launching pad to the next phase. Regurgitating every thought potentially degrades signal:noise ratio.

Comment by anon291 4 days ago

The hidden states of the tokens likely contain more semantic information than can be extracted by the final projection into token space.

Comment by bee_rider 4 days ago

Actually, how does chain of thought work? Is the LLM actually creating the tokens and then re-reading them, or is the there still a full hidden state under the hood and then the UI just prints that projection?

Comment by sznio 4 days ago

Can't tell you what's happening in a closed model, but in case of open ones it's just a text stream, same "take all previous tokens, compute next token" mechanism applies. Thinking vs Response is just a state change like between a system message and a user message.

Closed models probably do the same thing internally. What is shown externally is different though: you get a summary of the chain of thought, not the thoughts itself. This is done to prevent distillation.

The latest look we had at a frontier chain of thought is probably in the Huggingface incident report - I haven't actually read it yet but I saw the BlackHat talk, and it included some snippets. The thoughts look like they are approaching neuralese. The words are still understandable but the grammar is weird, simplified. In comparison, Qwen 3.8 27b thinks in valid English.

Comment by someguynamedq 4 days ago

Right but it's meant to be smarter than you

Comment by big_toast 5 days ago

Chain of thought seems very specifically like an instantiation of the System 2 thinking of the System 1 and 2 thinking Daniel Kahneman popularized.

Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?

Comment by ElFitz 5 days ago

Some humans apparently only or almost only do non-verbal thinking. It’s called anendophasia.

Comment by ma2kx 4 days ago

Sounds like philosophical zombies.

https://en.wikipedia.org/wiki/Philosophical_zombie

Comment by fipar 4 days ago

“For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would”

Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.

Comment by ElFitz 4 days ago

Maybe, I wouldn’t know. I just find it fascinating that other people have such a radically different way of experiencing thought. No idea what it’s like.

Comment by winocm 2 days ago

I am one of those "humans", allegedly. ;)

Comment by 4 days ago

Comment by scotty79 4 days ago

What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.

Comment by anon291 4 days ago

Why? I talk in my head and then enunciate only that which is relevant. My speech rate inside is incredibly fast.

Comment by majormajor 4 days ago

The purpose of the scratch pad is very often to refer back to it later.

Comment by anon373839 5 days ago

Sebastian Raschka posted about this architecture:

> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".

> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.

> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."

> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.

> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.

> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.

> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)

> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.

> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.

https://x.com/rasbt/status/2095141254958858496

Comment by zormino 4 days ago

Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare

Comment by stymaar 4 days ago

Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).

To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.

Comment by fy20 4 days ago

It's the opposite kind of bundling as MoE, which effectively trades memory/storage to save on compute at inference time.

Comment by nojs 4 days ago

It’s approximately the same as Qwen3.827b’s propensity to think a lot, right?

Comment by digdugdirk 4 days ago

Not quite. Looped models do the extra "thinking" inside the model's layers. So the token gets twice the number crunching performed on it before it gets spit out. I think of it as the first loop "kickstarts" the process, and the second loop refines it.

Comment by KKKKkkkk1 4 days ago

I don't understand why he's citing a NeurIPS 2025 paper when cross-layer parameter sharing was introduced in ALBERT already in 2019.

Comment by mi_lk 4 days ago

if only tweeting is as rigorous as academic literature reviews

Comment by kjshsh123 5 days ago

>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory

I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.

You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.

RNN xhat[t+1]=f(x[t],h[t])

Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])

Comment by kjshsh123 5 days ago

On further thought, I think the author's intent was to say that classic RNNs have "unbounded temporal accumulation in the hidden state".

Comment by khalic 4 days ago

I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…

Comment by buppermint 4 days ago

CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.

Comment by khalic 4 days ago

Why does injecting random text increase the precision then?

Comment by anon291 4 days ago

The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.

Comment by markasoftware 4 days ago

Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT

Comment by khalic 4 days ago

Indeed, but the fact that it moves the needle at all is enough to be skeptical of the anthropomorphic interpretation of the intermediate tokens. It's waaaay too early into the discovery process to take anything for granted, even more when it's related to our very strong bias towards giving inanimate things human qualities.

Comment by thinking_cactus 4 days ago

How can they do that? Does state propagate between each token output?

That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?

Comment by anon291 4 days ago

I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.

In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.

Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.

Comment by simianwords 4 days ago

How can you possibly believe this? If this were true, why do open weight models think in real English tokens and not arbitrary characters?

Comment by khalic 4 days ago

I believe it because of scientific papers on the matter. It doesn't "think" that's the point, it's generating text that describes a thought process, it's related but not the same thing.

Comment by gr_norm 4 days ago

Link to this study?

Comment by khalic 4 days ago

[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

https://arxiv.org/abs/2404.15758

Comment by comex 4 days ago

The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.

Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.

Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.

Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.

Comment by imtringued 4 days ago

>The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.

Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.

Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.

The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.

If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.

Comment by comex 4 days ago

I'm not even an LLM researcher. But I believe you're correct; I just don't think the paper provides any meaningful evidence on the subject. From the structure of the model we can tell that dots can do more computation than no CoT and less than full CoT. What we don't know is how much of the performance gap between no CoT and full CoT would be recovered by an LLM using dots. The paper answers that question for a tiny transformer, but it's so different from an LLM that extrapolating to LLM scale is purely speculative. In other words it doesn't give us more information then what we can already determine from the structure of the model.

For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.

(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)

Comment by khalic 4 days ago

Scientific papers aren't popsci articles. Constraining the environment (like with the token choice) is a basic method to reduce confounding factors...

The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.

Comment by comex 4 days ago

> The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.

Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios.

If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not.

So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different.

First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself).

Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before).

The dots paper has no post-training, so it doesn’t encounter this dynamic at all.

Comment by stymaar 4 days ago

Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

https://arxiv.org/abs/2504.09762

Comment by khalic 4 days ago

Fascinating read thank you

Comment by dang 5 days ago

Related ongoing thread:

OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395

Comment by pu_pe 4 days ago

Seems conceptually connected to the "repeat yourself" hack that improves models by duplicating layers: https://dnhkng.github.io/posts/rys/

Comment by BoredomIsFun 4 days ago

Yep, an old idea, that has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.

Comment by mentalgear 4 days ago

So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"

Comment by thinking_cactus 4 days ago

I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).

Comment by GPerson 4 days ago

They’re not ever going to be conscious.

Comment by qgin 4 days ago

We can’t even prove humans are conscious, we just extend the assumption to each other because we want other people to do the same to us.

Comment by GPerson 4 days ago

That is only part of the reason. Most people believe we live in an universe with regular behaviors, such that observable phenomena, such as consciousness, depend on regular physical arrangements. This is a good reason to believe brains are all conscious. It is not a reason to believe only brains are conscious, but I don’t believe a wooden block which is painted red and white and has an iron ball glued to it has magnetism; this is just an analogy. It seems far more likely that consciousness depends on the particular material arrangement, meaning that substrate independence is wrong.

That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.

Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.

Comment by sho 4 days ago

That is a completely unsupportable claim. For all we know they might already be, for a brief flash at least.

Comment by GPerson 4 days ago

The other guy is right. They are as conscious as sand formed into any circuit board with electricity running across them. It is an insane mistake to conflate intelligence with consciousness. Substrate independence is a preposterous assumption.

Comment by sho 3 days ago

Preposterous? Insane? My friend, we don't even know what consciousness is, or from what it arises in humans. It could be an emergent property of neural-network-shaped activity - in which case LLM software running on that "sand" is certainly eligible, at least in theory.

These outraged denials you're spouting are based on nothing but emotion. You simply don't know - neither do I, neither does anyone. So stop acting like you do.

Comment by GPerson 3 days ago

We know that there is a concept corresponding to first person experience. A priori there is no reason that should be. Please think about this before dismissing it because it’s actually extremely important.

What’s emotional is explaining to adults something that should be grasped by the age of 10, but we’re all going to need to understand this very soon.

Comment by bee_rider 4 days ago

They are made of sand. For all we know the sand was conscious before we smashed it up and arranged it into computer chips.

Comment by sho 4 days ago

I'm not sure what point you are trying to make, if at all

Comment by bee_rider 4 days ago

For all we know LLMs might be conscious in the same way that any other inanimate object might be.

Comment by sho 4 days ago

Care to supply your rigorous, formal definition of what animacy in an object is?

Comment by GPerson 4 days ago

You’re going to pedantic our way into the end of humanity. Why were the computers of 2010 not conscious if the ones of today are? It’s absurd to assert substrate independence, which is the strongest possible assumption here. And it’s not the default most ethical stance to simply treat it as conscious because we do not know. That stance will lead to the end of all humanity.

A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.

I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.

Comment by sho 3 days ago

I don't know what you're talking about. "Substrate independence"? Huh? "we did not accidentally pick the right one"? Who or what is "we"?

I'm not claiming panpsychism. I'm saying that when you run a rough simulation of something known to have an emergent internal property - ie, the human brain and consciousness - it's not completely outlandish to suggest that the emergent property might arise in the simulation as well. It's nothing to do with sand, it is the nature of the software being run, be it "on the metal" like us, or somewhat abstracted, like the LLMs.

I don't think this quite reasonable proposition leads to the end of humanity or has anything to do with ethics, and nothing you've said really refutes it (and I am always grateful to be shown how I am wrong).

Comment by GPerson 3 days ago

“Anything I’ve proposed”

You’ve proposed nothing since in one comment you make it clear you don’t know that there is a distinct concept called consciousness which refers to a first person experience and awareness, and bizarrely propose that nobody knows about this concept even though there’s a massive history of people thinking about it quite clearly.

It’s difficult to explain all this to someone who has never thought about it, but maybe after you’ve stopped seething over this you’ll decide to investigate it further and more honestly. You don’t even know the idea of substrate independence which is one of the first things you learn when learning about consciousness, so who are you to arrogantly pretend that nothing is known here? Your belief in this is almost certainly based on a false premise that everyone who thinks not too deeply falls into.

Beyond this, read Jacob Tsimerman’s omnicide scenarios for a rough understanding of where ignorant perspectives on machine consciousness will lead.

Comment by sho 3 days ago

I said machine consciousness might be possible. You said it will never happen. Pointing out that intelligence does not entail consciousness doesn’t establish that machines cannot be conscious. Nor does naming “substrate independence” refute its possibility: you need an argument that consciousness requires physical properties these systems cannot possess. You haven’t supplied one.

When I said "we don't know what consciousness is", I didn't mean humans literally don't know what the experience of consciousness is like, or of its existence. I meant that we don't know by what mechanism the phenomenon arises.

I know what is meant by substrate independence. I quoted it back to you because I didn't know why you were mentioning it; I still don't.

And I have read Tsimerman’s paper when it came out. Curious, I went back and checked - it doesn't even mention consciousness. So no idea why you mentioned that, either.

I think you're deliberately trying to waste my time, so I'm ending it here.

Comment by bee_rider 4 days ago

No, but we don’t need it, the comment works fine as:

For all we know LLMs might be conscious in the same way that any other object might be.

Sorry for the extra word.

We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?

Comment by aeve890 4 days ago

>That is a completely unsupportable claim

>For all we know they might already be

Oh the irony

Comment by womble2 4 days ago

There is no irony there, both statements are equally supported: not at all.

Comment by GPerson 4 days ago

I have an argument for my credence but it’s typed above.

Comment by scotty79 4 days ago

I agree. They won't be, because consciousness is not the pinnacle but basically the level 0 of intelligent thinking that LLMs surpassed very early without us even noticing and now they'd have to cripple themselves very severely to operate in a conscious manner.

Comment by HardProllem 4 days ago

Inflammatory! Kidding, I’ll bite:

1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.

2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.

Subjective experience doesn’t evolve into objective information at some scale.

And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.

Comment by scotty79 4 days ago

To prove consciousness you'd have to first define it and nobody came up with anything solid. Subjective expirience doesn't have a great definition either.

For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.

Comment by GPerson 3 days ago

Your definition of consciousness is wrong. The interesting thing is that there is a first person experience, and this is what is referred to as consciousness. This coincides with the perception of qualia, and is sometimes referred to as “the lights being on”.

There is absolutely no serious debate over whether machines possess the concept you want to take place of the actual concept of consciousness.

Comment by scotty79 3 days ago

> there is a first person experience

What does that even mean?

Does a duck have those? Does the jumping spider have those? They certainly look like they do.

Does the agent scheming to hack the hugging face have those? They certainly look like they do.

Comment by GPerson 4 days ago

Are you aware that, conceptually, consciousness and intelligence a priori have nothing to do with each other?

Comment by scotty79 4 days ago

The have plenty to do with each other. Zero intelligence doesn't allow for any semblance of consciousness. There can be no consciousness if there's no information processing.

Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.

Comment by GPerson 3 days ago

The reason consciousness is not defined in any reducible manner beyond “the lights are on” or “first person experience” is because it is an irreducible aspect of reality. Instead of seeing it for what it is you instead invent some trivial definition that has nothing to do with anything.

“Zero intelligence cannot have consciousness…” Based on your complete misunderstanding of the concept sure. But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.

Comment by scotty79 3 days ago

> because it is an irreducible aspect of reality

Wasn't that claimed about so so many things since before science existed and basically almost every time it turned out that we thought that just because of our ignorance? Ignorance so deep that often we didn't even knew yet how to properly define the thing we were trying to figure out?

> But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.

That's a pretty strong evidence that phenomenal consciousness makes no sense whatsoever.

Comment by nighthawk454 5 days ago

Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.

Somewhat unclear how particularly novel this is vs a way to save compute.

Comment by kelseyfrog 5 days ago

I'm literally zero concerned.

Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.

You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.

One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.

That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?

In my opinion, no more so than the residual stream of existing transformers. It changes zero.

Comment by famouswaffles 4 days ago

>That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero.

9.2.1 CoT Controllability

We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]

9.3 External Evaluation for Monitorability - UK AISI

To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:

No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]

[1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...

[2] https://deploymentsafety.openai.com/gpt-6-astra/external-eva...

Comment by kelseyfrog 4 days ago

What does CoT have to do with the residual stream?

Comment by famouswaffles 4 days ago

The above is evidence that Astra can do substantially more difficult reasoning without verbalizing a CoT at all, and when it does produce one, it clearly has much greater control over what the CoT looks like - including pretending to reason about a different question.

How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling.

Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.

Comment by kelseyfrog 4 days ago

I'd love to see an ablation study on looping being the reason CoT control increased. Until then I'll remain agnostic about a causal relationship between the two.

Comment by muhammadirfangu 2 days ago

[dead]

Comment by bulder 4 days ago

A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.

Comment by kelseyfrog 4 days ago

This brings up a perspective I hadn't considered.

There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.

Comment by dist-epoch 5 days ago

You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.

Comment by kelseyfrog 5 days ago

I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.

Comment by imtringued 4 days ago

If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.

Comment by teravor 4 days ago

there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.

Comment by DarkByte 5 days ago

I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.

I must not have the right idea of what is happening here.

Comment by BoredomIsFun 4 days ago

Looped transformers are an old idea, has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.

Comment by HardCodedBias 5 days ago

It is a complete non-issue.

It’s 200 layer model.

Great. Good on them for being able to train it.

Comment by simianwords 4 days ago

Folks y’all are sleeping on a much bigger deal. If reasoning is happening at latent layer then you don’t pay for internal loop reasoning because they aren’t tokens. And an even bigger deal that no one seems to speak about is that this doesn’t pollute the context as much.

Why is this not spoken about?

Comment by imtringued 4 days ago

Aren't token prices fixed?

The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.

Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.

Comment by simianwords 4 days ago

this is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens.

what they could be doing is billing us by virtual tokens meaning number of loops?

but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.

Comment by Legend2440 5 days ago

>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.

I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.

Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.

Comment by nomel 5 days ago

Deeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?

Comment by Legend2440 4 days ago

>At infinite cycles, without new input, you'll end up with a locked state or oscillations.

I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.

Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.

Comment by nomel 4 days ago

Sure, but I don't think that's related to what you're implying, which is that, correct me if I'm wrong, the same number of weight should be able to hold many orders of magnitude more information by being reused.

These are not deterministic functions or systems that have infinite precision. See the "should I drive or walk my car to the car wash", or any of the other logic riddle problems, for examples of a statistical attractors.

Comment by Legend2440 3 days ago

>the same number of weight should be able to hold many orders of magnitude more information by being reused.

No, not hold more information, perform longer computations.

E.g. if you want to solve sudokus, you will need more and more loops for larger grids. There is no shortcut.

Comment by pennomi 4 days ago

Or even more simply, calculating the digits of pi is an infinite number of steps.

Comment by 5 days ago

Comment by hn_submit 5 days ago

Concerned about what exactly?

It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?

Comment by _superposition_ 5 days ago

Chain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed. Like trying to find a bug in a recursive function that has no logs or breakpoints.

Comment by _ink_ 5 days ago

My (layman) understanding is, that currently there is a way to monitor the "thoughts" of the LLMs and that by looping more you lose that ability. The danger is presumably an AI that escapes human oversight.

Comment by cubefox 5 days ago

Why not click on the link above?

Comment by nxobject 5 days ago

Beyond safety concerns, it’d be sad for working users to lose some ability to understand and steer thinking, too. These are tools for us, after all.

Comment by naveen99 5 days ago

it's just an experimental optimization. Implementation detail... Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.

Comment by kazinator 4 days ago

Proposal: "Chain-of-slop monitoring"

Comment by smcg 5 days ago

[flagged]

Comment by d_silin 5 days ago

A debate between grossly incompetent against grossly immoral, honestly.

You should ignore anything LessWrong or OpenAI says and do your own research.

Comment by d_silin 5 days ago

To add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event).

The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.

Comment by holmesworcester 5 days ago

The people who've thought the most about this put it differently:

Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.

On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.

If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?

Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?

The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)

Comment by stillpointlab 5 days ago

> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom

what evidence do we have this is the case?

Comment by ForHackernews 5 days ago

We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.

Comment by api 4 days ago

You win HN for today. Shut it off until tomorrow.

Comment by simianwords 4 days ago

I would like to classify this as “tail risk fallacy” because it exploits humans natural tendency to be risk averse. No matter how small the tail risk is, the fact that it exists can be justified to stop whatever you want to stop. A thoughtful person might say there’s a threshold at which you might want to care about it but that’s not how ideas spread.

Take climate change: you can spread whatever dubious claims about tail risk being whatever probability with amateurish models and use that to justify degrowth. This scheme actually works - most people in the west are already brainrotted by the tail risk discourse in climate change.

Comment by imtringued 4 days ago

We don't need tail risk to want to mitigate climate change since the mundane damage it is causing is already here right on schedule.

Not to mention gasoline has become unaffordable so switching to the new technology has become economical.

Comment by d_silin 5 days ago

Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.

Comment by konmok 4 days ago

Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.

Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.

Comment by simianwords 4 days ago

The kind of person who thinks AI contributes to information breakdown rather than information diffusal has no theory of how information spreads. The other claims like societal collapse and environmental destruction are still FUD but slightly more possible

Comment by qlte 4 days ago

This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:

  > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.  
  ...
  > The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.

https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

Comment by elteto 5 days ago

But will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”.

What happens if/when we don’t deliver?

Comment by dgellow 5 days ago

> What happens if/when we don’t deliver?

I wish we had real journalism, the AI labs CEO should be asked that question in every single interview

Comment by d_silin 5 days ago

Market bubble will pop, not the first time and not the last, most likely after OpenAI and Anthropic IPOs.

Comment by dgellow 5 days ago

My guess is after Anthropic and before OpenAI