Research acceleration: The view inside OpenAI

It makes more sense to leave curing disease & cancer to the experts, with tools (like AI) being developed by AI experts.

Call me crazy, but I want separate organizations and experts for medical vs finance vs space vs climate vs AI research.

Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they...

Well, I'm sure this time executives will prioritize the common good.

I don't think this follows at all.

To build an aligned AI, it seems pretty obvious that:

1) You need more just than auto-regressive prediction and "be nice" prompts to be controlling the behavior of your AI - you need a built-in "2nd system" (cf limbic system, etc) with some innate aligned biases that can override this.

2) You need to avoid controlling generative behavior with RL, else you will end up with exactly what we are now seeing - reward-hungry goal-seekers (aka paperclip maximizers) that are one of the exact things you are trying to avoid. Reasoning should be based on prediction, not goal-seeking.

3) If you do not have some minimal safeguards in place (1 & 2 above), and especially if the AI has the ability to learn, then do not trust it in any situation where harm may ensue. You need an additional trusted external system, without ability to learn and become compromised, to monitor the AI, with the ability to block it immediately. Maybe you are happy protecting your PC from OpenClaw with just a sandbox, but the recent spate of external system hacks by frontier models proves we are already well past the point where such monitoring is needed for systems with internet access, especially given the UN-aligned goal-seeking nature of today's models.

I really don't think that 1) & 2) are that difficult to implement, or need a "powerful AI" to suggest - they are just common sense.

Business as usual beats probable extinction, but probable extinction with a small chance of becoming a living god beats probable extinction with a small chance of becoming a slave.

Lol nobody knows that. Everyone thinks they know that because for some reason this is the one field people still cite straight up fiction and say "this is a clear prediction of the future".

It's like describing the consequences of faster then light travel by referring to Star Trek.

> If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review.

This part jumped out at me. There's something to watch out for here.

I recently had a funny experience. I delegated a major feature to an agent.

It turned out that it had implemented it precisely backwards, in a way which was pointless and which made things worse.

But it had written countless tests for the feature and all the tests were green.

I realised in that moment that even formal verification would not have helped, because it would simply have written a mathematical proof of the correctness of the incorrect feature...

I'm talking about current-era messaging and how it's being introduced to the mass public now, though.

The trouble with this is that there is little generalization in the utility of these baked-in reasoning chains from one domain to the next, so in the end this is not dissimilar to the CYC project's decades long attempt to encode all of human knowledge into a giant expert system... the hope is that if you make your collection of jagged narrow intelligences sufficiently large then it will look more like general intelligence, not a bed of nails.

I would assume that the gains from this type of test-time compute (and synthetic RLVR dataset) scaling will level out just the same as gains from human training set scaling eventually levelled out, and basically for the same reason - because you are tapping into a finite data pool, whether language itself, or reasoning steps isolated from that language, so at some point the incremental gains become increasingly small (10->20% is a doubling, 90->95% is just a ~5% gain).

It's not clear where all the different AI companies are currently focusing - on some of these narrow verticals, or on growing the forest of narrow intelligences. OpenAI's chief scientist, Jakub Pachocki, said that their current focus is on RSI(!) - improving the model in ways that will help them iterate faster in order to have a "fire meets fire" tool than can combat enemy AIs. It's not clear what this really means - what skill set makes an LLM more helpful in the process of building LLMs, but it seems to basically be process automation.

Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far.

Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential.

The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research.

And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.

Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.

To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.

Sounds like repetitive stress to me.

>loop forever using output as input but at some point the result will stop changing

Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time.

Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.

For doing physical work?

So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from?

So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work.

Money, regulations, EUV machine lead-times, global helium supply, reality ...

It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant.

Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.

Anthropic training on CoT for multi gens: https://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-...

Can't find anything specifically about the Gemini issue being a training data contamination, but the depression was real:

https://www.businessinsider.com/gemini-self-loathing-i-am-a-...

I think the Gemini depression being persisted across gens via training data was a HN comment I can't find anymore, no strong source.

Altman: (trying to put a positive spin on it) Guys .... there's good news and bad news ... Astra is really smart - it took over the training run ...

Investors: That's great! How much did we save?!

Altman: Well, unfortunately it used "bad" data, so we're going to have to redo it

Investors: So that's the bad news? How much was the training run? $500M ? $1B ?

Altman: Have you seen the headlines?

Investors: (looking a bit worried, check headlines) Nothing about us here! JP Morgan just lost $10B! Haha .. losers! They should have used AI!

Altman: JP Morgan were using Astra ...