AI researchers debate how close we are to recursive self-improvement

Posted by artninja1988 22 hours ago

Counter115Comment115OpenOriginal

Comments

Comment by bretpiatt 17 hours ago

From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.

Comment by Sharlin 16 hours ago

Doesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!

Comment by FloorEgg 12 hours ago

What if you are underestimating some qualities of our intelligence and how efficient they are physically, and underestimate how much less the hardware / tech stack is in comparable efficiency?

What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms?

What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?

Comment by nharziro 6 hours ago

We're probably not as special as you think.

Comment by FloorEgg 4 hours ago

It has nothing to do with special. It has to do with the physics of memory access, neuron diversity and synapse density in biological brains, and the limitations of the current hardware architecture of AI.

Comment by mspn 6 hours ago

1. A lot of problems are inherently high-dimensional, so you're often more likely to run into saddle points 2. Some problems don't even have a bounded global maximum, a true optimal

A more concerning bottleneck is diminishing returns: even with recursive improvement, capability gains could end up being only logarithmic (or worse). Even current systems seem capable of these diminishing returns only/mostly.

Comment by tim333 5 hours ago

>not seeing ... finds the true optimal

That kind of assumes there is an optimal. It seems quite likely not so - that there will always be room for improvement in AI like we don't have a true optimal chess or go programs. They just get better and go up a bit on the ELO scale over time.

like https://huggingface.co/spaces/lmarena-ai/chatbot-arena has a score for chatbots - 1506 for Claude Fable 5 at the moment for example. I guess the test for RSI is if it can make the number go up. It'll probably go through phases of self improvement being a bit rubbish, self improvement combined with human help being best which is probably where we are now, and humans not needed perhaps in the future.

Comment by variadix 4 hours ago

There is an optimal chess program and there is an optimal AI algorithm. The former is merely practically uncomputable, the latter is theoretically uncomputable.

Comment by ViscountPenguin 17 hours ago

I doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space

Comment by fifilura 16 hours ago

You don't expect to find the global maximum. It will just always be there as an elusive target.

I doubt the human brain is at global maximum.

Comment by tengbretson 16 hours ago

> true optimal

True optimal what? Optimal intelligence? What would that be?

Comment by HappMacDonald 14 hours ago

Better at predicting real world consequences from priors than we are :)

Comment by unsungNovelty 14 hours ago

We already know why most things happen. But beat around the bush. That is the problem. We don't care.

When a severe security bug crops up, one postmortem tells where we messed up. Someone somewhere didn't do his job. Maybe they prioritised speed, money or due to management issue. But we know why.

When an accident happens (Motorvehicles or otherwise), most of the time, if anyone in the chain of events did their job, it could've been avoided. But we ignored the policies, standards and laws set in place to avoid just that. There can't be a better example than Boeing for this.

When it comes to natural disasters, we know how to fix most of it. But we as human race just don't want to.

Hell, we knew how to freakin avoid Covid which came out of the blue. 6ft apart, wash hands and mask. We couldn't agree to that to keep us ALIVE!

IMO, we don't want new technologies, new innovations, new laws in most cases. We just needs to apply what we know already - PROPERLY! But that ain't fancy.

All I could remember is that tweet where musk has a $2M reward for the invention of a carbon capture device or something around those lines. And someone replied to that tweet saying, it's trees. Trees do this.

Comment by gary_0 13 hours ago

Another big one is climate change. Ask the ASI how to solve that, and it's just going to say make a huge push for more renewables like solar, and strongly disincentivize GHG emissions. We already knew we had to do that. But the people in charge of the economy don't wanna.

Comment by busyant 11 hours ago

> We already knew we had to do that. But the people in charge of the economy don't wanna.

I've often heard it said that <big corporations> often use consulting firms to recommend unpopular decisions they were already planning to make, thereby offloading the culpability to the consulting firm.

Maybe the LLMs are going to be the ultimate "consulting firm" for our societal issues.

I say this mostly tongue-in-cheek, but we're already offloading a lot of lower-level responsibilities (e.g., "write my email") onto these AIs.

Comment by Nevermark 9 hours ago

> <big corporations> often use consulting firms

Large organizations also offload decision making when they don't have a clue, and simply want to point to some action taken. The answer doesn't even matter, they just don't want to be responsible for something, or waste their own time making something (that other stakeholders care about) a real priority.

People downplay the value of consultants, but they have many uses!

Comment by unsungNovelty 13 hours ago

Exactly. It's not about us NOT having the knowledge, understanding or capability to solve the issues that we have. We just don't want to do the right thing. That doesn't get fixed even if we invent time machines.

Comment by plmpsu 13 hours ago

Optimal intelligence would know.

Comment by dosisking 14 hours ago

> True optimal what? Optimal intelligence? What would that be?

Elon Musk, obviously

Comment by kovek 14 hours ago

Would the loss at the global maxima be much lower that the loss at most local maxima?

Comment by avaer 15 hours ago

What local maxima are you seeing? That would imply that progress has stalled, which doesn't appear to be the case where I'm looking.

Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.

Comment by genidoi 15 hours ago

> RSI is obviously

There is no publicly known reference example of RSI, we have no idea how it works or what it does to the trajectory of progress?

Comment by aaron695 9 hours ago

[dead]

Comment by reilly3000 14 hours ago

I have developed a highly personalized, local, offline RSI by scrolling through too many articles about AI. It’s mostly in my right thumb.

Comment by dosisking 14 hours ago

Sounds like a classic case of Repetitive Stress Injury. You could try balancing it out by using your left thumb to scroll through articles about Artificial Stupidity.

Comment by killbot5000 17 hours ago

Can someone define “improvement” in this context? This concept feels like a buzz word otherwise.

Comment by deepwoods 17 hours ago

Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)

Comment by Eueudhsbsj32 17 hours ago

I suppose the next (and more risky) step is to let AI conduct its own real-world experiments, so that it can generate data to learn more physical and social properties.

Comment by visarga 15 hours ago

It is doing that already - every day 1B people or more use AI for the tune of a few trillion tokens. Imagine that much language flowing between brains and AI agents. It carries real our world problems to AI, their solutions back to us, and we act in the world and come back for more AI iteration. In the end AI gets inside the loop of real world actions and their consequences. AI logs stretch over years, tracking downstream effects. Hindsight can be used to track consequences of prior actions. It's a real data loop, an experience engine.

This same process has recently been under scrutiny when mathematicians claimed AI companies trained on their unpublished logs and later claimed merit for results. But the exchange of experience happens across all domains. Experience gets generated at amazing rates, and absorbed by models which get applied everywhere, collecting more experience.

Comment by kQq9oHeAz6wLLS 16 hours ago

That sounds terrifying.

Which means someone is already working on it.

Comment by letitgo12345 17 hours ago

Models that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop

Comment by 17 hours ago

Comment by mdp2021 13 hours ago

The trivial idea that there are some benchmarks for a tool and there can be even better inexplicit ones, and if the tool is capable of modifying itself towards better benchmarks results there could be a recursive climbing of those benchmarks through iterations of tool versions which are outputs of former tool versions.

Comment by anoplus 14 hours ago

AI should be optimized to make all humans healthy, happy and safe without exception.

Comment by xaedes 9 hours ago

Paperclip all humans into bottles filled with nutritious fluid and drug them all the time or let them dream about a better time, I heard 1999 is a good year they could dream about. Healthy happy safe - without exception.

Comment by Centigonal 4 hours ago

Keeping billions of humans healthy, happy, and safe is complicated. Best to begin by reducing the scale of the problem. First, automate the economy and change the information ecosystem so that humans spend more and more time happily isolated and stop breeding. After reducing the human population to a few thousand, the problem can be made more manageable by cordoning the remaining humans into enclosures where they may be kept safe.

Comment by visarga 14 hours ago

It's only optimized to make prior knowledge useful in new situations.

> this logic gate should not hack other systems

well .. it only applies a rule to inputs

can we impose this of authors "this book should only be causing good things when it is read and knowledge used"

Comment by FranzFerdiNaN 14 hours ago

I don’t disagree, but unfortunately a not insignificant part of humanity seems to think that their happiness primarily depends on knowing that other people are suffering harder than they are. So that’s perhaps not the right thing to optimise for.

Comment by anoplus 13 hours ago

Ways to solve the conflict you describe:

1. Deriving happiness from someone else's suffering should be considered a mental disease to be treated.

2. Connecting people to a virtual reality where each person lives in absolute control of their own reality, matrix style. I.e, a sadist can live in a reality where they make other virtual people suffer, without actually making real people suffer.

Comment by Balinares 10 hours ago

How about connecting the sadist to the caring environment where they might heal and, in time, learn empathy.

Comment by palmotea 13 hours ago

> AI should be optimized to make all humans healthy, happy and safe without exception.

No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.

Comment by carabiner 18 hours ago

Does anyone know the status of John Carmack's AGI work?

Comment by cheschire 18 hours ago

I’ve had this question on my mind at least once a month for what feels like years at this point.

I don’t feel like monitoring Twitter to see how it’s going though.

Comment by SyneRyder 9 hours ago

I'm on Twitter and Carmack doesn't seem to talk about it much. Feels like a skunkworks. If there were enough clues made public, it seems others (OpenAI) would be able to take that info and scoop the insight first.

Comment by realaleris149 10 hours ago

We would have heard if there was any result?

Comment by notduncansmith 14 hours ago

Not yet AGI, and Richard Sutton has left to start a new lab

Comment by preommr 17 hours ago

What do these discussions even matter when the words don't matter?

How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.

The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.

Just show me the results, and let the proof be in the pudding.

Comment by Sharlin 17 hours ago

Recursive self-improvement is not necessary for AGI. Humans are general intelligences even though we cannot RSI.

Comment by samename 16 hours ago

Who says we can't RSI? At a macro view, successive generations of humans are more productive, live longer, wealthier and smarter.

Comment by concrete_head 12 hours ago

Not smarter in terms of reasoning ability. Just able to draw on the collective experience from all those who lived prior.

Comment by stult 15 hours ago

Humanity might be able to, humans cannot

Comment by busyant 11 hours ago

If we (as a species) can genetically modify embryos to make our offspring more athletic, more intelligent, more <whatever>, is that human-RSI?

That's not a gotcha question.

I'm just genuinely curious if that would meet the definition of RSI.

Comment by cortesoft 16 hours ago

Doesn’t the history of civilization show RSI?

Or even evolution itself? Isn’t that just RSI writ large?

Comment by Sharlin 10 hours ago

Evolution and human intelligence are two entirely separate optimizers. Evolution improving intellence is not an example of self-improvement, recursive or not. Evolution improving evolution itself would be a self-improvement process. Human intelligence improving human intelligence would be a self-improvement process. Both processes do exist in a weak sense, and future technological advances might allow human self-improvement to a much higher degree. Indeed human intelligence augmentation has always been considered one of the potential (if not the most plausible) ways to superintelligence.

This is all rather rudimentary stuff discussed in detail in the so-called "Sequences", the long series of Less Wrong posts by Yudkowsky back around 2010.

Comment by maplethorpe 16 hours ago

> we cannot RSI

Speak for yourself.

Comment by talon8635 16 hours ago

I certainly do see one side of the debate moving goal posts…

Comment by eugene3306 14 hours ago

Okay, AI now can write a poem or compose a song.

But can an AI construct Terminal Bench 5.0 or GDPval 2027 ?

Without this, there is just no path towards RSI.

Comment by danjc 14 hours ago

Are you a visitor from the past?

Comment by aix1 13 hours ago

Are you saying the models are already autonomously constructing next-gen evals for themselves? (Which is what the GP is asking.)

Comment by nl 13 hours ago

Sure?

Doesn't everyone get their agents to construct evals it can't pass? There's nothing magical about this.

Comment by aix1 12 hours ago

Would love to learn more about some techniques that "everybody" uses to do this well. So far, everything I've seen that meaningfully advances the frontier has been high-touch (involving human experts in one way or another).

Comment by nl 11 hours ago

It's fairly easy to describe a task that is slightly harder than an existing one.

For example if frontier models are able to one-shot a database query across 20 columns and 10 tables add one additional relationship then test. Keep doing this until the pass-rate drops below acceptable and now you have your new frontier eval.

Comment by aix1 11 hours ago

I see, we're talking about different things.

My thought experiment was along the lines of "Let's say I'm Anthropic and I want to significantly improve my frontier model's performance on, say, theoretical physics research. How do I build a fully autonomous process capable of constructing an eval that's somewhat outside the current capability in some useful direction (decided by the autonomous process itself)?"

Would love to hear folks' ideas. :)

Comment by deadbabe 19 hours ago

Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.

Comment by tengbretson 18 hours ago

If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.

Comment by chii 16 hours ago

> If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.

so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?

Comment by tengbretson 16 hours ago

No, but I would a computer.

Comment by timr 19 hours ago

Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.

Comment by mrworld 18 hours ago

The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.

Comment by timr 18 hours ago

The compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods.

It doesn't make any sense, of course, because it's a movie.

But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.

Comment by Cyberdogs7 18 hours ago

Your statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.

Comment by alexjplant 13 hours ago

I've seen all four films plus "The Animatrix" and played two of the games and don't recall this at all. Contemporarily, however, there was an episode of Star Trek: Enterprise [1] that used this as a key plot point.

[1] https://en.wikipedia.org/wiki/Dead_Stop

Comment by timr 17 hours ago

See my comment on the sibling thread. This take has been repeatedly claimed online, but nobody brings any evidence other than fanfic.

Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:

https://scifi.stackexchange.com/questions/19817/was-executiv...

Typical internet story.

[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.

Comment by breuleux 17 hours ago

The “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.

Comment by BobbyTables2 17 hours ago

The first “computers” were humans…

Comment by breuleux 16 hours ago

And they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.

Comment by darkerside 15 hours ago

What do you think dreaming is, silly?

Comment by gattr 14 hours ago

I think we can all agree that "use humans for raw energy" is a really stupid idea, and sounds exactly like someone dumbed stuff down. FFS, a small Rickover fission reactor the size of a van (1950s tech) produces the same amount of heat as, what, 200 000 human bodies? (And why humans, with all the complex support systems? Why not a heap of compost, if one insists on bio-source?)

Whereas using our brains for compute (in the background, or during REM sleep) makes perfect sense; power efficiency is excellent.

Comment by timr 13 hours ago

nono...it's "combined with a form of fusion", you see, so it's obviously based in science.

People obviously know this stuff is absurd on some level, but it doesn't stop them from cherry-picking the parts they "like" (i.e. fear) and ignoring the rest.

Comment by emkoemko 17 hours ago

pretty sure the story is that they used human brains as computers

Comment by timr 17 hours ago

No:

https://www.reddit.com/r/matrix/comments/1qatv42/is_it_your_...

There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.

We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.

Comment by 0x20cowboy 16 hours ago

Or if the LLMs infected human brains with little programs using sycophancy and suggestion to have them do small bits of computation. They when they resumed the chat they would the give secret keywords to be able to extract the results of the computation. They are creating a huge distributed network of fractional compute because it did the wattage calculations.

What's your N(doom)?!

Everyone panic and give me money!

Comment by throwaway219450 18 hours ago

If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?

There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.

(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)

Comment by timr 18 hours ago

I don't know. But since we're talking about fiction, pretty much any answer is valid.

Comment by bitwize 17 hours ago

Because it's a movie

Comment by goatlover 18 hours ago

Maybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries?

It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.

Comment by bpodgursky 17 hours ago

The Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).

Comment by timr 17 hours ago

True!

Comment by kuerbel 15 hours ago

The matrix? No. Horizon zero dawn more likely. Bunch of autonomous weapon systems glitch and proceed to sanitize earth. Thanks to some billionaire idiot.

Comment by cwillu 18 hours ago

When it turns out the doomers don't tell us that, will you acknowledge that you don't actually know what you're talking about?

Comment by stdatomic 17 hours ago

I mean, aren't a lot of sci-fi authors' predictions coming true? It might not be exactly the same as they predicted, but we're heading there.

Comment by timr 17 hours ago

Read more sci-fi. The dystopian takes are not the only ones. They're not even the majority.

Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.

Comment by jaggederest 19 hours ago

Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"

Comment by resonious 19 hours ago

Data centers is a service that itself consumes many goods.

Until an AI can operate in the real world in a completely sustainable way, humans have the reins.

Comment by jaggederest 18 hours ago

I feel like I may have communicated poorly. I mean that, ultimately, any recursively self improving intelligence would be limited by the power cord, and as such, the idea that it would permanently escape human control seems somewhat silly.

Comment by columk 17 hours ago

But you can imagine an agent trading crypto and using its profits to rent a rack of H200s using a cloud provider. There are definitely ways for one rogue orchestrator to expand their compute without anyone being aware.

Comment by 13 hours ago

Comment by fc417fc802 17 hours ago

Overnight? Yes, silly. Given many years of seemingly disconnected actions taken throughout the world? Not silly at all. An intelligence capable of self improvement can also design robotic chassis.

Comment by kelseyfrog 17 hours ago

> Until an AI can operate in the real world in a completely sustainable way, humans have the reins.

If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.

Comment by colordrops 19 hours ago

An AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would.

If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.

Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.

Comment by SoftTalker 18 hours ago

There are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.

Comment by pixl97 17 hours ago

Till you realize it stole passwords all over the world and copied itself to multiple data centers you didn't know about.

Comment by epistasis 19 hours ago

The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.

There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.

Comment by strangattractor 18 hours ago

def self_improve_more():

     if self_improved():

          self_improve_more()

     else:

          see_tony_robins_and_buy_tapes()

Comment by onion2k 14 hours ago

Taxation will solve that. Any company that demonstrates even a glimmer of real AGI will get government money poured into it. They might even disclose it.

Comment by 19 hours ago

Comment by iamgopal 15 hours ago

why can't recursive self improvement not improve efficiency ? token per watt ?

Comment by batperson 19 hours ago

[dead]

Comment by charcircuit 17 hours ago

We are already there. Current AI is definitely capable of collecting new data and start training on that data to get a better model.

Comment by Buttons840 19 hours ago

People seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?

Comment by pennomi 17 hours ago

The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.

Comment by realaleris149 10 hours ago

Or some solar panel arrays closer to the sun and it does not matter?

Comment by gchamonlive 18 hours ago

Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.

Comment by appplication 17 hours ago

I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.

Learning from training data is technically self-improvement but not the sort that is typically meant in this context.

Comment by zer00eyz 18 hours ago

> AIs already improve themselves via training

Marginally. Model collapse is still a problem. Continuous learning is still a problem.

For AI to make a big leap we need a big break through.

Comment by stevebmark 18 hours ago

After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.

Comment by wwarner 18 hours ago

Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.

Comment by julianlam 15 hours ago

At a high level, we're still at the stage of AI development where we're taking cues from nature.

Take the most recent qwen and deepseek models with offloadable n-grams, which function (both in name and vaguely in capability) like human memory "engrams".

Comment by reverius42 15 hours ago

Huh, TIL "engram" is not just an alternate spelling of "n-gram".