A misalignment of AI in mathematics

Lee Sedol said in an interview that "losing to AI, in a sense, meant my entire world was collapsing. ... I could no longer enjoy the game. So I retired", and I think there will be folks in the mathematical community who would feel the same when the solutions pages to hard problems are suddenly available.

But on the other hand, people learned a lot from chess engines. After decades of chess computers beating humans, there was still a renewed interest in watching Leela beat Stockfish, with many people trying to understand the strategy Leela used.

If your happiness comes from grinding on a problem and making progress, the prospect of having to dig through a corpus of AI-generated proofs might be hard to swallow. But if you're willing to do that, you will still find beautiful things that only so many people can truly appreciate.

I don't like nuance here. I think progress is really measured by what humans are able to do and understand, not machines. It is significant if we find problems we struggle to solve. That tells us something. What does it take for humans to solve these problems is related.

The best analogy I can give is if you wanted to climb Mt. Everest you might ask someone for guidance. Would it be better to ask someone who has climbed Mt. Everest or someone who took a helicopter ride up near the top and then went to the peak? This is like the AI versus human gap to me. The helicopter is like using AI to generate a proof. The person who actually climbed Mt. Everest has firsthand knowledge of the experience. Same thing for a difficult proof. The struggle people have is actually valuable here. Likewise, we know people are actually capable of climbing Mt. Everest but if they had only ever rode a helicopter to the top, the knowledge of climbing it would not exist, and surely that is meaningful knowledge given the risks.

So if we rely on AI for proofs I think we lose a sense of what is difficult and why. We lose a sense of what human achievement is. Surely climbing Mt. Everest means more than taking a helicopter up? For students, why bother grinding through all the material of climbing Mt. Everest and then attempting it if the helicopter ride is how things are done now? This would have the affect of destroying knowledge.

(please do not nitpick the analogy because it's the best but perhaps a clumsy way to describe my thoughts)

Dr. Tao said the same thing. Somehow, this letter came through. He wants to conduct Math competitions where participants who don’t have formal credentials can contribute to mathematical research through AI.

Title: Terence Tao - SAIR Competitions and the Future of Experimental Mathematics

https://www.youtube.com/watch?v=rB9YOi3lb7w

and this:

Daniel Litt - Working with LLMs to do high quality math

https://www.youtube.com/watch?v=0wL8NlhxXcU

This also sounds like a vector for trolling the community with complex putative proofs hiding a known flaw.

The pessimistic scenario is the AI labs will continuously hoover up new developments in mathematics and gazump everyone in their respective fields. It's clear they have no desire to participate in any silly academic niceties, like properly assigning credit or expository work for normal humans. What incentive then do humans have to do this work ?

Generally the mood amongst research mathematicians is pretty dire, and I don't really blame them.

To be clear I think AI is super useful for mathematical research, the problem is the methods of the big labs are massively disincentivising mathematicians from engaging in research, and other associated tasks like giving seminar, teaching writing books etc. These arguably have much more value than finding an obscure counter example to Navier-Stokes.

I think a better comparison is: mathematics just becomes like mining bitcoins.

The current wave of AI slop mathematics might end up driving the next generation of mathematicians away from the subject for the same reason that Mochizuki would have convinced me to quit if his proof had been accepted by the community. Luckily, my professors had the taste to immediately recognize that it was garbage.

It'll basically become slop fatigue if OpenAI starts dumping out proofs faster than the community can keep up, and some turn out to be wrong, never formalize it, don't stay to support it, etc.

Mochizuki was still one human and it required legions of other humans to unpack and untangle to confirm that it didn't lead to anywhere in particular.

AI is now capable of constructions so complex that no human or human team can unpack. And its ability to increase that complexity is growing while our human ability is stagnant.

meta-AI analysis cannot help. We (software professionals who use AI regularly) already know that if you run into a situation where a Fable/Astra-generated analysis reaches the limits of our comprehension/complexity due to their subjectivity, throwing more AI at the problem doesn't always converge.

There are many reasons to feel optimistic about AI, and ultimately its general ability to help science and mathematics.

I see no reason to feel optimistic about the future of mathematics and AI based on the current path of frontier labs, unless the misalignment Tao is writing about can be reconciled.

Imagine a world where these most complex mathematical problems are not accessible to a few hundred people, but a few hundred thousands people. ...Those original few hundred gifted mathematicians would have an even more prominent role, and their names and achievements would be known by orders of magnitude more people that they are now.

I'm not a fan of knocking down things that work, however I also find it hard to be against death of the gatekeeping old guard of any industry.

I think math is just gonna have to suck it up like every other industry now. Math productivity is longer out of reach of the average grad student. Like every other industry they are no longer untouchable and are gonna have to adjust to the new way of things or market forces will do what they always do which is refuse to fund ineffectiveness.

I've had to accept that tech/IT will never be the same. Just how it is. You can thrash against it all you want.

I genuinely enjoy doing this, it's really fun to think critically about what an author wrote or how a particular approach works.

But it does make you wonder, why bother? Probably a frontier model could one shot my algorithm in a day or less. It's incredibly depressing. At least I'm not forced to use it now, but I fear I will have no choice after I join academia or industry in the future.

You must be new here.

Those who think it's me or the machine will fail.

Those who realize how much you can accelerate your research with the help of AI will succeed.

    > Many graduate students (I know) are having a crisis if any of their research worth it? If AI can (or will) do everything, what's the point of doing experiments and all? This will eventually deter a whole generation of curious minded students from research.

we stumbled into a way of brute forcing intelligence with gradient descent.

And then to get retorts of "it's just just rich techies that want to find 'meaning' in their jobs while millions starve in the third world". Why would we expect the most wealth concentrating technology in history to lead to mass benefits for those in the 3rd world?

So I started saying that it only made sense to focus on problems whose solutions would be useful immediately. I even emailed my supervisor about it, arguing that our efforts were “pointless” in the sense that AI-related problems were much more pertinent and had to be prioritised.

My supervisor thought I was bonkers. I still have the email, though. Quoting myself from April 2015:

> By 2030-2040 we will have enough computing power to simulate a human brain neuron by neuron. Once we manage to create a human intelligence we will be one little step away from super intelligence: just set the intelligence to modify itself and see the exponential growth in action. Our human intelligence is bounded by a number of biological factors (e.g. size of a skull) and even the smart human who has ever lived will appear to be a primitive ant to a supper intelligence (machine intelligence will also have perfect motivation). There is plenty of literature on this if you are interested in discussing this further. > > What does it have to do with research in pure maths? I can say that research in pure maths which won't come handy in the next 60 years is just wasted effort. The super intelligence will be able to do maths way better than humans. I believe a lot of current efforts should go into researching of artificial intelligence (or areas to do with AI) instead rather than the pure maths. I want to be proven wrong but most mathematicians I interact with are too narrow-minded to counter me and they just laugh about even contemplating the above. Frankly I am myself so perplexed that I take the above seriously, but I do and it's hurting my motivation.

I'm quite curious what my supervisor thinks of that email now.

Pointing out that regressive Ai politics likely are not going to help researchers crisis of purpose is not unethical.

You also assume in your comment that we need new large generations of researchers, and that knowledge transfer through academic institutions is important.

I get the fear. But entertaining ideas of impact to institutions and potentially new post ai institutions is not unethical.

This is a very disrespectful way to make a point about acting with integrity.

You should consider that maybe your views on what makes something ethical or moral are not universal -- and that coming to a discussion with the assumption that your position is the only valid one is not conducive to convincing others who disagree with you.

Just compare with the comments on https://news.ycombinator.com/item?id=49639408

AI companies are investing these resources primarily as a marketing exercise. There is no near term commercial value to a 100 page Lean proof of blow up in an extreme special case of Navier Stokes, besides the bragging rights. As the statement says any commercial value in this stuff comes a very long time later after new insights and techniques have been digested, integrated into the mathematical canon, expressed in ways that don't take a lifetime of study to understand, etc. (things that AI is not yet capable of doing itself). The bragging rights, on the other hand, are massively valuable. There is a mystique to maths that makes "our AI solved a Millenium Prize problem" an irresistable headline for a company like OpenAI.

What the mathematicians are saying is stop pouring resources that most mathematicians can only dream of accessing into projects that are actively damaging to their field. They face a massive challenge of figuring out how maths can evolve in the face of this new technology, and this is not helping.

It's more about bypassing the culture and processes mathematicians have developed that lead to human understanding, generating new ideas, and bringing up new generations of mathematicians. (See also his article about "non-renewable mining" of good problems.)

Reducing mathematics to "let's just generate results through an isolated and automated system" is a misalignment since it bypasses those processes.

This goes much broader than mathematics or academia. This is the entire basis via which society distributes its wealth: based on a labour market derived valuation of ‘contribution’.

it feels like an unintended consequence of the millennium prize is that people view the [last contributor to the solution] as the only one to make progress on the problem. I've never viewed Poincaré as solved by one person and the objective of the prize was to encourage more people to make attempts and contribute towards progress.

this issue is independent, but in these circumstances perhaps interweaved, with the 'ai is taking over math' concerns

When there's a discussion about doing something against the damage of the AI industry: "whoopsy, sorry, another cat escape, nothing can be done".

When there's a concrete mention of an actual solution to avoid more cats escaping: "that won't happen, and even if it did, the damage is already done, and in fact it’s not that bad you all just have to go with the future we decided for you."

So the bag is wide open, more cats will escape, and nothing can be done about any of it. not about the ones that got out, and not about the ones still inside. Sounds more like a preemptive excuse for inaction, cosplayed as pragmatism

Markets defined entirely by law have distorted our collective understanding of what can actually be built with the knowledge our species has accumulated thus far. How will traditional shields that have protected capital accumulation in tech to survive in a world where governments now realize control of technology is a national priority? Especially as we see its impact on modern warfare, and that such conflict looks like it’s only escalating over time.

Mathematicians appear to me (as an outsider) to exist in a field without such distortions, and I think offer engineers a preview of what’s to come. I certainly have completely ceased sharing original ideas online at this point.

It's more than the yardstick for individual contributions. It's also the yardstick for the contribution of ideas. And that kind of yardstick is very important.

Math has a challenge that's only occasionally acknowledged imo, that's it's possible to just go up the ladder of abstraction, formulate ideas but have those ideas actually not be useful or interesting. The ideas that math has developed, that mathematicians consider important, are those that have helped people solve hard problems.

In a parallel thread omnicognate correctly pointed out that for AI companies it's a direct commercial loss to pour all this money into bruteforcing the solutions to these problems, and that a lot of times the solutions by themselves are not directly commercially valuable. They are doing it for stock price, trying to lure in private capital in preparation for IPOs.

Their models are good, but they are not the moat because Chinese models are good too, so what they are doing, in my opinion, is more harm than good. Mathematics is a science by humans for humans.

There is no doubt that AI has changed the practice of mathematics, just as it has changed the practice of software engineering (and will soon change almost every intellectual job).

Trying to deal with change by saying, "please stop the change" is foolish, IMHO. Mathematicians need to redesign the discipline with AI in mind. But I get that it's easy for me to say that and hard to actually do.

Photography decimated other forms of visual art, so the concern wasn't wrong. But AI threatens the entirety of human intellectual endeavors. I can make do without oil paintings in my home. I'm not sure I want to live in a future where we make do without brains.

Baudelaire popped up in this article two days ago.

https://www.noemamag.com/a-new-kind-of-creative-poverty/

AI is being treated and pushed as a replacement for every medium.

this sort of lack of clarity is what drove people to learn more about the game.

like think about what it must've been like to go to a chess tournaments pre-engines: finals matches had everyone at your chess club watching the game, calculating lines with each other, and you HAD to calculate to understand the direction of game. it sounds so much more engaging and fun!

now, the top games have the stockfish bar next to it, and you instantly know who's got better odds without having to really follow along. and you're not encouraged to calculate beyond a few moves ahead because you just offload the real thinking to stockfish. honestly, it's a vibe a killer when you go back and see images and read about the culture beforehand.

I agree engines didn't really "kill" chess - and I think most of that is due to chess not being tied to economic value. but it did kill what I believe was a superior culture compared to today's chess era.

all this to say, I think what AI did to chess culture in the 90s is happening to stem right now with LLMs

Genuine question: who were those people saying that? I don't know that criticism.

The criticism I know is from AI researchers and it is that beating humans at chess using a computer running an algorithm unlike anything that humans do when they play chess, tells us nothing about the way that humans play chess, which is what we are trying to understand when we try to get computers to play chess.

This criticism is exemplified by John McCarthy's article (the real godfather of AI; because he named it) "Ai as Sport" whence I quote:

Ideas about chess algorithms as well as advances in computer hardware were involved. However, it is a measure of our limited understanding of the principles of artificial intelligence (AI) that this level of play requires many millions of times as much computing as a human chess player does. Moreover, the fixation of most computer chess work on success in tournament play has come at scientific cost.

In 1965 the Russian mathematician Alexander Kronrod said, "Chess is the Drosophila of artificial intelligence." However, computer chess has developed much as genetics might have if the geneticists had concentrated their efforts starting in 1910 on breeding racing Drosophila. We would have some science, but mainly we would have very fast fruit flies.

https://www-formal.stanford.edu/jmc/newborn/newborn.html

A more recent criticism that can be made is that getting computers to beat humans at chess (and later Go, and Shoggi and Atari and Stratego) has turned out to only be possible with techniques that are too narrow to have real-world application. Btw, this includes Reinforcement Learning which is still much more capable in virtual environments than in any real-world environment.

A but like whenever the first sprinter hits a new world record other runners follow along.

Knowing that something is possible tends to strengthen our ability to work with it.

We will potentially see the same with math.

Well yeah... how would there be fewer??

But the point itself is silly. Few people are putting effort into Maths for the fun of it (and of those that are many derive fun from being the only one who can produce a solution). Chess differs in that it never had any point but the game its self.

This situation suits me fine, since I am anyways more of an ideas person, than a crunching open problems person. But I understand the desperation of my colleagues who mad solving hard problems their identity.

In general, in our existing society, status matters, and it is tied to achievements. Therefore a thing that devalues the achievement is also causing people to lose status, and everything that comes with it (income etc). Of course they are mad.

But maybe the answer to that is to acknowledge that our civilization is basically a dick measuring contest at scale, and try something else for a change? Starting with the part where people don't have to be "productive" to have intrinsic value as a person?

They should explicitly advocate for the Fields Medal to weight teaching achievement more heavily going forward.

But there's still play. There's still curiosity. And there's still the drive to understand something for yourself.

But now, LLMs can generate hundreds of books per hour. They make up 80-90% of new arrivals in many nonfiction categories on Amazon. They short-circuit the system, allowing their "authors" to extract money from the system with zero effort by crowding out human work. And it's not even the question of whether these books are good or bad (although overwhelmingly, they're terrible). It's whether it's actually accomplishing anything worthwhile, or just destroying incentives for humans to write or go into any other sort of intellectual work.

In fact, I see many professions push back. Artists, writers, now mathematicians. And I'm amazed that our profession doesn't and that we have so many people who are hooked on vibecoding. I'm still waiting for that 10x payoff. All this velocity and somehow, the landscape of the software I want to use still looks the same as it did in 2021.

Is the fear post-apocalyptic in nature ? We need some human priesthood to carry on tradition ? why ?

Let's assume in the next decade GPT-7 PRO Ultra is cheaply ubiquitous, inspectable, reproducible, transferable, reasonably un-constrained by any institutional interests.

What say the 1% ?

Just a thought.

The hopeful note is that I do think we are entering a golden age for the curious casual/semi-pro mathematician and for niche mathematics areas that won't get the attention of the top labs. Everyone is sprinting to solve the millennium problems, but this is a very exciting time to be in a sub-sub-field where you and 4 others are keeping things alive.

[1] https://substack.com/home/post/p-214740151

The petition is about a much broader problem: if we’ll demotivate people from understanding math, vibe math in a rush without making it understandable - math becomes useless

Basically with pic generation at least we all understand what is on a pic - some cat or something

With math it is trivial to generate a lot of stuff that is half-assed and just gives a plain yes/no but doesn’t get absorbed into human technology and better math

Basically it is not inly about the answer 42, it is about the questions, _meaningful_ _full_ answers and empowering people

By giving a short answer AI can burn the find and make it boring, hide the fertile ground around that answer that any human would’ve highlighted

If you have Einstein - it is not just relativity he did - you can continue asking him questions and he continues the research direction, teaches students, etc

With AI it is often 100% gate keeped (most of the time experimental AIs do the stuff, you have no access even for 200 bucks/month or more) slave that was killed after finishing - hard to “compete” with that and it is not very inspiring approach to researchers ;-)

Well, one thing is stopping them. There will be no more adoration for their genius.

If you truly do it for understanding and not the attention, carry on. AI should change nothing about your motivations.

AI companies are alienating the communities they serve. Instead of a win-win dynamic, they are keen on a win-lose proposition. You dont win trust by one-upping your customer. This is unfortunate and suggests a lack of adults in the room. It also reeks of hubris and is all good when making profits is not a concern. But watch the narrative shift when there is an AI slowdown which is inevitable.

Are we really going to take what they say in public seriously?

Ironic or what. Mr Tao may be remembered as Mathematics' Canute.

Yes, this has gone so well

It's not necessarily clear that this statement requires infinity, if you're willing to treat "... =" as a shorthand. You might prefer something like "1 + 1/2 + 1/4 + 1/8 ... -> 2" if it's more clear, where "->" means something like "gets as close as you like without ever getting further away than that", but really the "=" sign is already overloaded in all sorts of subtly different ways anyway, so there's not really any trouble using it here. Almost any rigorous definition you can write down of exactly what that statement means would not rely on the use of infinity.

A better example would be a limit that equals sqrt(2) which finitists would probably say cannot represent a real object because it is only defined as the end of an infinite process.

Would you argue it didnt benefit humanity, because taxi/bus drivers are nolonger required

Sure, but in your scenario, the process is the experience for a small group of people, while the result benefits everyone (perhaps not directly, but ultimately via a chain of applications). Which one is more important?

It's not at all a new dilemma either. Jacquard looms destroyed the "human experience" of manual weaving for craftsmen who spent decades to perfect their skill. They also made clothing orders of magnitude cheaper for everyone else.