Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

Honestly the fiction writing was pretty impressive!

Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.

In my limited usage, it has fixed issues which has not been correctly fixed by sol-6.1 (medium) - https://github.com/auto-gnufied/markdown-modern/commit/a189a...

I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.

I feel like you're being overly dramatic here. Argon was 13 - 15 days.

Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.

This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.

The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).

(googler, opinions are my own)

That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.

Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.

At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.

Is that chat messages that involve tool calls (web search, file reads, etc) or actually just messages?

When I'm on the $20/month OpenAI plan, I can get about 3-5 turns out of Astra before the 5 hour limit is up.

When Fable was on the $20/month Anthropic plan, I remember getting about the same.

3-4 turns out of the $20/month plan would seem about par with other providers for their best model, but really Carbon should occupy that slot, not Argon.

If Argon's limits are that low, it wouldn't make sense to use that plan for anything when models of similar intelligence are available with much higher limits on other $20 plans.

From Google saying that Argon would be 'available soon' (as they did just now for Carbon), it took them 13 days to release it. So... unclear that Carbon won't see the same trajectory.

Gemini models always suffered from this. They were almost always best in terms of intelligence, but one of the worst to use in practice, because they have a lot of failure modes on simple tasks that simply give factually incorrect answers, or completely break and output gibberish.

It looks like the best sort of bang for your buck is still GPT Sol and GPT Astra. Claude and Gemini have just been going for that number one intelligence spot by sacrificing a lot of efficiency.

It does sort of show their hands a bit that their larger goals might be outside of just consumer sales but also seeking hyper intelligence that could bring value in new ways of discovery (if it ends up being a legit outlet for such things consistently)

I don't know what it is about Google models, but they are the worst at hallucinations. I don't understand how they can so consistently be this bad.

Yesterday I used Google AI-Search (so Gemini Flash 3.5?) for opinions on a traffic accident and it continously constructed law-interpretations and then referenced the wrong paragraphs. It talked about rules on road-crossings and cited paragraphs pertaining to alcohol intoxication as sources.

the problem with gemini is the whole app itself, not the model.

when you use other chat app, like chatgpt or deepseek chat, the app will use web search for everything to verify. and if it can not find anything useful, or the current fact is disputed, it will try to call web search tool again and again until the consensus.

meanwhile gemini will refuse to do web search if possible, and only do it a single time if demanded. worse, it uses the same result from google search, so the top ranking answers are almost all SEO garbage.

since then I just stopped asking gemini again. and I don't think the newer model will change the situation.

Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).

Many other relatively great things were done this week in parallel. I have some budget for research.

https://www.youtube.com/watch?v=NO2I3ueVQrk&lc=UgxRXtipA3QbA...

tldr; it's not as good as opus 5.5

Trip planning is the most often used use-case for agents/bots I see in marketing. I'm lucky to travel once a year, but if I did have a trip, I wouldn't feel comfortable letting an LLM take the wheel. Too many variables for wrong flight or room, closed locations in itinerary, etc.

Now, critical server infrastructure? Sure, have at it Claude, make no mistakes!

Could you expand on this? I’ve never seen a frontier model that doesn’t hallucinate details fairly often, what do you mean by “actively” here?

Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.

Comparison with other models.

It is unfortunate that having such an amazing engineers that are capable to solve any leetcode exercise in 5 minutes on the spot with a white board, they aren't able to even deliver something useful on the AI space.

The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.

https://artificialanalysis.ai/models/qwen3-8-flash-next

https://artificialanalysis.ai/models/glm-5-3-flash/

edit to add: It's very disturbing, because these open models (GLM particularly) were among the best of all LLM's on their benchmarks. On the intelligence/cost frontier, specifically. Archive.org remembers that GLM Flash was in their green quadrant just a few days ago, next to Haiku:

https://web.archive.org/web/20261014031624/https://artificia...

It's completely gone now. AAI (as far as I've seen) haven't publicly commented about why they did this.

That seems arguably pretty cheap compared to the human alternatives.

I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.

Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.

Example problems are:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.

Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.

This is also arguably more of a test of a model's knowledge base than reasoning ability.

But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.

The coding agents subindex, in contrast, looks at least superficially plausible [1].

[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...

Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.

At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.

Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.

Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.

So ... why bother to cheat?

"Hey we are launching this thing but we've got to let the safety folks see it first".

So they look like they care about alignment etc.

To be honest there is a lot of rage on here about it being announced before being able to access it. I don't understand why people are getting so upset about this - like no one gets into a rage because they see a movie trailer before it is released?

I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.

Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking

True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.

While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.

They have top of the line research, I mean, they invented the thing, and they are still doing stuff that goes beyond making a bigger LLM, as it seems to be the only thing the others are doing, research-wise.

They also have hardware capabilities no one else has, they invented the TPUs, and have lots of them.

They also have the brand recognition, ecosystem, user base and lots of money.

So why are they lagging behind on the actual products, and by lagging behind? I mean they are always in the position where they are just behind the frontier, but still in the game.

It seems almost deliberate. Maybe they are playing the long game, waiting for the bubble to burst. Maybe they don't want to compete against the more profitable parts of the company. Or maybe their researchers are not interested in making an actual product unless being forced to.

Even if these numbers are correct, they don’t have any useful product for coding or search. Not sure what you’re referring to

I mean, that's the main reason. How would you expect to see the kind of hype that Claude or GPT can generate when nobody can use it? The hype of those models are primarily generated from people using them to build impressive things and share those results. Can't do that with an unreleased model, right?

Beyond that, it seems a lot of people have learned that these agent leaderboards really don't tell the whole story. They're a good litmus test, per se, and there's plenty of hype being generated around these scores as-is. But anybody using AI in practice knows that benchmarks can only say so much, and one model scoring better than another doesn't necessarily imply that that model is actually better in the ways those people care about. You can see that in this thread as people are pointing out that even these benchmarks seem to suggest that Opus 5.5 is still leading in coding (which is what a lot of people, especially on HN, ultimately care about).

>just not enough publicity or hype as the other labs do?

Well, it's effectively both. Google is certainly not that pressed to generate as much hype as these other companies. They can afford to take their time. It's also worth noting that Google is definitely targeting different consumers, investors, etc., so their strategy will definitely be different.

For what it's worth, I've been using Argon since it's been available in Cursor and I really like it (at least at its current price). I think Google really did close the gap a meaningful amount (maybe not all the way, but enough to prove they're still in the game). Carbon being significantly better while being the same cost is pretty exciting to me. But until I can use it, I don't really care about it.

"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."

Want a private benchmark and see where you'd rank? $$$.

Want to show up day 0? $$$$.

Want to show up day -0? $$$$$.

Argon did not leave a mark or change anything

Why would anyone care about carbon as well