Artificial Analysis v5 and the benchmarker's dilemma

Posted by 6thbit 1 day ago

Counter33Comment25OpenOriginal

Comments

Comment by matesz 2 hours ago

"The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars."

Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.

Comment by ekidd 1 hour ago

> Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.

Interesting. My experience is that Opus 5.5 is smarter than a model like Qwen3.8 Flash Next. But given the choice, I'd actually rather work with a model like Qwen3.8 Flash Next.

Even on medium reasoning, modern Qwen and DeepSeek models are ridiculously diligent. If I ask them a question about which library is best for a given task, they'll download 4 different candidates, read the source, and write test programs. Meanwhile Opus will look at a couple of web pages and give me glib bullshit answer. Opus is often like a brilliant coworker who can't be bothered to do their job. The modern Qwen and DeepSeek models are like someone who is merely smart, but who has an alarmingly good work ethic.

And I'm not just writing simple React CRUD GUIs. I'm doing stuff like modernizing ancient software forks, working on high throughput distributed systems, or finding bugs in other people's parser generators. And even the 120-240B open weight models are normally entirely adequate for these tasks.

One of the benchmarks where open models tend to perform badly is Terminal-Bench 4.0. I looked at some of the sample tasks on that benchmark, and some of they are really hard and specialized. Look, all I actually need out of a model is things like "tell me where this distributed algorithm might break", "get this ancient C code to build on modern Debian with modern dependencies", or "write a plan to migrate this database from MySQL to PostgreSQL and to test that the app still works." The modern open models can do all that, no problem.

Comment by 35 minutes ago

Comment by nsoonhui 1 hour ago

I believe so, this checks out with my experience, especially when comparing between the US models and Chinese models which benchmark scores are neck to neck with US ones.

My experience with Chinese open models is that they couldn't cope at all with my production work; both GLM 5.3 and Deepseek v4 ( or was it 4.1? Couldn't remember too well) went into infinite loop and wasted my tokens until my OpenRouter wallet reached 0; good thing I didn't enable the auto topup. US models, by contrast, breezed past them.

Even for simpler tasks, Chinese models took long time to complete, and I needed to supervise closely. The price for Chinese models, in the end, didn't come cheap, mainly because too much time was wasted on thinking, and this was regardless of whether they finished the tasks or not.

Comment by rubslopes 2 hours ago

If they don't care, why are them presenting them when launching a new model, every single time?

Comment by embedding-shape 1 hour ago

Because people, for some stupid reasons, still read and make decisions by these public benchmarks, which makes zero sense. Create your own benchmarks and evaluations, then use those, it's pretty the only way to get any qualitative answers to how well a model will work for you and your user cases.

Comment by no-name-here 52 minutes ago

How many users are willing to spend the time and repeated cost (once for each new potential model+version+variant) for that - 1 in hundreds? (Edit: I was thinking about this as an individual dev. Especially for AI uses in a biz workflow, custom benchmarking makes perfect sense yes.)

Comment by embedding-shape 44 minutes ago

Probably the same amount of users who actually care about figuring out how well a model works for their use cases, instead of what amounts of guessing.

Most companies I've consulted for when it comes to LLM usage in various ways all have their own evaluations, not sure why others would avoid them.

This is to save time, for what it's worth. Doing that testing and evaluation will take you MUCH longer than doing so automatically, for obvious reasons.

Comment by lordofgibbons 2 hours ago

I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.

I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.

Comment by artemisart 2 hours ago

Maybe you need to enable them in the filters, but AA is the best website I found with benchmarks numbers on all reasoning effort levels of recent frontier models.

Comment by lordofgibbons 1 hour ago

I dug into it more, and it seems like the reasoning efforts are available for the Intelligence Index, but not the Coding Agents benchmark I was interested in.

Comment by ac29 2 hours ago

The other reasoning levels are almost always benchmarked, AA just doesnt show them in the default charts.

Comment by frenchtoast8 1 hour ago

They don’t benchmark different reasoning levels for Deepseek models.

Comment by lordofgibbons 1 hour ago

I see them for GPT 6.1 Terra, but not Opus 5.5. I'm looking on the Coding Agents page.

Comment by chvid 1 hour ago

How does AA make money? Who are their investors?

Comment by kilroy123 11 minutes ago

Asking the right questions.

Comment by pcwelder 2 hours ago

We just need transparency. It's okay if you change criteria, but pre-register the trigger for when you're going to change them.

Example could be, we'll update X criteria when top-N models are within 2 points of each other.

Comment by maxglute 1 hour ago

Bench switcherooing seems worse than bench maxxing, at least goodhart law shenanigans still has target everyone agrees on.

Comment by spider-mario 1 hour ago

I’m tempted to say that ideally, we’d have more in-depth, qualitative reviews of AI models so that it’s possible to form an impression of how suitable each model would be to one’s particular use case (instead of just “this model gets you 34 intelligence per dollar; this one, 38” as if they could be reduced to such numbers), but that would be a lot of effort given the sheer number and frequency of model releases.

Comment by bananaflag 1 hour ago

I personally would like more reviews of new capabilities unlocked. Like there was a lot of talk when o3 got released that it can do geolocation. Feels more interesting than being better at an existing capability.

Comment by electroglyph 1 hour ago

AA's intelligence score is based on many benchmarks, you can go look at them all individually if you'd rather focus on specific areas.

Comment by spider-mario 1 hour ago

Those are still numerical scores and, as far as I know, essentially based on “does it solve the task correctly or not”. By “qualitative”, I meant for example some analysis of how it succeeds (e.g. quality of prose or code) or its failure modes (lack of persistence, etc.).

Comment by api 1 hour ago

Tangent but the fact that quen3.8 27B even places on these top rankings versus hundred B and T models is incredible.

It’s the crow of current LLMs: tiny brain, outsized performance.

Comment by avaer 1 hour ago

I just realized benchmarks are the trading card PSA of AI.

The numbers are fuzzy and hard to verify, especially if benchmarks are private, putting authenticators in a powerful position to determine market value of AI in countless ways; a recipe for corruption. I'm not saying it's happening, I just realized the social ingredients and temptation are there.

Comment by dist-epoch 2 hours ago

Mentioned in the article: https://www.intelligence.ai

Not sure if the scores are better, but the website is nicer to use.

AA has such terrible UX, especially on mobile, trying to select a subset of models you care about is an exercise in frustration.

Comment by no-name-here 47 minutes ago

My own (first world) problem with AA: deep linking to the specific models + benchmarks has been hit or miss (for comparing a new model vs my standards).