Kimi K3.1 scores 55 on Artificial Analysis, placed on the Pareto line

Huh, this is an interesting hypothesis! It would make sense, to a degree. Part of "big model smell" to me is the model being able to pull a huge range of interesting algorithms, architectures, mathematics, and everything else that would come with greater world knowledge and having a much more sort of innate, like, full, well-rounded understanding of it instead of surface-level ICL pattern matching on web results.

+1 for the omniscience benchmark!

Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.

That definitely matches my experience, I have found Astra to be more capable than many other benchmarks show, have been very impressed with Argon for text, and generally like Sol. I find Haiku and Sonnet to be not worth my time as compared to the speed and cheapness of something like DeepSeek.

The only exception on the chart is DeepSeek - it scores exceptionally low there, which doesn't exactly track my experience. It usually does what you tell it to, and it's not exceptionally intelligent, but I do find it quite good at what it does (the bulk of code, scripting, etc). I know a lot of folks like it too ... curious if I'm just not exploring the simpler models enough?

Yeah that tracks completely with my experience as well.

I always found them too chatty and verbose. I'd get a wall of text and miss things because it was lost in garbage about load bearing and blast radius

This seems more a measure of anti promp injection than general intelligence

I agree, but my point wasn't about being able to use a weaker model, it's about benchmarks not being in any way representative of the actual quality/level of the model. Of course for a lot of scenarios Opus is overkill and Qwen/DS are more than enough.

I think also heavily depends on what you're doing.

If you're in a stable codebase with lots of best practices, patterns, controls (linters, analyzers, etc.), examples already established, then any model will do ~the same plus/minus since there's not much novelty.

If you're building ground up, entirely novel, no reference, I think results will vary with lower capability models.

For me, I do a mix, leaving implementation to Haiku/Luna/DeepSeek and research/feedback/critique to Opus/Sol/Astra/Fable

Yeah, but if you need to provide less details to Opus to get the same output it means the model is better at deducing your needs, which again implies it's a higher quality model.

I spent the last 4 months on a mix of chatgpt and glm building some prototypes of product ideas I've had for years because claude was in the phase of obnoxious conversation and verbosity.

I took the output and original specs and handed them to Claude today and the level of insight and concise detail in it's criticism and problem solving was fantastic. Dropped the others immediately

It's not just 5.5, every iteration of the frontier moving - Opus has had some edge. I'm sure it's not 100% in every scenario, I love interacting with Codex models more, but the quality of the output, in most scenarios that I've been exposed to, has been the best.

I use that specific model as my primary LLM for coding and diagnostics on projects in Python, C++, CUDA, and for web research.

I'm also using it with my own harness (you mentioned "in my harness"), so I'll share two things I found about using open weight model providers. The first is that getting the configuration right isn't a per model thing, its a per model per provider per point in time thing. The point in time thing comes from middleware updates as those introduce or fix defects that can very seriously impact inference quality. The second thing is inference quality varies a lot between providers, even when they are ostensibly using the same quant of the same model.

If you want to try it something else for comparison I open sourced my harness a couple of weeks ago: https://fluximp.org

I recommend trying deepseek v4.1 flash from together.ai (not the 1017 one the one released last month). deepinfra.com is cheaper for that but its noticeably slower and the inference from together.ai feels like it might be better (not quantified; just vibes).

I can have 2-3 sessions running at the same time for pretty much the entire working day with DSF4.1 where the responses are fast, cost is pretty negligible, and the results are good enough 95% of the time.

On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.

Yes, people should say if they are using it for HTML/CSS/Javascript or Rust/C++; it makes a huge difference.

Coding. What were you using last year? It's better than that.

If you have well-defined, long running tasks that are relatively disposable but requires constant ML monitoring and overwatch e.g. web crawlers, monitoring prices, logs etc. DS4 Flash is perfect for those.

For tasks like crawling and real time log analysis etc., the costs really add up at scale.