Benchmarking Opus 5 on SlopCodeBench

Posted by dhorthy 10 hours ago

Counter274Comment62OpenOriginal

Comments

Comment by robbomacrae 1 hour ago

SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs.

I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs/do-skills-improve-coding-agent-accu...

There is a small but growing community on discord for discussing SCB so if interested please join https://discord.gg/BrC4BA9sVj

Comment by WilcoKruijer 28 minutes ago

I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions).

What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.

Comment by mellosouls 52 minutes ago

This is nice, but would be really useful measured against human performance, and yes - I understand that is likely a difficult challenge.

Many people though are going to read the headline figures and think it means - say - Opus 5 is only a quarter the strength of a human coder.

Comment by Vgoose 8 hours ago

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.

It's especially relevant now that models are good enough to solve ~most point-in-time problems.

Some relevant but disconnected thoughts:

- deterministic scores are so nice

- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently

Comment by NitpickLawyer 4 hours ago

> another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system

State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a complex state and allows visibility into it, and possible manipulation in a structured way.

Comment by nicoty 1 hour ago

I've been thinking a lot about this recently. In my case: how do I get an agent to see the important parts of the current plan and get it to stick to it without deviating, especially as loops get into longer and longer cycles and compactions erase prior context?

I think the problem with just encoding a whole plan in a single markdown file is that it gets polluted really quickly (agents can stop adhering to instructions to keep it clean and conform to a specific structure), which makes it harder for agents to see which parts of the plan they should focus on. As such, I've reached a similar conclusion about giving the agent access to CLI tools to help them deal with this. To try to mitigate this, I've been getting Claude to develop for me a CLI tool that:

1. Scaffolds reusable, structured plan templates and a reusable workflow that structures how to tackle the plan, step by step.

2. Validates that the plan files still conform to the correct, parseable structure.

3. Parses and evaluates the plan files, to determine what the current state is and what the next valid transitions and states are according to the workflow, like a state machine, and outputs instructions and reminders for agents as to what they should do at each step of the workflow.

So far, I've been dogfooding the tool and it seems promising: I can leave Claude running for longer and it doesn't drift as much. However I haven't ran any benchmarks yet and I'm still not entirely happy with the state of the codebase ( https://github.com/nothingnesses/agent-scaffold ), so take this with a grain of salt.

That said, I'm also bullish on using agents with formal methods and proofs. Type checkers and compile-time checking in general are great because they surface errors early and with great specificity. So if you can encode your specifications with, e.g. dependent types, you can use the type-checker as a way to steer the agent when it goes wrong and gets off-track.

Comment by dhorthy 8 hours ago

> - what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

this is a nicely succinct way to put this - a multi-dimensional space where no single metric is really useful

state space of the system is interesting too. I would guess that for any production software with dependencies like databases/third parties that might be too hard to measure, but if you can silo off parts of your system into bounded state machines, it may be a value metric on some module behind a clean interface.

I think the kubernetes control loop model is a great instance of this, a handful of scoped components that own a control loop across a well-defined state machine, that can operate / recover in the face of most network partitions or downtime - the promise of CRDTs but rather more a pragmatic approach to it

Comment by sothatsit 8 hours ago

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.

I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

Comment by rubicon33 8 hours ago

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

Comment by jbird99 4 hours ago

At this point I don't even bother with it. Constantly falls back to Opus anyway, so I may as well save myself some time.

Comment by swader999 7 hours ago

Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes

Comment by Espressosaurus 7 hours ago

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else.

It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

Comment by conception 6 hours ago

https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.

Comment by nerdsniper 4 hours ago

I mean they could just be routing known benchmark questions (which all of SWEBench are) to a full-performance variant.

Comment by Gareth321 2 hours ago

I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the performance? They get fantastic benchmarks and glowing reviews on release, then slowly squeeze more performance out of the model. By the time the next model is ready for release, the jump feels quite large again.

Comment by fastball 8 hours ago

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

Comment by ValentineC 6 hours ago

Medium or low supposedly prevents Opus 5 from overthinking:

https://xcancel.com/danshipper/status/2080700057892815114

Comment by conception 6 hours ago

https://cognition.com/frontiercode

Quality vs cost - medium is the sweet (perhaps better too!) spot.

Comment by dhorthy 6 hours ago

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward

Comment by sothatsit 7 hours ago

If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.

Comment by dhorthy 7 hours ago

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap

Comment by 7 hours ago

Comment by 4by4by4 4 hours ago

I'd be curious to see the raw test results.

I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression.

There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper.

I think a cool experiment would be to adjust the order of the features implemented (e.g. checkpoint 3 then 2 then 5 then 4) where dependencies allow it. Then one could account for some checkpoints being more difficult than others.

Comment by dhorthy 4 hours ago

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

Comment by Johnny_Bonk 9 hours ago

I haven't joined your chats in a while but glad to see you put this together, I truly feel as though opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing

Comment by dhorthy 9 hours ago

yeah this was just a start - the fastest cheapest thing we could try for a brand new model.

I'm hoping to do some more work with sol/fable in the mix as well as exploring more languages and curating the problem set to include more of the benchmark

I also kinda felt like opus4.5 was dumber than 4.1 personally, maybe a little biased since 4.5 was 2.5x faster and 2.5x cheaper seems to indicate its a smaller model

Comment by Johnny_Bonk 9 hours ago

yeah i agree

Comment by dan_gee 9 hours ago

[flagged]

Comment by scrollaway 9 hours ago

How is this useful or insightful?

You ever go to forums full of entomology specialists and tell them you don’t understand their fancy terms?

Comment by dan_gee 9 hours ago

My point is that the differences between these models are so minor that obsessively benchmarking them comes across as navel-gazing.

Comment by joatmon-snoo 8 hours ago

The evidence that proves a model is actually a step function change is these benchmarks.

If a model isn’t a step function change? Welcome to research.

Comment by 7 hours ago

Comment by Johnny_Bonk 8 hours ago

like all good science, measure everything

Comment by piazz 4 hours ago

Great writeup. The excessive function thing has always driven me crazy; I guard against this explicitly in Claude.md.

I have found that models are generally poor at managing refactors / complexity while also implementing new features. But I’ve had some success with a semi-lights-off approach where you decompose it and prompt the model adversarially in a second pass to look for new rough edges and areas of complexity or refactors that might simplify the codebase.

So I’d be very curious to see this benchmark but with something like a periodic “refactor turn” interleaved in.

Also eager to see Fable benchmarked; anecdotally that was the only model whose code I felt I could actually trust to not review closely.

Comment by killingtime74 9 hours ago

Did you not benchmark latest GPT 5.6 or GLM 5.1/Kimi K3 because of cost? I can run them if you share how you ran them

Comment by dhorthy 9 hours ago

no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off.

> fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me.

> i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try it. What do you need from me?

> Actually I changed my mind. I want to do two of the easy ones you picked and then I want you to pick the one with more checkpoints, maybe one of the harder ones, not the very hardest one but one with a higher number of checkpoints.

> Actually let's do one easy, one medium, and one hard problem please. If we have a hard problem I'd like to see that.

> lets rock - i think lets just do opus 4.8 and sonnet 5 and opus 5 since we our ZDR will block fable

Comment by killingtime74 6 hours ago

Thank you!

Comment by ajwin 8 hours ago

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

Comment by igregoryca 7 hours ago

At least for Claude Code, putting "run /simplify at the end" in an "implement the plan" skill helps a little. It still often leaves new code in bizarre places, and/or with bad/alien-sounding names and comments.

Comment by dhorthy 5 hours ago

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

Comment by dhorthy 7 hours ago

I agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture.

For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too

Comment by lostdog 5 hours ago

Clearly the first step of slopbench2 should be to have the agent first write its own harness!

Comment by 6 hours ago

Comment by drcongo 18 minutes ago

> [...] with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges.

Is this bad? I have McCabe complexity switched on in Ruff and find it a handy watermark for when something should be broken up into smaller, individually testable callables. Five times as many callables could make for much more readable and testable code.

Comment by willsmith72 3 hours ago

> The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper.

a 41% improvement is not much higher? come on that's just doomer

Comment by well_ackshually 2 hours ago

If you tell me that 76% of your code is dogshit but that's it's a massive improvement over your previous 83%, I'm firing you.

Comment by 1123581321 23 minutes ago

I’d enjoy working with such a person, a bleak Eeyore-Gilfoyle.

Comment by subarctic 4 hours ago

I still don't have access to Opus 5. I'm on the latest version from Homebrew, I guess the update hasn't made it there yet

Comment by sunaookami 2 hours ago

Have you tried running /model claude-opus-5?

Comment by sethherr 4 hours ago

Why are you on the version from homebrew?

Comment by dhorthy 4 hours ago

somebody get this man a curl-pipe-bash stat

Comment by cesarvarela 6 hours ago

Please add Fable; a good benchmark should show that Fable is less prone to just autocomplete and instead pushes back or is at least more tasteful.

Comment by dhorthy 5 hours ago

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

Comment by weiliddat 1 hour ago

For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting.

Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2

Comment by dcl 9 hours ago

finally the benchmark for me

Comment by dhorthy 9 hours ago

i hope that is because you hate slop and not because you write it

Comment by akurilin 7 hours ago

Really hoping that all of the attention you're bringing to the longitudinal sloppification of codebases makes it back to the labs and creates some pressure to improve that trait of the models. This new benchmark seems promising.

At the same time, I imagine it will be hard for them to prioritize this over improving flashy one-shots of impressive zero-to-one feats that demo so well and attract more customers.

Can we have simonw make "pelican on a bicycle after 1000 requests for iteration" popular?

Comment by dhorthy 6 hours ago

i laughed at the pelican bit its good

yes the labs will always prioritize the vibeslop dopamine casino as far as I can tell - making the models useful and addictive for unsophisticated users, sometimes at the expense or at the very least at the ignorance of the needs of power users

Comment by knighthacker 8 hours ago

This is where Opus 5 shines

Comment by tamimio 7 hours ago

So many benchmarks more the models themselves.. just make one unified standard to benchmark all or stop calling it “benchmarking” as this word lost its meaning.

Comment by cute_boi 8 hours ago

Opus 5 is an overconfident stupid model. It tries to generate too much slop, tries to act like everything will fall. I have reversed back to fable and codex sol.

Comment by dhorthy 8 hours ago

yes sol is still my daily driver for most coding tasks

I did find opus 5 quite handy for general knowledge work and visual design, without the cost of fable (e.g. the graphics in this post are made by opus 5)

but its not noticeably better than opus 4.8 in those regards, and I would not miss it if forced to go back to 4.8

Comment by roncesvalles 8 hours ago

I almost prefer Opus 4.8. Opus 5.0 has the overly scholastic tone of Fable but without the intelligence.

Comment by patwolf 6 hours ago

We've been running automated code reviews on claude with a bunch of skills/subagents with different specialties. Any review feedback is then fed back into claude to fix. Since switching to Opus 5 I've noticed the reviews are overly pedantic, and that leads to feedback loops where each fix generates more feedback, which requires more fixes, i.e. slop. I had, for example, a simple SQL migration script with a single CREATE TABLE. After a few rounds of review, it ballooned into a complicated 200 line script.

I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.

Comment by latentsea 4 hours ago

> I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.

This is a known failure mode. Sadly, working on a software engineering team doing agentic engineering now means we need to build and maintain suites of evals that measure these things, so that we can measure the effects of changes to harness components including how they perform under model upgrades.

But... for a traditional software engineering team that has no experience in this... How do we even do it?

Comment by sqemo 5 hours ago

[flagged]

Comment by joka88xj 4 hours ago

[flagged]

Comment by delbertty 6 hours ago

Mostly a harness problem in my experience. Slop accumulates when the agent can touch anything, so constraining it to one seam and having it add alongside rather than edit in place does more than model choice.