Elevated errors on Claude Opus 5

Posted by flyaway123 11 hours ago

Comments

Comment by rich_sasha 8 hours ago

I've noticed something else - as Anthropic models get even more and more superhuman, they seem to serve me more and more casual nonsense.

Not like adding glue to pizza. Here's an example from today (paraphrasing): "you need to run `git merge-base branch1 branch2`. Pay attention to the order of arguments, it is important: `git merge-base` is symmetric and returns the same value regardless of the order of inputs".

So which one is it? Symmetric or not? It's not even one of those where it self-corrects, it just happily contradicts itself halfway through the sentence.

My pet theory is that these frontier models do quite a bit of brute force at the end - some kind of beam search - and silently downgrade you depending on demand or compute availability.

Still not great. This particular nugget is from Sonnet 5, default settings.

Comment by wongarsu 7 hours ago

I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models

It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still possible

Maybe at some point someone figures out how to train models with a backspace token

Comment by maweaver 5 hours ago

Have noticed similar things using it for code review. "Issue #3: This variable is defined but never used. It is actually used later in the method though. So this is not a real issue to be concerned about", that sort of thing.

Comment by zhoBEENG 6 hours ago

Diffusion language models do this. I’d guess they are the future.

Comment by bellowsgulch 2 hours ago

First we started by extending chain-of-thought to a formal thinking system. You could extend this again with a "thought scratch-pad" or "thinking draft" with n-number of passes without needing to train backspace tokens.

That being said, all of these are just variations of scaffolding on generating more tokens.

Comment by willmadden 5 hours ago

Or a strikethrough token!

Comment by sjsdaiuasgdia 6 hours ago

Yes, LLMs are prone to semantic consistency traps. I see a version of this in a system I have that transcribes a lot of noisy low quality audio (radio comms).

Sometimes it works out, in that an unrecognizable word or two is replaced with reasonable assumptions based on the semantics established by the words that came before the signal degraded.

But the bad audio might also result in words that don't align well to what came before, or represent alternate (mis)interpretations. Now this is part of the context and the next several tokens align to this new path regardless of what is said in the audio.

In pre-LLM transcription, you might get a nonsense word or two when the audio transiently degrades, but the specific meaning of the nonsense words doesn't influence the transcription of audio following the degradation.

Comment by sunaookami 8 hours ago

I also see stuff like this with Opus (4.8 and 5). It does the work correctly but then the explanation or "aftermath" description is kinda nonsense. It feels like they are tuned hard for agentic coding and everything else is getting worse. There are people that say older versions are also better for creative writing which would confirm this theory.

Comment by Culonavirus 7 hours ago

> they are tuned hard for agentic coding

Well you can't blame them, it's what brings in the cash. Other uses will suffer proportionally.

Comment by o10449366 7 hours ago

Claude has always noticeably degraded under heavy load. Opus goes from "decent to work with" to "dumb intern" depending on whether you're working at 3 AM west coast or 10 AM - 5 PM. It's part of why I cancelled my subscription - "max" plans and "extra high" effort are meaningless when there's so much variability between model availability, model performance, and harness bugs every day and every week.

Comment by brookst 7 hours ago

I don’t find Sonnet usable at all. Especially for the price. I’d much rather get half the usage of Opus.

Comment by croemer 8 hours ago

Incident is claimed to be resolved but I'm now (11:22 UTC) getting:

   API Error: 529 Overloaded. This is a server-side issue, usually temporary — try again in a moment. If it persists, check https://status.claude.com.

And the status page says all green.

Update 11:27 UTC: I saw the error first at 11:22 UTC. Retry at 11:27 UTC still failing. Status page is still green.

Update 11:28 UTC: Incident has been declared dated 11:27 UTC https://status.claude.com/incidents/mfdtrknpxghq

Comment by Khaine 8 hours ago

I'm also getting this error

Comment by matheusmoreira 7 hours ago

So, how's the OpenAI situation? Is the grass greener on the other side?

Comment by seunosewa 7 hours ago

GPT 5.6 Sol is so free of drama for me for brainstorming, writing prompts, etc. Tasks I always did with Opus until 4.7.

Comment by vintagedave 6 hours ago

Yes. I switched to Codex some time ago. I feel like I get fewer outages (had one over the weekend), and coding results are good. I built my own agent harness using OpenAI models too and feel it was the right decision.

Comment by croemer 6 hours ago

And here we go again, new incident started at 13:39 UTC: https://status.claude.com/incidents/rkk5x44tndw9

Third outage today. Per https://downdetector.com/status/claude-ai/ the people affected grows every time: first one had peak 19 reports, second 24 and now it's already 39.

Comment by rvz 8 hours ago

Claude decided to take an extra day off, after going on vacation yesterday [0] and when Codex went and took a longer break the day before that.

No wonder the amount of water that both Claude and Codex are taking they also need so many frequent hydration breaks.

[0] https://news.ycombinator.com/item?id=49056739

Comment by ModernMech 8 hours ago

And I was told that AI is superior to humans because they don't need rest!

Comment by ACCount37 7 hours ago

Can we not do the "AI is drinking all the water" bullshit at least on HN?

Comment by breezybottom 6 hours ago

If you want a safe space, try the Fox New comments.

Comment by rvz 6 hours ago

There are many sources that already document it being true so we both know it is not bullshit. [0][1][2]

So you think that the water that comes out of these data centers is safe for humans once released and the mass consumption of them is not a concern?

Sounds like a way to sweep this environmental issue under the rug.

[0] https://theoec.org/news-and-information/behind-the-data-boom...

[1] https://fieldreport.caes.uga.edu/publications/TP121/how-data...

[2] https://www.wsj.com/tech/ai/ai-data-centers-water-use-901e29...

Comment by ACCount37 3 hours ago

Sounds like a way to distract from the real environmental issues with yet another "plastic straws" debacle. A fake issue that makes for some nice headlines in the media - but when you run the numbers, it obviously doesn't matter and never did.

Distract from the real issues like, you know. Clean energy projects in the US being throttled on federal level due to lobbying by the oil companies - lobbying that borders on regulatory capture. At the time when the oil prices are rising globally - and the demand for energy in the US is rising too.

Unfortunately, you can't stop an environmentalist from throwing his weight behind the cause that's being promoted the loudest. So influence groups can just push one meaningless "headline cause" after another, and keep them distracted from anything that matters in perpetuity.

Comment by exac 11 hours ago

Déjà vu.