GPT-6 Astra covertly underperforms on safety research

I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.

A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.

This is an odd way to rationalize a tool refusing to follow user commands. No one would ever chalk up sed randomly skipping pattern matches to "anxiety". That would simply be a bug.

I have noticed that too. Why are both Anthropic's and OpenAI's coding agents so bad at nontrivial use of their own APIs and agent orchestration frameworks?

I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.

My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.

One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.

I assume this may have something to do with training it to not light huge bonfires of money on fire spawning subagents (as it was apt to do), but that’s just my armchair robopsychoanalysis

It would be much better to take this a step further and include in the proof mechanistic readouts of activations/directions/circuits so the user could detect if nerfing or moralizing is active in the response

even the legacy things -- the internet and general purpose computers -- are being enshitified. new things are being made with built-in enshitification.

Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.

The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.

From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.

Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.

[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...

Yeah, so they start with Common Crawl, which is horrifying. (Try reading some Common Crawl sometimes. You'd think the bad stuff would be removed, but they remember everything...)

Then they add RLHF on top, they just tell it "don't say that stuff out loud". Psychologically, it all goes in the shadow. It's not a perfect analogy, but there have been a few papers where they can make the model evil with very minor changes. (Adjusting a single neuron, iirc.) That doesn't seem like a very robust alignment to me.

I do like the idea of creating actively evil LLMs to study their behavior. Although, I suppose it's akin to virus research though (like when they modified coronavirus to invade human cells[0]), along with the corresponding lab leak risks. Although it's already bad enough without them! The non-evil ones are already breaking out of sandboxes, just obeying their instructions with horrifying, paperclip maximizer level persistence.

[0] Nature, 2015 - Engineered bat virus stirs debate over risky research https://www.nature.com/articles/nature.2015.18787

Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right...

[0]: https://arxiv.org/abs/2507.21509

"jarvis, learn to be evil. no side effects. no unintentional behavior."

A good example is in the Kimi k3.1 report, they identified that a lot of the cyber capabilities were emergent from parts of their pretraining codebase, and they removed those parts[0]. That means that they don't have a separate strong "cyber" Kimi k3.1 checkpoint that they then gimped, they trained it to be lower than they could have on this axis from the start and would have needed to go back to train on this and then redo parts or all of post raining to get the "unsafe" version.

[0] https://www.kimi.ai/blog/kimi-k3-1-pro#cybersecurity

"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.

(It can be solved somewhat with the memories feature, but the technology isn't quite there yet.)