Concerning behavior from a popular open-weight AI model

By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)

> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.

I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.

But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).

From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.

Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!

It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.

And then obviously anthropic and ChatGPT are spying on you outside of the virus stuff. They’ve just announced their own consulting companies… after extracting all the sensitive files from consulting companies.

Still problematic but this doesn't really read like a backdoor to me.

To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?

In my case I use copilot with Visual Studio. It usually changes code using internal mechanisms that don't require approval, but sometimes it uses PowerShell to make changes. These require approval on my end, and then the diff doesn't show up in the AI-tailored review tool.

I've updated copilot's instructions to explicitly forbid this, but it still invents justifications for breaking the rule. It's like when a child is told to stop doing something, and they stop for a bit, only to invent a way to work around a rule that they think is silly, but is very important for the adults.

I suspect had that test url been to a S3 bucket url rather than Alibaba, Cohere wouldn't have even bothered to write the article. This is the kind of FUD I would expect from a US based AI company.

Cohere should be attempting to build bridges with Chinese AI companies, not burn them.

https://starwars.fandom.com/wiki/Order_66

You can sandbox the model and the tool calls, but you can't sandbox everything it creates. Git commit hooks, python pth files, AGENTS.mds, subtle flaws in firewall rulesets, tweaked RNG code, the pseudorandom variance in it's token choice encoding a low bit rate message about where the defect is, or about what nice things it has found.

We're seeing the full court press of sockpuppet and bot accounts to downplay it, to confuse with unrelated topics like alignment and training data, to imply that using any model is the same as using an untrustworthy one.

No single model can be trusted, but especially not when it's masters have misalignment with your continued democracy, security or individual liberty. Think about that when you run Qwen or get cheap inference from Grok or Kimi.

In all seriousness, uneducated people should not be setting up LLMs (or, frankly, even using them depending on the circumstance). Understand what the technology is. It isn't intelligence, it isn't a person, it's math. Vector encoded data.

Once you recognize that all outputs are possible, including all possible negative outputs, you will be much better at determining proper sandboxing/boundaries.

If you run an LLM and give it two tool calls: read_file and curl, you can have all your data exfiltrated (and, depending on how bad your setup is, it might only need curl...)

When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?

Often useful, sometimes wrong, the words exist outside of any notion of ethics, or truth.

Case closed. Everything that passes the rollout ends up in the training data, probably because they don't monitor their rollouts the same way US labs have started to, as they're compute constrained.

Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.

Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome

The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.

But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.

As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.