Our position on open-weights models

The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though

I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.

I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.

To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.

Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".

Lastly: "What if they use their models in their military and local police services like we do? Communism!"

As always: https://pbs.twimg.com/media/B_AiI9_XIAA67_t.jpg?name=orig

A much simpler summary:

- open models good.

- smart models _can_ be bad

- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.

- banning open models just in US is neither good or bad: is stupid.

The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.

I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?

The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.

To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.

In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.

It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.

Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.

--

There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.

Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.

And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...

But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.

Curious if this was more of an emotional take or if there’s actual evidence behind it.

And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"

I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.

And you know this how?

Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.

"Dangerous to our bottom line"

"We call it dangerous to hype up its abilities"

If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?

While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.

Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.

1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair

2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh

If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips

Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.

Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.

In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.

I might have missed something but wasn't the big story that Dario refused the Department of War's demand to use Anthropic's models for such purposes?

Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.

He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.

He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.

Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.

If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.

No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.

From a new "fast" company arising, you can sometimes see the way that before it started up there was nothing but "narrative" and that is what established the initial business model since there was nothing else yet. After some momentum is gained whether there is a pivot or not then the narrative going forward has to be aligned with the now more-well-proven business model.

From his leadership standpoint there are 3 big recommendations right now. That's what this message is all about.

>We should not sell powerful chips or chipmaking equipment to China

Well you and who else?

If there's not already somebody who is compromising the well-being of a nation in exchange for a handful of gold, with the ever-incresing glorification of greed & dishonesty at all costs it's only a matter of time for this one.

>We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process

Wait a minute, what's always been needed by everybody are more compute-efficient processes for everything. I've mentioned this before and it's been a while but back in 1980 it took less than a year to figure out I was going to need other chips that were not regular CPUs if I was going to get the most intelligent response from the silicon on a single square-foot of PCB. At the same time it was obvious you were never going to get far without what they now call "distillation", especially with only kilobytes of memory. Otherwise you would be wasting such stupidly large amounts of memory & storage there was no way you could really call it "intelligent". Now with ML & AI on the rise again there are so many people more deeply immersed than ever, and nothing has really contradicted these basic concepts yet, which have been easily recognizable since like forever.

>All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.

Righteous concept, and I'm always in favor of 100x the amount of testing in general normally done.

But "just" test says it pretty well, and those who are gifted enough to "draw the rest of the owl" freehand can test things the most skillfully until they are blue in the face. It's not going to help if an adversary decides not to test, or to enhance these exact things so it can have some kind of competitive advantage. Amodei does not ignore this and there is a footnote about it.

People realize that some of the elements that are coming to mind were expressed in some fairly early "science-fiction" so all this is nothing new

Still looking for the most intelligent responses I can get, since way before 1980 ;)

It's WeWorse.

It does give me the faintest glimmer of hope about the people who live here, though.

The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.

Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.

Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"

It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.

Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?

I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.

If everyone has mythos, no one has "Mythos."

Just let people fix their code.

Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.

What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.

Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.

All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...

There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.

The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.

So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.

He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.

From Hassabis’s essay:

“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”

> Open-weights models that don’t have dangerous capabilities are a public good

Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document:

- Being able to research energy, robotics, or AI is an unsafe capability

- Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities.

They want to ban pretty much anything open-source that's above cat-level intelligence.

1: https://www.anthropic.com/responsible-scaling-policy

It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.

Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.

Therefore only open weight models can be allowed, since this allows genuine third party evaluation.

Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.

Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.

I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)

-The Libraries of Power

It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.

Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.

Is the pessimistic view. Their message on safety has seemed pretty consistent to me.

"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.

This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.

I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.

> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—

Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves

Please stop giving this company money, people.

Anthropic does not support a ban on open models, except for any models that aren’t closed.

> Yeah, this is anthropic advocating for a ban on open weight models.

I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.

More self-serving trash from the US AI companies, disguised as "being reasonable".

Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.

Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.

Regulate GPUs? Ban general purpose computers?

I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)

I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.

Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.

The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.

Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.

The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.

If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.

In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.

But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.

If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.

"Power corrupts and absolute power corrupts absolutely." Lord Acton

Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!

Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.

But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.

There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.

Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.

Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.

> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.

https://news.ycombinator.com/item?id=49078376

----

And related thoughts on past posts:

https://news.ycombinator.com/item?id=49034988

https://news.ycombinator.com/item?id=48516722

Bad actors WILL have access. The question is will these mega corps stop innovation?

It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.

Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.

Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.

Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )

> and cyber-offense capabilities?

You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.

The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.

If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.

But more people having access to the potential tools for defensive is the best possible scenario.

Every other scenario is worse off for everyone except for those with enough money to do something about it.

We'll be fine.

The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.

In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.

> Something should be done

and

> Something can be done

In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.

The US can attempt to stop those models from being trained in the first place but good luck with that.

The software has to be built better.

I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.

The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.

...general-purpose computers

...unbreakable encryption

...unbackdoored communication

...unkillswitched vehicles

...unsurveiled dwellings

>what should be done about ...?

nothing

>Do you seriously want this level of capabilities to be generally available with no guardrails?

yes