IBM CEO says open weight models are now good and cheap on-prem

I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.

I also die to be able to run the OSS setup locally to stop being spied on all the time.

For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.

An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.

I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.

You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.

For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.

In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.

If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.

At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.

I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).

I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness

Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?

I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?

If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?

This is a win though. But not for the reasons people might think it is.

It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.

And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.

The frontier labs are in a tough spot. It's hard to justify their cost when you can get something that's easily as good for cheaper (and anecdotally, more reliable/accessible). I do HITL so my perspective will be different than a fully-agentic setup, but as soon as I started using open stuff regularly, I realized that any moat the US labs had built is drying up.

This is what will cook frontier AI and probably prevent the investment scale to happen for AGI. Humans + good enough cheap AI is just more economic and unbelievably productive. Like why we fly on jet technology from the 1960s still with incremental improvements.

It’s why Anthropic is desperate for capital and trying to get to a scale where AGI appears. But I’m afraid they won’t get there in time and the money to build the bigger model will have dried up.

I really wanted to tell her -- what she needed was compute, not hardware. Her best bet was to rent GPU which is easily available from many services. If she was paid to run inference, cost is not a concern. If she did it for her own work, she wouldn't need to shell out a ton of money initially and then be left with hardware that she no longer needs when the work is done. In any case, she wouldn't need to worry about purchasing hardware or battery life. ssh to a remote machine is simply way superior than running inference on a laptop she carries around by any means except in a few rare occasions (e.g. no Internet access).

Same thing for a company. Why would any company who has a business to run get themselves burned in acquiring GPUs, setting up racks, load balancing etc when Bedrock/Azure Foundry etc is right there with tons of both open and closed models and ZDR? And then see hardware depreciate in just a few years?

I was able to get this related article

https://rcrtech.com/semiconductor-news/what-happens-to-old-a...

I am not a believer in winner-take-all mechanics for models (even for AGI -- look no further than the natural world for many examples), and a large F50 company committing money/strategy to something outside the frontier labs is a great thing for the ecosystem.

Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.

In the IBM-specific sense, the most unintentionally hilarious LLM interaction I have ever had was asking Granite’s 4.2 30B reasoning variant the car wash question.

Reams and reams of corporatese about why it wasn’t wrong when it was.

Is it though? You’re likely paying more on prem per unit of electricity and the racks of NVIDIA cards also aren’t exactly cheap.

I don’t see how that leads to 10%. I can see it being a good pitch if you’re in the biz of selling on prem servers though

What are the tools to look at first for usable models at that thin compute capacity and how do you go about exploring training for hobby and learning purposes?

  While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.

  “The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”

DeepSeek v4.1 Flash or GLM 3 are already very very capable models.

Only bottleneck at the moment is the hardware prices.

[1] - https://www.ibm.com/downloads/documents/us-en/17f55bd310d5fd...