DeepSeek V4 Pro beats GPT-5.5 Pro on precision

Per Merriam-Webster [^1], a lede is:

> the introductory section of a news story that is intended to entice the reader to read the full story

(Emphasis mine)

You may prefer more matter-of-fact phrasing, of course, but criticising a lede for attempting to achieve its goal is unjustified.

[^1]: https://www.merriam-webster.com/dictionary/lede

I found the writing clear and quite even handed. The lead is a bit salesy, but leads typically are. Knee-jerk dismissals based on vibes that something is LLM generated are quite low-effort.

It shows DeepSeek is competitive, if not better sometimes, than GPT 5.5. Also shows there is no moat. As such it is a highly significant signal.

No one ever says this about the “pelican on a bicycle” metric

The remaining 5% of time you get a big boost for your high-reasoning problem solving needs and evade a lot of pain. Now, I just need to be able to predict accurately when I need this extra 5% and when not :)

I wonder if I am using the same models as everyone else. To me, LLMs still give good answers 80% of the time, but 20% it fails in such a miserable way that makes it obvious that the "intelligence" is not there.

That being said the models still surprise me with a broad range of hallucinations, lack of epistemology or common sense or inability to follow instructions on a daily basis.

Today it was trying to get opus 4.8 to just follow a simple architectural pattern for controllers in a rails app. It was pulling teeth out of a shark.

Already the fact that we could have to ask "there where", the fact that we have met clearly unintelligent bots, creates a requirement about defining where it (intelligence) is and investigating what put it there, to get the warranties that intelligence will be met consistently, structurally, and not casually, apparently.

Casual use, casual tool; mission critical use, certified tool.

not really. it happens in training and RL. your harness is not going to override what it has been trained to do.

sure harness is useful if you are trying to build crud websites if model is trained on stamping out crud websites. But thats just a waste of time remxing things better.

We are just getting into the nitty-gritty of LLM benchmarking - to be fair they still need to go a long way still IMO. But it's incredibly exciting that a local run LLM is capable of producing similar results as a SOTA model.

What? You can and you should. That's exactly what product tests are enabling you to do. If you need a glue, you want to look at someone who tried to glue some things with few glues so you know what to roughly expect form which specific glue.

> With $3.88 & 690,003,591 tokens and 5 hours, Deepseek Pro & Flash combined, managed to reverse engineer Teamspeak's Licensing System for 3.13.8 (latest of post)

https://www.reddit.com/r/DeepSeek/comments/1txcfrh/with_388_...

9 bugs is probably a bit low of a sample size to get a ranking.

That being said the ranking does end up roughly how you'd expect.

Deepseek is Pro, right? Not Flash? I've been using Flash for a lot of smaller tasks and finding it reasonably good. It's good for "interactive" use. Very fast, does small tasks nearly instantly.

It's also decent for investigating large codebases. I wonder if it could do security work too.

And nice to see the cheap models doing so well.

I don't know whether models are over fitted to benchmarks and people take them at face value, but I spend less on DS4 apis than I do for Claude Code 100$ subscription and I code everyday. So far I'm quite happy with the results.

It's maybe not quite as knowledgeable as the most expensive American models and maybe makes more mistakes (just a feeling based off of vibes, don't take my word for it), so you need to constrain its scope more. That suits my workflow, half the time I have it generate code in the chat window and then write it myself, and I'm mostly using it at the level of generating function bodies and stuff, not entire features. Although it is writing a lot of SwiftUI without me really knowing the language and doing a fine job as far as I can tell (which isn't much admittedly).

One benefit I don't see talked about is it's speed - it's really quick, doesn't spend too much time reasoning even on "max", and the flash model is pretty dang good too. This lets me get into "flow state" when I'm writing code, compared to my experiences with Codex and Opus which would take minutes to complete even basic tasks and kind of ruined my focus.

It's so cheap though, you could download a different harness (Crush, OpenCode, Pi etc) and load $5 in credits and test it for yourself.

    export ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic
    export ANTHROPIC_AUTH_TOKEN= *** PUT YOUR DEEPSEEK KEY HERE ***
    export ANTHROPIC_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_OPUS_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_SONNET_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_HAIKU_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_SUBAGENT_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_EFFORT_LEVEL=max

> I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US)

Do you mean you don't want to use the models created by a non-US lab? In that case, yes you're stuck with US models, but there's a half dozen big labs in the US. If you meant just where your inference is done, there are providers in 12 different countries through OpenRouter, including the US. Several subscription providers host in multiple countries. There's a lot of choices.

As usual, different models get stuck on different things. I run DeepSeek v4 API for most of my Cursor experimentation / poking around / proof of concept stuff, but I trust it less than OpenAI/Claude for writing production code. Sometimes DeepSeek is great for debugging, planning, etc. Sometimes it gets stuck or outputs low quality. That's true of OpenAI and Anthropic models as well though.

Overall, DeepSeek seems serviceable but a rung below Opus 4.8 and GPT 5.5. I run them all on maximum thinking settings.

Repo reference here: https://github.com/aravindhsampath/agentic-template

DeepSeek and Xiaomi's deals on cache reads go with their models' latest gens making caching cheaper (using less space for KVs). No open-model inference provider has decided to match the pricing. I'm sure that says something about how inference pricing works, but not completely sure what.

Agree with others that top open models aren't on the frontier, and I would expect differences doing big-picture planning or anywhere you're only giving broad brushstrokes and looking for a lot to be guessed. But they do seem fine at coding from a a concrete plan! No experience in huge codebases because I only use them outside work, but they seem good enough about gathering info before they dive in that I'd expect them to grep around as they need.

An annoying caveat: individual subscription plans, used heavily, are much cheaper than the API -- see https://she-llac.com/claude-limits -- which complicates any argument about cost. I still think open models are worth playing with. They're one of the things that let us treat this as a technology rather than just as the product offerings of one of a few companies.

Hardest stuff i threw at it... i did like a set of 3 each for claude/gpt/ds, it was all pretty steady across all providers. I think claude won but it could have just been it rng'd into the 3 easier tasks, they are all similar tasks but not identical, these aren't like benchmark tasks just a steady flow of annoying html/json/regex type stuff. Almost always they need a second pass regardless of what model i throw at it, just to tighten up some loose ends, and it fit right into what my current expectation was of gpt 5.5 and opus 4.6.

For evals in particular (tuning workflows that agents are using), effectively not having to worry about price is an incredible multiplier - getting statistical significant signal is not cheap otherwise.

https://artificialanalysis.ai/evaluations/omniscience

Esp check the Hallucination rate for Deepseek - it's not good.

if it's 99.9% comparable performance for less money I'm interested, but I'm skeptical it's there

Expectations are not always reality. Give the model a try. I just stuck with flash tbh, didn't even use pro. I do webdev in PHP.

It helps but you often have to step in the failure cases and guide them or forcibly fix certain paths to get a solution.

At the moment of writing https://news.ycombinator.com/item?id=48343690 MiMo V2.5 Pro had a lower cache hit ratio. From the article:

OSS models, depending on who you use them from, make a huge difference, mostly due to cache-hit rates.

  Model                   Cheapest effectiveInputPrice (Provider)  
  MiMo-V2.5-Pro           0.3720 (Xiaomi) 
  DeepSeek V4 Pro (Max)   0.0560 (DeepSeek)

We've been using it for async "heartbeat" processing and sms replies, but it's just too slow for live chat replies (which is a shame, as I'd really love to use it there).

Very capable model, but also very slow.

Requests to grok-4-1-fast-non-reasoning now silently route to grok-4.3 (a 5x more expensive model), with reasoning set to "none".

https://docs.x.ai/developers/migration/may-15-retirement

TFA was published today, which implies grok-4.3 was used.

Hopefully this dynamic continues long enough to make local/private inference the leading solution for coding.

So it doesn't surprise me at all that the methodology is weak, too.

1. MoE (nothing new here, but, this helps a lot)

2. Compressed Attention Mechanisms (this is their core innovation) - this dramatically reduces the Key-Value (KV) cache requirements for longer contexts

Another thing that helps is significantly lower energy costs in China.

Another point from my own guess: they are running (some percentage) the inference on their own home-grown AI inference chips.

My guess is that they do aggressive caching / some proprietary optimizations in their hosting setup that they haven't published. Maybe also running at loss to gain market share.

And judging from latency / network performance, I don't think what you access, when you access deepseek.com from Europe, is hosted in China.

Also, which SOTA western models are you comparing it with? Just to give more flavor.

"Harry finally had control of the broom. Draco was dead in his sights. The matchup feels earned."