So you want to use OpenRouter?

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.

OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.

I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.

Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).

  if you want to control the routing, you'd lock down providers

  This is quite literally the point of OpenRouter.

  I don't think anyone serious is just willy-nilly making individual requests against OpenRouter

  If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRoute

Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often have terrible quality). Don't they monitor for simple things like this? Their own logs show how clear this pattern is for some providers. Simple cache % stats would allow them to block providers nearly instantly.

> The same model will benchmark very differently

Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.

BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.

The best part about OpenRouter is 200 OK is probably hardcoded into their responses.

I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.

Yeah, this is pretty accurate. Some providers are basically scams too. I encountered one provider for GLM-5.2 which ran at 200 tps (absurdly high), and was so broken that it would issue 20 full reads of the same file in one turn and quickly jump up to 300~400k context and charge me for the prefill. Went straight on my deny-list, but they got a few dollars out of me first.

Another common annoyance is having a request go to a provider that dribbles out ~1 tps (even for small models like DeepSeek V4 Flash). If you cancel the request, you still get charged for the prefill and the handful of generated tokens. If you don't cancel the request, you might be waiting 10 minutes for the turn to finish.

The overall experience is pretty good, and it's the best way to try new models, but they don't appear to do any real vetting or apply any quality standards to their providers, and occasionally it bites you.

This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!

Pretty damning. We're using OpenRouter for some research tasks and it's making me question everything. I suspect lots of the model providers are running into issues like these: https://forum.level1techs.com/t/why-your-local-llm-feels-dum... (HN discussion: https://news.ycombinator.com/item?id=49402232)

Co-founder and COO of OpenRouter here.

Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).

Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:

1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.

2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.

These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.

Some quick thoughts on the article itself:

1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!

2. That is bad and we will fix it. Sorry.

3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".

4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.

5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto

6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...

7. Will investigate.

8. We attempt to heal these, but obviously missed some. Will fix.

9. We do not rate limit by IP. Would love some more information here, as that is very surprising.

10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.

  this is amazing!!
  [8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
  [8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
  [8:20 AM]We should hire this guy.

That looks absolutely horrifying. What are the alternatives ??

I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.

The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".

Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!

Yea I can believe all these. I've personally have been having issues on these points:

"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.

"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them

The big advantage of OpenRouter IMO is that you don't have to manage keys, URLs, and subscriptions with 10+ providers. You definitely should still pick providers and pin them, at least if you care about repeatable results.

<Rant> I have been working on a product for months and for 1 of the specific models have become the number 3 user and very likely soon the number 2.

Its a google model. The edge cases are crazy to deal with and have taken a long time to find. I am also finding that since it has no fallbacks but no published rate limit I am single handedly taking the model down on what I thought were a reasonable amount of request. There is no other fallback that isn't google. I'm worried that with sustained usage from my users on launch in a few days. Clearly google can't be making that much money on it it im like 10% of the usage and the top 2-3 user of it spending several hundred a month in pre launch testing.

So why would they care to help a small time start up? I will have to jump to a cost effective different model and find the footguns all over again at somepoint.

Not sure where im going with this but just wanted to let others know </rant>

"OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task)." why I haven't never seen it? Click on the link brings nothing! Benchmark is only for the model. Per provider is a performance matrix (latency, throughput) What did I miss?

--

Update: oh, that is the AutoExacto Benchmarks! Now I see it.

I've had the same experience in my personal use. Unreliable output, thinking token leaking into the conversation... Time to first token and tk/s also vary wildly per provider.

The worst case of hallucination I had, was DS v4 flash switching to Italian mid-conversation and impersonating a podcast host for no reason.

We're very happy with OpenRouter in production, although we use it in a pretty limited fashion. We have only ~10 providers allowlisted (based on their location and ZDR policies) and we use only a few models, often frontier models. Has been very reliable for us in production so far. It's generally been great for us to be able to have a single place to manage spend, policies, view logs, etc. Also really nice in code to have a single-line change for switching models. OpenRouter's console is so much better than what you get from the actual frontier labs, and it's nice not to have to translate numbers and concepts across providers.

> The same model will benchmark very differently

This is surprising to me. Does anyone have a definitive answer that accounts for this difference between providers?

Do the benchmarks that Open Router runs not account for the stochastic nature of the models? Do the providers lie about quantization or context window sizing? Does Open Router not take into account variations for a given model?

I could understand latency/cost benchmarks varying but not the actual generated token responses of the models given the exact same model parameters.

Open router have one more annoying disadvantage: credits you buy expire in around 1 year +/- ~30 days. They do sent warning email 30 days in advance, but then just make them disappear.

If you for some reason had unusee balance and forget about it; its just gone.

LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder.

So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!

There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do. Even AWS Bedrock has a Claude that feels different to me, but I haven't had a chance to do actual benchmarks that would show that.

Former Openrouter user, unhappy to report that Openrouter will expire your tokens with very little warning with absolutely no business justification other than they can. If they were a bank and tokens money, their return is -100% interest after a year. Don't expect your tokens to be there for you after a year of no use....

I noticed this with DeepSeek Flash and DigitalOcean. They are doing something seriously wrong when serving that model; we need something like an SLA, but for intelligence; it's almost fraudulent behavior.

I was just trying deepseek v4.1 flash on OpenRouter. After running reasonably well for a while, I return to the window and see that my scrollback is nothing but this repeated over and over again:

``` OK.

Let me write.

Let me go.

OK.

Let me write the script.

Let me go. ```

I'm not sure to what extant this is a model problem, vs some providers being fairly broken. If I chose a single provider, I could know how to blame and to avoid them. With OpenRouter, I don't know which provider I was on when this happened.

The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).

If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.

That first diagram is striking: that deepseek's own inference is at least 5% higher on tool calling (TAU Bench) than most other providers. I wonder if they make sure their responses are valid json at the token generation level using a grammar, similar to the feature in llama.cpp.

As a regular user of openrouter I didn't know this info was available. Will have to check it out. I've definitely noticed that a high level of deepseek flash responses were looping endlessly before the 0731 release.

Edit: looks like the diagrams data is from the "Auto Exacto Benchmarks" section of the performance. Looks like they haven't run the benchmarks of deepseek flash 4.1 on the deepseek provider yet: https://openrouter.ai/deepseek/deepseek-v4.1-flash#performan...

Same experience. I migrated over and am happy but that intro was brutal.

My basics are I have a test suite that: 1. Finds newest models of my versions 2. Inferences every single model and a few providers for each with a short problem 3. Analyze latency and if a model failed the stupid simple questions drop it and the provider 4. Run larger context haystack kinds of problems.

It's cheap and fast less than 5$ so I can do this daily, hourly, whatever depending on how much I care. If it's mission critical I would say you need a two day study running once an hour to know the STD of model variance.

Then lock a top 3 contenders via latency dropping routing.

Is this easy? No. Is it cheap? Also no. Is it better than just using a trusted labs api? Also not really.

But it does give you exponentially more flexibility. Being able to run 10 unique models at the flick of a switch on a problem for pareto front analysis is amazing. And giving a dropdown for customers for multiple model options is powerful.

Looks like someone could make some dollars re-creating openrouter from scratch and proving reliability across evals/models/providers/configs

> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.

This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.

This is why it's not recommended to use providers below the Silver level in Semianalysis' rating index.

https://newsletter.semianalysis.com/p/clustermax-20-the-indu...

This is an issue self hosting as well. There’s a lot of footguns that give you slightly bad results.

I wonder what tricks one could use to ensure the model is actually performing on par with the reference api, like matching seeds or running exact benchmarks.

Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.

could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.

I've just been using openrouter to be able to quickly switch models and keep tight control over how much i spend and for that it's great. i also love their chat thing on the site where you can ask multiple models simultaneously , very good for brainstorming

Until yesterday I used openrouter mostly as a portal to the big providers: google, anthropic, openai, without having to maintain keys and accounts and credits for all of them separately.

Yesterday I wanted to do a bit of benchmarking a prompt across multiple models. Small requests. Outside the big providers, the experience became awful. This explains that experience.

I'm happy with OpenRouter. They're good at routing requests & spend tracking/management. You should just expect to directly own your relationship with the downstream model provider.

Source: Enterprise customer doing $XXM annual run rate of inference spend on their platform

Under privacy, ignored, I put providers that always suck. Digital Ocean, OpenInference, DeepInfra, AtlasCloud, and Alibaba (that one is different, it doesn’t suck in the same way as the others with a dumb model, but instead it’s heavily censored and doesn’t like being used as RP model in Skyrim).

> The tool call is in the text

Noob question: do good harnesses automatically optimize for bad tool calling behavior automatically?

I also believe some providers fallback to another model entirely. I was recently using Kimi K3 and saw that some requests had no reasoning trace whatsoever. Unsurprisingly, those requests were routed to the less reputable providers (Sail Research).

I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.

I haven't dove into OpenRouter yet but I see it everywhere now. I'll have to take this into consideration when I finally get around to trying it or something similar.

Wish this had more insights on 'cost' - I found OpenRouter credits to burn up fast.

Been using Olly for a while now. Very useful when you only have your Apple Watch. Didn't realize the maker had a blog.

I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.

great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).

ZDR is the main reason to use OpenRouter as it's difficult (impossible?) to get from OpenAI/Anthropic as an individual or small business.

I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.

if you'd rather have your router running on your own machine, check out the router i'm building: https://role-model.dev/

Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.

On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...

Without pinning providers, you pay let’s say 2x more for agentic coding, since that many input tokens aren’t cached as you bounce between providers.

provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to providers for higher rate limits and writing our own fallback logic.

I keep trying to sign up but they ban my account before I can use the service, and their support team doesn’t respond to my emails.

Great write up.

This aligns with what I've experienced using openrouter.

Are there competitors that handle these same issues better?

Would model values like 'openrouter/pareto-code' or 'openrouter/auto' be of any help?

Great post. I had realized there is variation, but the charts are much worse than expected

I couldn't agree more. This articles describes how frustrated has been OpenRouter experience. At the end of the day, I ended up configuring to use the owner provider of each model I needed to use.

> That's enough volume to hit every edge case at least once.

Is it, though?

Yeah, it is actually very bad now. And somehow keeps leaking keys.

Excellent post, thanks for the article. This is very relevant for anyone using openrouter and similar services seriously.

I feel there is still a lot of progress to be made before we can really trust LLM providers.

I agree that OpenRouter kinda sucks, but it took about 15 seconds to see that a few sentences got reverse-compressed into what is now this article, and it's slop.

God that was painful to read.

I still don't get the appeal of OpenRouter. Why not just generate API keys from the providers you want to use, which should not exceed 3-4, I assume, and integrate them into your apps to call them? Are people so lazy, or am I missing something?

> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.

This isn't necessarily an OpenRouter issue, Google's Gemini will sometimes do it on their own API. Since they summarize reasoning this means you pay the whole cost and can't get anything out of it.

Never had any issues using it for personal small scale development. I think the premise of OpenRouter as a magical way to fall back on error or select providers based on price/tok-per-sec/uptime isn’t delivered upon. But it’s an excellent way to simplify the experience of hopping between model providers with a single unified billing system. I mostly only use first party model hosts. Theoretically there are alternatives with higher throughput or lower prices, but it’s simplest to not worry about that optimization.

My limited impression is, that all the 3rd party inference providers are absolute garbage.

I was so disappointed that I won't consider any of them for at least a few years.

This is super insightful

Openrouter should obviously lock providers off who are lying.

Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless.

I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.

What am I missing? Is this really true?

[flagged]

[dead]

[dead]

[dead]

[dead]

Good job

As expected vibecoding bros cannot even read the manual properly.

It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.

Yes