Qwen 3.8 27B available on Cerebras at 1500 tokens/s

Posted by altertable 5 days ago

Counter690Comment228OpenOriginal

Comments

Comment by nostrebored 5 days ago

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool.

Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information.

``` Billing access restricted Self-serve billing is not available on Enterprise accounts. Please contact your team for further questions. ```

We have no team (they removed themself from our slack channel after we talked about rate limits). Perplexingly, none of this even shows up in the request, which gives:

``` {"message":"Model does not exist or you do not have access to it.","type":"not_found_error","param":"model","code":"model_not_found"} ```

When the error is really about billing.

I always want to like Cerebras, but I get the vibe that as a tokens in tokens out consumer you are not valued at all.

Comment by Aurornis 5 days ago

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks.

I don't understand. How does that make it unusable? Is the limit shared by an entire team at once?

150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find.

I think the 128K context limit is the real ceiling. These models aren't amazing at long context, but once you account for a short input prompt, the input files, and headroom for a compaction summary, there isn't a lot left for the problem.

Comment by wild_egg 5 days ago

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute.

I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time.

Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.

Comment by kristjansson 4 days ago

They made the coding plan a bit better toward the end, but it was pretty tough to use throughout.

Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.

Comment by vlovich123 4 days ago

That’s not what caching is for. Caching lets you resume with a pre computed KV cache saving you from having to ingest everything in the chat history as input on every single round trip. You still need caching regardless of SRAM or not as it saves a huge amount (and ever growing) of compute ingesting the preceding history every time you want a completion.

I don’t know why they don’t give a price discount. Maybe their hardware is incapable for some reason of saving/restoring the state? Or maybe they just haven’t built the infrastructure to do it?

Comment by jmalicki 3 days ago

> Maybe their hardware is incapable

I am unsure if it is incapable, but it sounds hard.

They have tons of cores with 64k of SRAM each, and relatively slow paths in/out.

On a GPU you can leave it in SRAM. On Cerebras, you have to send it out of the system which is a giant bottleneck.

Comment by amelius 5 days ago

Can't you do something with multiple accounts?

Comment by jychang 5 days ago

You would lose caching (if they cache)

Comment by sandworm101 5 days ago

Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.

Comment by ma2kx 5 days ago

Thats not the point if you choose Cerebras as provider.

Comment by embedding-shape 5 days ago

Or buy a raspberry pi with a SSD, about the same difference, if you're giving up on the 1500 tokens/s anyways.

Comment by interactivecode 4 days ago

The whole point is the speed

Comment by nateb2022 4 days ago

[dead]

Comment by gerdesj 5 days ago

128k context is not a limit of the model, that's a limit of implementation:

"Context Length: 262,144 natively and extensible up to 1,000,000 tokens."

https://huggingface.co/Qwen/Qwen3.8-27B

Comment by Aurornis 4 days ago

We're talking about the Cerebras implementation, which is limited to 128K.

It's in the link.

Comment by gerdesj 4 days ago

Yes, and I said its a limit of the implementation and not the model.

Comment by selcuka 4 days ago

TPM means Tokens per Minute.

Comment by zxexz 4 days ago

GP is referring to GGP’s last paragraph. 150k t/m, yes, and 128k context.

Comment by datadrivenangel 5 days ago

150k tokens per minute at 1.5k tokens per second means you can have like 3 users concurrently and that's not a lot.

Comment by conception 5 days ago

150k by account. At 1.5k a second you hit it very quickly.

Comment by devy 5 days ago

Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.

Comment by gerdesj 5 days ago

I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware.

I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve.

To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.

Comment by conception 4 days ago

The problem is most providers hit tok/sec limits really fast. 1m/min is the default and the only place I can get 10m+ is from first party providers without a lot of upfront cash.

Comment by jacquesm 5 days ago

How fast is it?

Comment by wincy 4 days ago

Parent already responded but just for reference an RTX 5090 with Ninfer hits 160 tokens/second with qwen 3.8 27B which is very usable.

Comment by kristjansson 4 days ago

With MTP and FP4 I max out at 30ish t/s on mine. Without MTP or in regimes where the drafter performs poorly it’s about 10 t/s. FP8 is about half that

Comment by jacquesm 4 days ago

Thank you, always nice to see real world performance figures.

We run a pretty large rig, 10 GPUs right now (this goes up and down with various experiments, getting this many GPUs to play nice at x16 GEN4 with any motherboard is a challenge), 240G VRAM in total. 256G RAM and a TR PRO. For small models the comms overhead is larger than the gains so there I have to reduce the number of active GPUs. On this machine I'm getting between 150 and 200 tg/s with FP8, but it took a lot of time and tweaking to get to that, and not all of the improvements held up when combined with other improvements. I've been playing with this stuff for a while now and it is interesting how fast the frontier is moving and how much you can now do on your own hardware. For larger models the communications overhead is low enough that we can run them on bigger groups of GPUs, and using hacked drivers to give us p2p capabilities on some of our GPUs also boosts performance considerably once you start to hit communications limits. Typically we get 50G/second in p2p mode (full duplex, half that one way).

From a cost perspective running locally is not interesting, but it allows us to do experiments that model providers would likely balk at, gives us censorship free access and allows us to work with data that we would not want to share with model providers (or can't share due to NDAs).

I will look into running ninfer, I was aware of them but had not yet gotten around to using it.

Comment by a012 5 days ago

Unusable is too stretch IMO, you can still use it in tiny tasks that’ll respond almost instantly

Comment by 5 days ago

Comment by 0xbadcafebee 5 days ago

Yeah, their public service isn't a serious/competitive offering. They don't have the capacity to serve all the customers who might want to use them at that speed. The public service exists so they get some users on OpenRouter, and that shows them as #1 on speed, which proves their tech is very fast, which gets them billions in hardware sales/licensing. If you have big enough pockets they can probably dedicate capacity to you. But for reliably fast small models you might want to rent some GPUs.

Comment by LoganDark 4 days ago

GPUs can't reach these speeds. You could build a supercomputing cluster and still not reach these speeds.

Comment by gpugreg 4 days ago

MiMo-V2.5-Pro-UltraSpeed gets pretty close with over 1000 TPS on 8x B200. It has 1.02T total parameters and 42B active, compared to 27B total/active for Qwen3.8-27B. Also, B300 are out now. I think 1500 TPS for Qwen3.8-27B should be doable.

Comment by LoganDark 4 days ago

That model uses a lot of tricks to achieve 1000 t/s. I would not use raw parameter counts alone for such comparisons, in general.

Comment by nostrebored 4 days ago

Celeris reaches ~50% of the speed on commodity hardware. celeris-magnus-1 is based on qwen3.8-27b. Maybe we will get there without custom silicon!

Comment by puppymaster 4 days ago

all the above. They just simply do not care about non enterprise customers. Today they announced qwen, guess what - it's also the same day they pulled Gemma off their shared tier. No migration notice and all developers are scrambling as we speak trying to migrate. They gave a soft head-ups on discord a week ago and when folks complained about zero-day migration they started saying 'you aren't suppose to build production app on shared tier'.

Comment by ryukoposting 4 days ago

On discord? Jeez. How professional.

Comment by olivermuty 5 days ago

Cerebras the tech is awesome, cerebras the company is a trainwreck

Comment by dd8601fn 5 days ago

Is this the chatjimmy asic approach with a bigger model?

Comment by ericd 5 days ago

No, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.

Comment by vel0city 4 days ago

I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning.

Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.

Comment by ericd 4 days ago

Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs.

But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.

They may have had a little flexibility by supporting finetuning via LoRAs.

Comment by liamwire 4 days ago

To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.

Comment by ericd 4 days ago

Yeah, I'm guessing this isn't unique, but I remember the first time I used ChatJimmy, I missed the fact that it had responded because I was still hitting the enter key, and getting ready to see tokens stream in, but they were already all sitting there, and I'd missed registering the visual diff somehow.

Comment by runako 4 days ago

This seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.

Comment by ericd 4 days ago

Yeah, it'd be a lot easier to maintain flow, less need to work on more than one session at once, etc. And then tool calls would be the limiting factor, especially network access. I hope AMD keeps the project moving forward post acquisition.

Comment by Tuna-Fish 4 days ago

Yes and no. A single chip cannot have it's weights adjusted, once it's out, it is what it is.

But also, the model weights are in a single mask rom layer, high up in the metal stack. They could manufacture the die specialized for a given geometry of a model up to that layer, wait for updated weights, and then get the final product out in weeks after they got the weights, instead of many months which is what it would take to redesign the whole chip for the new weights.

Comment by cute_boi 4 days ago

i hope groq wins if they start doing such things with consumer.

Comment by collin 5 days ago

This was my experience a year ago on some other model they could run super fast. Routine coding tasks would hit the per-minute token limits.

Just the math there... 150k TPM... and 15k TPS means... you can run for 10 seconds every minute?

The basic math boggles the mind.

Comment by msdz 4 days ago

> means... you can run for 10 seconds every minute?

It’s one order of magnitude less TPS, but still, that’s the limit with just one user…

Comment by baegi 5 days ago

Not sure how the rate limiting works, but it's 1.5k TPS, not 15k, so you could run it for 100s/min, which seems good enough to me

Comment by nostrebored 5 days ago

iirc input (uncached) goes towards the limit as well

Comment by fc417fc802 5 days ago

What's the tok/s when they process input?

Comment by fc417fc802 5 days ago

It seems you forgot to account for the fact that cerebras uses a baker's minute which is 144 seconds instead of 60. (Seriously though what's the supposed issue here?)

Comment by RussianCow 5 days ago

The issue is that all input (including context) counts towards that limit. So 10 requests with 50k of context will blow through the limit, even if little to no output was generated, which is incredibly easy to do with agentic workloads.

Comment by collin 4 days ago

ah, yes, that seems right

I was using it quite a while back, different model, different quotas, but for coding tasks it routinely hit quotas which made it quite difficult to actually use.

100s/min seems pretty poor actually with sub-agents etc.

Comment by ricardobeat 5 days ago

What kind of coding tasks would you expect to hit that limit? In my setup, on a very large codebase, it takes each agent 3-4 minutes at minimum to go past 100k tokens.

(note it's 150k uncached tokens, the total limit is 450k/min)

Comment by nostrebored 5 days ago

in my last tests with cerebras for coding tasks, most large tasks or anything greenfield would hit token limits. note that smaller models and the gpt-oss-120b style models they used to run are very prone to overthinking, so individual turns may be 3-10k tokens of just thinking + input + output.

i don't think it's quite apples-to-apples to compare to a frontier model or even a k3. the odds of success (file compiles? read the right context?) are lower and thinking is longer.

Comment by conception 4 days ago

So that’s about 400 tok/sec. Times that by 3, you get 100k in under a minute. That’s doing nothing special and just using your current setup.

Comment by ricardobeat 4 days ago

150k is 2500 tokens/sec.

Comment by 5 days ago

Comment by lukewarm707 4 days ago

use 3rd party marketplaces. cerebras is resold on vercel, openrouter and huggingface.

Comment by vidarh 4 days ago

[dead]

Comment by gpugreg 5 days ago

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit.

For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size of 55217 tokens, while Qwen3.8-27B was not even close to being done with a 64178 context window.

This is a very efficient way to burn your money, but I would not recommend it for programming.

On the positive side, I got a $5 signup bonus, so it wasn't my own money.

Comment by eveningtree 4 days ago

The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low?

This doesn't work for my use case at all (code generation).

These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't see myself using llms for them in the future too.

Comment by 70rd 4 days ago

Throughput is useful if you want to generate a lot of transcripts for RL. It's for making Qwen better, not for actually using Qwen.

Comment by wongarsu 4 days ago

Might be usable for short-context utility workloads? Generate the title of your chat session based on the first three messages at the speed of light

Comment by brookst 4 days ago

Reminiscent of race to sleep: not suitable for sustained workloads, but for bursts ones it’s a good approach.

Comment by irthomasthomas 5 days ago

Without prompt caching this becomes more expensive than fable 5.1 after turn 50, assuming you start with 40k tokens and add 2k per turn.

Comment by d2p 5 days ago

> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds

I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?

Comment by gpugreg 5 days ago

Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.

Comment by nullbio 4 days ago

Cached tokens counting toward the limit is ridiculous.

Comment by perching_aix 5 days ago

then it's basically useless lol, wtf, this has to be a defect

Comment by selcuka 4 days ago

It's PR:

https://news.ycombinator.com/item?id=49556302

Comment by Pxtl 5 days ago

Could this also be coming from the problem that Qwen3.8-27B's default mode being "extra-high reasoning level"?

Comment by gabri200 4 days ago

To get a good coding agentic system you need to use big context (Specs and conversation context can't be condensed every minute), so you need to use prefix caching, and the price for the hit cache tokens can't be the same that miss cache or the final price could be insane.

Comment by jurgenburgen 4 days ago

I don’t think I understand. Why would faster token generation burn more tokens? The LLM should not be generating anything in between tool calls so the only difference should be that the human waits less between turns.

Comment by codygman 4 days ago

Qwen 3.8 on xhigh defaulr needs 128k context minimum or you'll spend most of your time compacting context.

Also make sure you use the instruct temperatures/etc for implementation.

Comment by pllbnk 5 days ago

Just a couple days ago I learned about ninfer (https://github.com/Neroued/ninfer) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.

Comment by lowbloodsugar 5 days ago

Ok, I need to try that. I'm getting 45tok/s with vLLM on my 6000. >600tok/s concurrent, but 45tok/s single request.

Comment by pllbnk 4 days ago

Even without ninfer I would get over 80 on LM studio with default settings, so it should be noticeably more on 6000. You might want to try different a different inference engine or settings.

Comment by aizk 4 days ago

Is there an equivalent but for 4090s?

Comment by pllbnk 4 days ago

The repository has many forks, suggesting that folks are trying to (vibe) code support for different GPUs. Might be worth a shot.

Comment by beastman82 5 days ago

can't second ninfer enough. amazing tech

Comment by jakswa 5 days ago

dang only for certain nvidia GPUs, had my hopes up

Comment by jasongill 5 days ago

It would be great if they made their inference capacity for this model available via OpenRouter; the fastest provider on OpenRouter right now is at ~80tps https://openrouter.ai/qwen/qwen3.8-27b#providers

They do appear to host other models on OpenRouter so maybe Qwen3.8 will be there soon: https://openrouter.ai/provider/cerebras

Comment by zackangelo 5 days ago

We're serving it around 150-200tok/s (uses our new speculative decoding implementation on a DFlash2 draft model).

https://mixlayer.com, LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.

Comment by danielklnstein 5 days ago

I tried in your playground and got 14.2 tok/s?

Comment by zackangelo 5 days ago

apologies we just got a sudden burst of new users and traffic, it's scaling up now.

Comment by zackangelo 5 days ago

just added 8 more H200s to the cluster, if you (or anyone else) runs into issues please feel free to drop me a message: zack at mixlayer.com

Comment by danielklnstein 5 days ago

Works much better now! Got 103.9 tok/s, not quite 200 - but still amazing! Thanks for sharing

Comment by zackangelo 5 days ago

Something a lot of model providers don't talk about: any time an engine uses speculative decoding the throughput will depend on how much your output token distribution matches what the draft model was trained on.

The DFlash2 draft model we're using was trained on a lot of code, so if you use it in a coding agent you'll probably notice it run a lot faster (we've seen it break 300 tok/s).

Comment by danielklnstein 5 days ago

FYI, I might be missing something but I think your billing system might not be working well - I'm not seeing any indication in the UI that my usage is being deducted from the $5 of free credits.

Comment by chrisboulton 5 days ago

Hey Daniel! It's a bit hidden, but at the bottom of the billing page there's a "Credits" section which should show usage of any active credits and the balance remaining. The usage/billing metrics are batched/handled async so it might take a minute or so for usage to be reflected. Let us know if it feels off.

Comment by RussianCow 5 days ago

I don't see any kind of input cache discount listed on your pricing page. Do you offer that, or is all input priced the same?

Comment by egorfine 4 days ago

Trying now. History question: got 96 t/s. Code review task: 130 t/s. Nice!

Comment by 5 days ago

Comment by scratchyone 5 days ago

any way to see the tok/s for all the models listed on your homepage? curious which has the best speed/quality tradeoff for me

Comment by bookernath 5 days ago

This feels great

Comment by eli 5 days ago

I just did a little anecdotal test. Had pi + cerebras review a recent commit and asked a few quick followups on it. Worked great.

The Cerebras session cost me $1.60 and took a total of 5.1 mins. I did get a few brief 429 rate limit errors in there. The p50 speed was 890 tok/s and 0.64s TTFT.

Using OpenRouter averages, that would've cost $0.29 (no cache discount at Cerebras!) and would've taken about 14.4 minutes.

So on this one short session, cerebras was 5.6x more expensive in exchange for being 2.8x faster. Or, another way, $1.32 buys back about 9 minutes of your time. Not a bad trade IMHO but the cache situation is a real bummer. The longer your session the more relatively expensive Cerebras gets. The "good" news is you're also limited by its short context window.

(Also, I used to be on the Cerebras coding plan and the support is pretty bad for end users. My guess is these public endpoints are really just product demos for potential enterprise customers.)

Comment by irthomasthomas 5 days ago

Thanks! Is there something about their platform that prevents caching? Or are they just not passing on the discount?

Comment by eli 5 days ago

The session had a 91.4% cache hit rate. They just give zero discount.

Comment by imtringued 4 days ago

It sounds to me that they don't have enough capacity and they want to discourage people from using the service.

There is nothing about their architecture that prevents reusing the KV cache other than the opportunity cost of keeping the memory occupied.

Comment by hexa00 5 days ago

Just tried it on a medium size coding/debug problem on an existing codebase, observations: - Input doesn't look faster than other models, it spends a lot of time reading Read about 5M tokens - Output is awesome, super fast as you expect from the 1500t/sec I think that's correct - Tool call is failing more than say DS4, which leads to time wasted on retries (complex tools like browser control for example) - Shell commands are still somewhat of a bottleneck

The net effect is that I spend about the same time waiting, and I still need to read that output so, at least for coding, it actually reconciles me with the 100-200t/sec you can get on DS4 or the like. Maybe that's a good sweet spot after all and faster t/sec is not where the bottleneck is.

Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy

Comment by peri-cl 5 days ago

> "Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy"

I don't believe Cerebras has a cached input pricing? They don't list one on the model page:

https://inference-docs.cerebras.ai/models/qwen-3.8-27b

edit: See the sibling discussion,

https://news.ycombinator.com/item?id=49554520#49555094 ("Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate")

Comment by hexa00 5 days ago

lol yeah just saw that, yeah that makes it unusable I think at least for me.

I wonder if they will do that with sol ultrafast!

Comment by olivermuty 5 days ago

They have cache, but it costs the same indeed, no idea what the point of the cache is

Comment by lostmsu 5 days ago

They don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!

Comment by orbifold 4 days ago

More precisely they can't cache it.

Comment by lostmsu 4 days ago

Yes. Their architecture recomputes every time so at 150k context every request will have to spend 1.5 min waiting for the model to reread the context.

Say avg model response length is 1024 tok. At 50 tok/s normal providers do your turn will only take 20s (vs Cerebras 101s) and will cost 20x less. That time and cost is per single tool call.

Comment by irthomasthomas 5 days ago

I can't believe this situation has not improved in years. Is cerebras' main business selling the hardware, then?

Comment by redman25 5 days ago

Maybe they’re gunning for speedy non-interactive pricing? Or its a limit of the technology or a business decision?

Comment by tandema 5 days ago

Cerebras is super constrained on capacity right now, all the support is going to enterprise customers.

Comment by orbifold 4 days ago

they have exactly two customers, both of whom are also investors.

Comment by nkhs89 5 days ago

[dead]

Comment by gardnr 5 days ago

I used their Coding Plan for a few months. It is genuinely difficult to keep up with the models. The output is so fast. Qwen 3.8 27B is likely one of the strongest models they've hosted so far.

Edit: it looks like this is only available on a API token pricing. Does anyone know if they have rolled out prompt caching yet? It used to get pretty expensive for agentic coding tasks with no prompt caching.

Comment by jasongill 5 days ago

It appears that they do support Prompt Caching: https://inference-docs.cerebras.ai/capabilities/prompt-cachi...

Comment by the_duke 5 days ago

It doesn't reduce the price though.

Comment by jasongill 4 days ago

Good catch, I guess I got lost in the marketing speak of the page!

Comment by abtinf 5 days ago

> How are cached tokens priced?

> There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model.

Well, talk about flipping the narrative.

Comment by Barbing 5 days ago

heh

Is there a speed increase or is that purely marketing spin on “we might cache on our end but no discount for you”?

Comment by lostmsu 5 days ago

Pure marketing.

Comment by qlte 5 days ago

[dead]

Comment by 5 days ago

Comment by 5 days ago

Comment by cute_boi 5 days ago

i believe they used to have monthly plan, what happened to that?

Comment by eli 5 days ago

Strongest model that they host on the public endpoint. They do a super fast version of GPT 5.6 Sol for OpenAI and have bigger open models on dedicated endpoints.

Comment by singpolyma3 5 days ago

The coding plan is gone now right?

Comment by gardnr 5 days ago

Last time I got one, I had to log into a Discord server and wait for "the drop" and IIRC Daniel Kim was giving them out based on who was there at the time. They were gone in less than a minute. This was ~8 months ago.

Comment by altertable 5 days ago

Agreed, but in our SAAS I can tell some UX will sky-rocket to next level with this

Comment by dshat 5 days ago

I'm saddened that Gemma4 is replaced by Qwen 3.8 on PayGo plan. Gemma4 31B is not coding model but it is excellent at intent understanding and task execution used in agentic software. This just shows that real world dominant usage for llms so far is to code generate. And not to augment business products. They must had barely anyone using Gemma to remove it from that tier.

Comment by foundfontic 5 days ago

I really wish they had their customer support somewhere else than Discord, which seems to think I'm a bot and doesen't accept my email or phone numbe

Comment by londons_explore 5 days ago

discord support can fix such issues

Comment by threecheese 5 days ago

If you need customer support to access customer support, something is wrong; no?

Comment by Zambyte 5 days ago

Discord is simply a liability.

Comment by tacone 5 days ago

Noticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon.

For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.

Comment by walrus01 4 days ago

128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.

Comment by imtringued 4 days ago

I have done the vast majority of my agentic coding work under 128k on a 256k context model and when I cross 128k, all I do is just ask for a handoff prompt to feed back into the next session. I do this because I was told the quality degrades as the context fills up and I fundamentally dislike random compaction, I don't even know what compaction does, it is completely intransparent.

Comment by fulafel 4 days ago

Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc.

I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.

Comment by walrus01 4 days ago

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator.

128k is pretty much only good for chat/conversational/question asking (including tool calls for searching things and spitting back/parsing a set of results) or human interactive agent purposes.

Comment by imtringued 4 days ago

One of the most boring tasks to give to a subagent is to make it run cargo test and summarize the result so that the main agent doesn't get polluted by the irrelevant tests.

You have to have a pretty inefficient use case for a subagent to think that 128k is not enough.

Comment by fulafel 4 days ago

128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all.

I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.

Comment by gilgoomesh 4 days ago

It is common for the agent tools, MCP tool, skills, project context and system prompt to take between 5k and 25k tokens. This depends heavily on your agent and tool setup.

But that's before the prompt.

Then after the prompt, every tool call the agent makes adds to the context. Longer turns can easily consume 50k-100k tokens between the agent and various tool calls (reading the filesystem, reading files, reading compiler output, reading memories).

Then each "turn" with the agent stays in context and is fed into the next turn. Two or three turns and you're up near 250k.

Comment by imtringued 4 days ago

I actually decided to run YOLO mode once on GPT 5.6 Luna setting it to the fastest speed and I struggled to fill up the context while I kept sending several dozen feature prompts in the same session.

Comment by walrus01 4 days ago

Tokens are also occupied by the internal reuslts of 'thinking', for instance, using the latest opencode, give qwen 3.8-flash-next with 'high' thinking mode 50KB total of python to read in six different files, count the expected tokens you'd expect it to occupy in just the size of the python. Then count the actual token count taken up after it's finished thinking about the contents of the python and analyzing it before you give it a single directive.

Comment by srcreigh 5 days ago

Great observation. That’s not enough context even for some one shot xhigh requests.

When I put Qwen3.8 27B xhigh towards adding scope proxying to the Guice library, it one shotted a great impl using 250k context before stopping.

Part of the greatness of the model is that it just keeps going until it gets a great result. 128k context is disappointing.

Comment by freehorse 5 days ago

I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.

Comment by bitexploder 5 days ago

The thing I didn’t realize for a while is 27B is rather smart. As many (or more) activated parameters as the flash models of the universe that we know about. It reasons very well. It just doesn’t have a lot of knowledge.

Comment by nicce 5 days ago

They seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.

Comment by bitexploder 5 days ago

But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)

Comment by nicce 4 days ago

You can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.

Comment by bitexploder 4 days ago

I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.

Comment by codazoda 5 days ago

Really an aside, but yesterday I got the Gemma-4-12b (128k context) to build it's first web app in the minimal Dark Software Factory I've been building for myself.

https://joeldare.com/a-local-open-weight-model-builds-its-fi...

Comment by blaesus 4 days ago

May I ask what you used Gemma 31B for? Last time I try it wasn't bad but then it wasn't particularly good either.

Comment by freehorse 4 days ago

It has been used in kagi's "quick assistant", so basically summarisation tasks (gets context from a bunch of webpages and finds the answer to a question in them, spits all the relevant to the query information, or similar).

It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.

Comment by RomanPushkin 5 days ago

The question is whether Cerebras is available... I've been trying to get https://www.cerebras.ai/code for at least 1 year now. It's all sold out. Always. I once joined their Discord, waited for the drop, and it all sold out in seconds. I haven't had enough time to put my card details. Somebody recommended that I should put my card details in advance, lol.

The next time I hear about them I am laughing, because when I could enjoy these powers? How many years I should be sitting in a waitlist...

Comment by latentsea 4 days ago

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

Comment by mncharity 4 days ago

> 1500 t/s [...] all you can eat tokens as fast as you can eat them

Pfft, you can eat tokens far far faster than that. Many orders of magnitude more. Just switch from frontier-style bespoke artisanal pets to cloud burst-parallel, latent subspace exploring/exploiting/searching, mass ensembles of cow herds.

There's N-Version Programming. Work the problem in English, in Chinese, in Haskell, Lisp, Rust, etc. Then work ports to the target lang.

There's design space sampling. Work the problem emphasizing performance, or security, or monitoring, readability, etc. Then work a synthesis.

There's non-determinism sampling. Work the problem order 10 or 100 times. Then work to combine the best bits from each.

There's sample synthesis. NP-hard aggregation of insights.

There's genetic exploration. Work populations of trees of work variants under selective pressure.

There's repo quantum superpositions of implementation space. The unspecified remains indeterminate - state space collapse occurs not upon each edit/commit, but as JIT-synthesized fuzzing/search upon each execution.

There's maintaining a pretty dev UI, but that >>10k tok/s is trivial, because like symbiotic adversary cocreation, fine-grain agent swarms, scenario analysis/forecasting, etc, etc, it is unlike the preceding items... which scale combinatorially.

"All you need is 1500 t/s"? "All you need is 640k RAM" is only 5 orders of magnitude off from 64 GB. It takes "All you need is a single Intel 3101's 64 bits", to get 9 orders of magnitude from 64 GB. Then datacenters...

Comment by latentsea 4 days ago

I too know words.

Comment by mncharity 3 days ago

Too compressed? Something better unpacked?

Comment by latentsea 1 day ago

No. No... I'm... I'm good. Thanks.

Comment by gilgoomesh 4 days ago

Easy. I just need... 50 times more speed.

Better download some more RAM.

Comment by kilroy123 4 days ago

This too is my dream. But I want fable level intelegence. Not Qwen or deepseek flash.

Comment by orliesaurus 5 days ago

Qwen 3.8 27B is an exceptional model for coding and ranks as one of the best local models for coding....BUT in my head I am confused why a company that's IPO'd doesn't invest in RL'd super specialized, super-damn-fast models for very specific tasks - instead of giving us the OSS GPT model from what feels like 200 years ago

Comment by anthonypasq 5 days ago

almost of their business is hosting Sol ultra fast or whatever for OpenAI to use internally

Comment by kroaton 5 days ago

Especially since they still serve Codex-Spark, which is dogshit.

Comment by lsb 2 days ago

The context window is limited to 64k or 128k. If you’re using this with a coding agent, there’s going to be a lot of compactions. I found that I had a subagent whose compacted context plus prompts and such was over the window and it errored out in opencode.

Comment by apatheticonion 4 days ago

I just want an API that takes these crazy small / cost effective open weight models and charges peanuts for access.

Think, DeepSeek Flash (before the price hikes) prices.

If I can run this on a 32gb card while they have a datacenter with wholesale electricity prices, why are we not seeing "cents per billion tokens" pricing?

Comment by mrngld 4 days ago

Because your 32gb card isn't running this at 1500 token/s. Serving these things at scale with the enormous context windows real use demands and doing some with usable performance takes a lot of expensive hardware. Yes, their margin on straight inference is allegedly really high, but that's severely offset by high capital costs.

If you want to spend a new car worth of money and still not serve as fast as Cerebras because you can't simply buy their mammoth custom chips, then yes you too can self host a huge Deepseek or GLM model.

Comment by apatheticonion 2 days ago

Yes, that's what I'm saying. 200t/s is more than enough to 5 - 10x my productivity and the intelligence of current open weight models covers 90% of both my guided and software factory workflows

Comment by gabri200 4 days ago

I tried to get Qwen3.8 27B working properly under high concurrency, and while the quality level is spectacular for the size, the performance wasn't the best, even with MTP. Unless you have a very big infrastructure, it's difficult to run a dense model concurrently with high throughput.I suppose that's why almost all large models are now MoE. On the other hand, 1500 tok/s is an impressive speed, and that speed is very important for agent tasks, so a service like this instead of local infrastructure might make sense, although it also depends on your busines constraints.

Comment by iamleppert 4 days ago

Completely unusable last time I tried it. You got hit with rate limits after the first few minutes of using it. It's fast but cannot sustain its claimed speed before it immediately hits its rate limit. What good is it if you can't finish a task? It's like having a sports car that can only drive 180 mph for 5 seconds every minute and then has to cool down for an hour.

Borderline fraudulent to advertise to developers when it's completely impractical to use. Their only support is their company Slack channel.

Comment by ecshafer 5 days ago

I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"

Comment by FeepingCreature 5 days ago

I have a self hosted Qwen 3.8 27B and I find it unbelievably cracked and dedicated. It's at least credibly attempted everything I've thrown at it. Just today I had it write a toy compiler with a JIT backend just to test out a concept, and that was with 4-bit quantization and 8-bit KV cache. Something has to be going wrong with your deployment.

Comment by codazoda 5 days ago

I want a Qwen 3.8 27B hosted locally but I don't quite have the RAM for it. And, I don't want to buy the RAM until I prove I can use it.

Yesterday I did have success with Gemma-4-12b with 128k context. It fits in my RAM and it's relatively fast on my hardware.

I had to give it prompts that are quite a bit different from the way I use foundation models, but I did get it to work quite well. I feel like I could learn it's differences and get good at using it for real work.

Comment by apitman 4 days ago

You could try renting a GPU or VPS

Comment by pyrolistical 5 days ago

I run it locally at q4_k_xl on a r9700 with kv cache bf16 and while it thinks a lot, it’s still fast enough to do the task.

This model had its knowledge replaced with reasoning ability. The chain of thought what makes this reasoning effective.

So this is why you need to let it think and don’t quantize the kv cache.

Comment by FeepingCreature 4 days ago

Or at least use a modern llama.cpp with KV activation rotation.

Comment by hedgehog 4 days ago

Check sampling parameters and chat template, make sure you have adequate context window, turn reasoning effort down. It should be able to one shot a small app without intervention.

Comment by Almondsetat 4 days ago

Which quantization?

Comment by grav 5 days ago

Should be available in OpenCode once this lands: https://github.com/anomalyco/models.dev/pull/6199/changes

Comment by irthomasthomas 5 days ago

It's going to cost a fortune in opencode without prompt caching.

Comment by Roark66 4 days ago

Don't they have very low limits?

What do people use these tiny limits for?

I started measuring my Claude Max x5 use and last week (they gave me 50% more) I used 1.3B input tokens. Some 130M were cache writes, rest was cached. And 5M output.

This puts things in perspective. We're taking thousands of bucks weekly even if I managed to switch to Kimi K3.

What is the majority of this use? Infrastructure upgrades, troubleshooting and so on. Ingesting quite a bit of documentation at beginning of each session.

Sessions run from few hours to a month long and 1M context usually hovers near 30-60%.

Comment by peri-cl 5 days ago

(Was anyone able to create an account just now? I tried but onboarding falls into a redirect loop)

(update: I got my answer. support@ replied and said my email domain is on their blacklist. It was just me (and I've resolved it)).

Comment by bakies 5 days ago

yeah - used sign in with google

Comment by porphyra 5 days ago

Why do they only host small models rather than the 2.4T version? Is the I/O and interconnect between the wafers bad due to the limited beachfront relative to the massive size of the chip?

Comment by gardnr 5 days ago

They make a giant inference chip. Their inference service is basically just advertising for their core value prop: hardware.

The CEO was on Gradient Dissent a couple years ago: https://www.youtube.com/watch?v=qNXebAQ6igs

Comment by codexon 5 days ago

The wafer only has space for 44 gb of sram. If they offload ram they lose the speedup of having everything on 1 chip (the whole point of cerebras).

Comment by porphyra 5 days ago

They can host larger models by pipelining it on multiple wafers. Each wafer stores one layer and N layers can serve an N * 44 gb model with N concurrency. The limitation would of course be inter-wafer I/O, which my comment was getting at. That's probably how they can serve bigger models like GPT 5.6 Sol [1].

[1] https://www.cerebras.ai/blog/accelerating-gpt-5-6-sol-ultraf...

Comment by codexon 5 days ago

I never said offloading was impossible. It will result in a large slowdown.

It would look bad for cerebras if other people are hosting the 27b version and show a higher TPS than cerebras.

Comment by minimaltom 5 days ago

[dead]

Comment by altertable 5 days ago

Mostly economics I'm sure

Comment by codazoda 5 days ago

Do I understand their pricing correctly? This is $10 per month for a developer account PLUS you pay $1.49/M for output tokens and $0.99/M for input tokens on Qwen 3.8 27b with a 128k context?

EDIT: Or, maybe it's just token pricing, but $10 is the minimum? Maybe it's that.

https://www.cerebras.ai/pricing

Comment by low_tech_punk 5 days ago

No. You buy a minimum of $10 worth of credit, then use it at $1.49/M rate. There is no recurring charge.

There is a separate subscription based plan, which is sold out now.

Comment by codazoda 5 days ago

Got it. But, they also charge the same for cached tokens, so that probably closes the gap on Foundation models quite a bit.

Comment by ma2kx 5 days ago

I guess Cerebras didnt intend the model for agentic coding but rather for small one shot task like title generation. At least thats why I use the free tier for.

Comment by low_tech_punk 4 days ago

my hunch is that their hardware can't cache

Comment by the_duke 5 days ago

Funnily enough the pricing isn't that much worse than on openrouter, where the best price at the moment is $0.24 in / $2.55 out, vs $1 / $1.5 on Cerebras.

Sure, 4x input , but cheaper output. Though Cerebras doesn't have prompt caching, so not great for agentic workloads. (they do, but it doesn't affect the price.

Comment by srcreigh 5 days ago

It is 15x more expensive. Openrouter usually charges like 1/4 for cached input.

Most of the cost for agentic coding is input tokens, you pay for the whole context at each tool call or message. Output tokens is just a small rate

Comment by 5 days ago

Comment by register 4 days ago

I tried it. Not impressed. The gain in development speed is only marginal as there are other bottlenecks that affect overall development speed. Probably gains might be more significant for pure content generation tasks rather than software development

Comment by laweijfmvo 4 days ago

Kagi used to serve gemma4 31b on cerebras, and i got so used to the speed that i basically stopped using other models. now that they no longer offer it, i find myself reaching for a model a lot less

Comment by darkbatman 5 days ago

I have been their user for more than year even used coding plans, though for normal coding the quota will definitely be a blocker if you are using opencode because rpm are bit less. Good for products/api though.

Comment by polygot 5 days ago

Ut oh, might be down: "Unable to connect to the server. Please check your connection and try again." when sending a message to Qwen 3.8 27B.

Comment by vb-8448 5 days ago

At that speed it's too pricey for agentinc tasks.

Comment by yipinwong 5 days ago

The target audience is who needs raw speed.

Having the choice is good as you can make a trade-off between speed, perf, and quality.

Until last year, people had a single AI god they believed in (mostly Anthropic stuff). Now we have power to make choices (open-weights, SOTA, speed-optimized, etc) the same way you do for system designs.

Comment by vb-8448 5 days ago

It's not a criticism, I was really looking forward to trying out such a powerful model at this speed.

But I burn my 5$ allowance in 10 minutes ... and only because I was hitting rate limits, without it would probably be less than a minute.

Comment by yipinwong 5 days ago

I hear ya... the best option is to use company budget as normies will rack up ridciulous amount soon with that raw speed.

Comment by storch 4 days ago

But what happened to Gemma 4, it disappeared from the catalog? Has the Gemma 4 Cerebras API been "deprecated"?

Comment by walrus01 4 days ago

I don't see the point of paying for external inference on Qwen 3.8 27B with a bunch of arbitrary limits, when you can run it locally without ridiculous memory requirements. Even the unsloth Q8-XL version of it with full context and extra llama-server --cache-ram (like 10GB instead of 8GB) fits in 64GB.

Paying for external inference for a much larger model like qwen 3.8-flash-next Q8 with full context makes a lot more sense, since the model consumes something like 188GB RAM when fully loaded into an inference engine.

Yeah I guess this is cool and all that it runs at some ridiculous token/s rate but if the actual usage of it is highly limited... What's the point? I'd rather have a much slower tok/s rate that can chew on things 24x7.

Comment by Marciplan 5 days ago

used their Code product with GLM4.7. its fun but if the model is bad it just doesn’t do much useful.

Hope they add such models to Code too :)

Comment by altertable 5 days ago

Yeah GLM 4.7 is from another decade at the speed we're going

Comment by fulafel 5 days ago

What are the best benchmarks/leaderboards that compare task completion time between provider+model combos?

Comment by explorigin 4 days ago

Yep, dumber than Deepseek4 at 10x the price. Cool tech demo though.

Comment by naklitechie 4 days ago

Has reliability improved? It's flaky in Production.

Comment by srcreigh 5 days ago

How many years until chips like this are available to consumers?

Comment by nicce 5 days ago

Many. Too lucrative for certain companies and even governments to allow that to happen

Comment by mrngld 4 days ago

You're right, but no need to be conspiratorial, demand is just so high if you ordered one today you might be lucky to see it in 2028.

Comment by karim79 5 days ago

Tokens are the new latest and greatest nonsensical shit on the planet. It's amusing. I can't wait to see the world in 1-2 years and the hilarity of looking back on this day.

Comment by mrngld 4 days ago

Someone on horseback probably said that after seeing a car broke down on the side of the road for the first time.

Well, cars only got better.

We're going to be talking about how to leverage this new tool for many years to come.

Comment by khalic 4 days ago

I can't wait for Qwen 27B on an ASIC

Comment by dev0p 4 days ago

I wonder how much it would cost?

It's something that's been bugging me for a while: once we reach a "good enough" small model, and qwen 3.8 27b is already getting damn close to it, does it make sense to just bake weights and everything directly into an ASIC, and use that for highly optimized inference? AFAIK only groq and cerebras are moving in that direction, and only to be providers themselves... It would be a dream to buy one for <$1k.

Comment by nrub 4 days ago

This is exactly what Taalas has already done, and the reason they got quickly acquired by AMD. Their chip runs Llama 3.1 8B at 17k tok/s, and even if llama 3.1 is dated, I can think of many problems I could use it for, especially at those speeds. They're certain to be working on a newer set of weights by now.

Comment by drchaim 5 days ago

The idea of custom software on the fly is coming

Comment by forlorn 4 days ago

Is Kimi 3 available anywhere like that?

Comment by WithinReason 4 days ago

Now imagine having it running 10x faster on a Taalas chip on a card you can buy for $1k. I hope that future happens

Comment by trvz 5 days ago

Normal people: tok/s or t/s

Psychopaths: tok/SEC

Comment by altertable 5 days ago

ok fair, caps lock kept ON /o\

Comment by scotty79 5 days ago

I like tps

Comment by verdverm 5 days ago

do you get reports on them?

Comment by actionfromafar 4 days ago

Got you covered.

Comment by onesandofgrain 4 days ago

Tried it limits are shit

Comment by TawResearch 4 days ago

[flagged]

Comment by 4 days ago

Comment by jing09928 5 days ago

[flagged]

Comment by alescalaios 4 days ago

[dead]

Comment by byako 5 days ago

[flagged]

Comment by miohtama 5 days ago

Your brain can wash laundry and cook pasta, so there is still a long way to go

Comment by qiine 5 days ago

(requires additional fleshy bits sold separately)

Comment by davrosthedalek 5 days ago

regarding my brain, my mother might disagree on the laundry part.

Comment by dgellow 5 days ago

Your brain updates itself constantly and maintains your whole body, LLMs are static.

Still, 1500tokens/s is indeed wild

Comment by ripbozo 5 days ago

fyi this is an AI bot account

Comment by eli 5 days ago

If you read the reasoning trace for Qwen 3.8, it does a whole lot of "uh" and "But, wait..." too

Comment by howunfortunate 5 days ago

You're absolutely right - filler words are genuinely load-bearing

Comment by Zambyte 5 days ago

At 1500 tps, "uh" is about 0.7 ms, instead of 200-300 ms for a human.