Cerebras Deploys Trillion-Parameter AI Models

Posted by gmays 4 days ago

Comments

Comment by keyle 4 days ago

Cerebras is good on paper but all I can reasonably access is "GPT OSS 120b, Qwen 3.8 27B" which aren't very good for my use.

Openrouter on nitro gets me there. Although the TPS is 5X less still.

I have tried GPT OSS 120b on cerebras and it flies. But it's mostly wrong.

Comment by preommr 3 days ago

> GPT OSS 120b

It's extremely frustrating that these fast models are very old at this point. Chat jimmy launched in february, and nothing has come of it, not even GA let alone improvements with intelligence.

AI is currently moving so fast, that might as well be a lifetime.

We get SOTA from anthropic/openai every few weeks on models that seem to focus on one useful thing (e.g. opus) and then 5 useless things like booking restuarants. We're ignoring really useful applications for some reason.

It took jev to shake things up. I remember seeing reactions that were basically "people want this? the big labs probably have some version of this lying around"... and it's like yea, maybe these labs need to fix their priorities a little bit.

Comment by everforward 3 days ago

I suspect they’re ignoring it because there isn’t money in it.

Jev is super cool, and it’s super cool that it can handle novel categorization. Most organizations aren’t spending the big bucks on novel classification, though. They’re doing the same thing a billion times, which means Jev has to compete with having a custom model built and doing your own inference.

It’s very hard to build something generic like Jev, but it’s fairly manageable to build something more specific that handles a single kind of task. Especially at the scale where OpenAI would care about your attention.

I don’t really understand neural networks and I managed to train one to do hand gesture recognition on an old Leap Motion. It’s a pretty limited domain, but it trains on the CPU of my decade old laptop in under 3 minutes. The accuracy is quite good too, especially when you discount errors from the Leap (like misidentifying which finger is extended).

Comment by Miraste 3 days ago

When talking about Jev, people love to bring up the whole "bitter lesson" idea that generalization will win out. Maybe it will in the end, but I tried most of the recent Jev-likes on my own not-very-complex use case and they weren't competitive with domain-specific classification at all.

Comment by weitendorf 3 days ago

General classifiers are actually incredible for two things:

1. inference on arbitrary context / world knowledge with fast, structured output

2. I almost don't want to say it because I don't know if people realize this yet... but that makes traces from any kind of non-AI process or system "learnable", you can easily model it by by just asking it to predict the next outcome during learning and compare against what happened.

which also means it could also learn new strategies through rollouts in a way that generalizes across semantic space which means...

so these models are actually quite promisingly useful for existing companies because they do have general traces of what happened in their company and want to discover ways they can tweak their business processes, and then there is durable demand for models you can embed within a product or UX flow without exposing your user to slop + then train or customize and build real IP or custom capabilities

general classification itself is quite interesting IMO for applications that need to make judgment calls or select one of many UX actions, and a much better fit for this than LLMs. So if you think about it, this is the type of thing businesses might be calling a lot - every trace (eg user session, package shipment, sales motion, marketing campaign, job outreach) + proactively throughout dynamic experiences to decide whether to eg end a call or which of multiple UIs to manipulate

this is just an unfamiliar product experience to most people because it hasn't really been bootstrappable with a strong general baseline until now. But now it's posible and I think it will be very col.

Comment by everforward 3 days ago

I think you’re misunderstanding me; I’m not saying places will use an LLM instead (whether cloud-based or on-premise).

I’m talking about them training their own models with a much more specific purpose.

You brought up choosing a menu option in a UI. Jev is really good at something like “find the menu option that best fits the users prompt”. That’s general, and requires a big complicated model.

Companies are usually more interested in something less variable. Ie the menu option they’re looking for is fixed. “Take an arbitrary website and find the Login button”. That’s a _way_ simpler model, and well within the abilities of many large companies to train.

You only need to pay Jev if there are so many variables the model basically has to be general.

Eg classifiers are a separate thing from LLMs and are literally only capable of structured output. It will output one of the X options it was trained to recognize.

I think some of this is that the Venn diagram of people who like AI and the people who like machine learning isn’t a circle. There’s a lot of “intelligences” out there that aren’t an LLM or decision model.

Comment by p1esk 3 days ago

what might be useful to you (e.g. coding) might be completely useless to 99% of the people

Comment by halJordan 3 days ago

Chatjimmy got taalas aquired

Comment by froh 3 days ago

yes, and by amd which announced to integrate it with their hardware.

Comment by catlifeonmars 3 days ago

I don’t think I really understand the motivation to move as fast as possible. Who cares if you’re behind by 6 months? Objectively you’re still ahead of where you would be without the 6 months old tech.

Comment by nexus2045 3 days ago

This - I use Cerebras supported models here and there, the speed is amazing and I can see the potential with having almost instant inference, but the models are so outdated at this point.

Cerebras if you're reading this, get your act together - maybe that will also help your stock price. Give consumers more options, not just to OpenAI.

Comment by jubilanti 2 days ago

I'm sure they were planning on running old models forever, but now that they've read your comment, they'll be motivated to get back to work.

Thanks for your contribution to the discussion.

Comment by splicebot 3 days ago

Cerebras isn't public and are almost certainly going to exit by being acquired. So demos to the public don't matter.

Comment by adityamwagh 3 days ago

They are a public company since quite a while.

https://www.cerebras.ai/press-release/cerebras-systems-annou... https://uk.finance.yahoo.com/quote/CBRS/

Comment by hijinks 3 days ago

they are public $CBRS

Comment by adastra22 4 days ago

GPT-5.6 Sol Ultrafast is running on Cerebras hardware.

Comment by Topfi 4 days ago

That has never been made available even to limited customers as far as reporting goes. Currently available ultrafast on GPT-6 Astra and GPT-6.1 Sol are all using Nvidia for good reason [0].

The fact that Cerebras, which has a willing and major customer like OpenAI begging at their heels to deploy on their hardware, now posts about a 2.8T open weight model says a lot.

[0] https://blogs.nvidia.com/blog/gpus-openai-gpt-6-astra-ultraf...

Comment by chrismustcode 4 days ago

You can get 6.1 sol on ultrafast.

For some reason though on openrouter the tps of 6.1 sol ultrafast is usually lower than opus 5.5 fast. Is there readings off as surely cerebras hardware is faster than low user density nvidia still?

https://openrouter.ai/openai/gpt-6.1-sol

Comment by Topfi 4 days ago

GPT-6.1 Sol Ultrafast also uses Nvidia hardware, not Cerebras, just far better provisioned vs how they usually provide inference allowing for higher throughput.

Comment by fastball 3 days ago

"Much better provisioned" might be overselling it. Mainly they are just serving very few customers per Nvidia GPU, which is an expensive solution to the problem.

Comment by stingraycharles 4 days ago

So then how is OpenAI using Cerebras? Or are they not?

Or, from a different perspective: who is using Cerebras for what purpose?

Comment by davidweatherall 4 days ago

Jane Street bought up all of the Cerebras x OpenAI compute I believe. Couldn't find a great source, this tweet has a good few replies citing other sources as well https://x.com/cryptopunk7213/status/2106834536868856223

Comment by stingraycharles 3 days ago

Oh that would make a lot of sense, basically supply limited.

Comment by keyle 4 days ago

It is not available to "developers".

Comment by madaxe_again 4 days ago

Sure it is, you just have to pay for it. TANSTAAFL.

Comment by 2001zhaozhao 4 days ago

I was a subscriber of their now-unavailable coding plan. The 1000 TPS was very unique at the time, but nowadays their qwen 3.8 gets roflstomped by Mercury 3 at both cost and performance at similar TPS speed. There's not much reason to use them unless you're an enterprise with a custom deal.

Comment by ekidd 4 days ago

Qwen3.8 27B is a dense model with expensive K/V caching. It's fantastic if you have 32-48GB of VRAM and want to run 1 or 2 agent sessions locally. But if serving batch inference for many users, it's expensive.

DeepSeek V4 Flash 0731 is dirt cheap at scale, less than 10% of the cost of Qwen3.8 27B. This is because it's an MoE and it uses much less VRAM for KV cache. And it's generally a better model. V4.1 is also cheap at scale, though I've heard mixed reports on the optimized K/V caching.

You should not be running Qwen3.8 27B on an API normally. Maybe Cerberas's hardware is bad at batching or supporting more efficient K/V caching schemes, or something like that?

Comment by cyanydeez 3 days ago

If you have 48GB of VRAM, you should be able to get Qwen3.8-Flash-Next running and from my use, it's basically the tipping point. 27B is great for a bit of work, but I have QFN running non stop (when I can gather the correct ideas for it to implement).

Comment by xmddmx 3 days ago

What quant and engine are you using? The smallest ones I’m seeing on ollama are > 100GB e.g. https://ollama.com/library/qwen3.8-flash-next/tags

Comment by sgsjchs 3 days ago

This arch uses engrams aka per layer embeddings. Not all of those GBs need to go in vram.

Comment by cyanydeez 3 days ago

Halogen flash server for strix halo with quant 4. Its offloading the PLE.

Comment by piterrro 3 days ago

Lots of folks here say Cerebras serves outdated models.

Here’s my view: lets assume the above is true and the average serve with a delay of 6-9 months. Given that, its fair to expect them to start serving DS flash v4, glm 5.3, mimo 2.6 pro or kimi 3.1 pro in a matter of months. DS flash is already fast to the point I’ve became a bottleneck. But my point is: Cerebras should soon release better models that are not that far from frontier ones, at a fraction of cost and with multiples of the speed.

Just be patient and plan for the future.

Comment by pants2 3 days ago

Isn't Cerebras serving Sol 6.1 Ultrafast?

Comment by fastball 3 days ago

No, Cerebras is serving GPT-5.6-Sol (released in July).

OpenAI is back to using Nvidia chips (with small batch sizes) for the ultrafast mode of their latest models.

That takes the speed down from from 750tok/s on Cerebras to 300tok/s, and presumably is much more expensive than the Cerebras solution, but Cerebras has supply constraints. I believe they are planning aggressive data center expansion in this quarter but much harder to scale up than Nvidia chips.

Comment by TiredOfLife 3 days ago

Open models are 3-6 months behind by themselves.

Comment by ilaksh 3 days ago

From my testing, the problem Cerebras has is they need to give people a taste to get new customers in, but since it requires so much hardware to get that speed and they don't have OpenAI-type money (< $3 billion compared to over $100 billion) and it's just incredibly hard regardless to manufacture that many of these types of crazy giant wafer nodes, there is just no way for them to keep up with the demand.

So what they do is they have these cheap plans that let anyone get a little taste of the incredible speed, but they aren't reliable at all, because they have to share limited resources dedicated to sales. I believe that is really the main purpose of like the $50/month or whatever plans -- part of the sales funnel. The cheap plans compete directly with resources they need for their "real" paying enterprise customers.

The "real" plans start at like $1500 per month, and I assume those have reasonable service level reliability.

But actually I believe those $1500/month plans are totally worth it for some applications.

Comment by KeplerBoy 3 days ago

For which applications do you need that kind of single user throughput? A lot of agentic stuff is tool-heavy anyways other stuff are better served by sub agents / parallel work for more throughput.

Comment by Daishiman 3 days ago

Would seem reasonable for hedge funds to devise a pipeline of tools specifically made to be taken advantage of by LLMs that do goal-based strategy at scale.

Comment by xpct 4 days ago

Ah, this prompted a few questions from me. Perhaps someone with more hardware background could chime in.

They don't really show it here, but supposedly a Cerebras' chip is sub-divided into a 2D mesh of what they call PEs (cores). So data flow generally flows in two axis, and maintains consistent timings to its 4 neighboring cores. They show the layers flowing from left-to-right being streamed by PEs, but I wonder how true that ends up being: maybe the communications are easier to stream via diagonals, or some other shape that has better topology for data flow.

Then, even from die shots of CPUs and GPUs we often see that cores are tiled as squares, and those don't have the same cross-core connection and streaming requirement. Now, I assume that some parts of what we call a core can be moved around more freely while maintaining whatever power/heating/timing requirements we have, so perhaps it's easier to 'force' a square by moving those loose parts around. But why do we confine those to squares? Is super-core layout just something not worth investing time into, like how interconnects are timed, non-convex tilings, and distances to I/O?

And for streaming architectures like Cerebras, why aren't other types of tilings considered, something like a hexagon grid? This could have 3 inputs / 3 outputs, which could allow more complex data flow (?). Similarly, we could have 8 connections with longer diagonals in a square tiling, etc.

I guess, what are the tradeoffs in place such that so many chip designs seem to reuse square tilings both locally and globally?

Comment by bubblethink 4 days ago

The reason 2D grids work well is because you can split tensor axes along whatever dim you want across PEs (batch, sequence, hidden, etc.). And your per PE code is relatively simple. It's of the type : get data from neighbor, compute my subtensor, send to neighbor. And you can implement broadcasts and reductions along any axis.

Comment by dnautics 4 days ago

You have to route around disabled subchips though.

Comment by Nevermark 3 days ago

My guess is it is more like the opposite. They distribute spares evenly along every array in every dimension, so that there are always the same number of selected and unselected cores in each core-set.

With flexible addressing and bus-linking, along any set-line. So configuration happens the same way, with the same efficiency, regardless of whether any of the unselected elements are failed or not. As long as failures don't exceed the unselected-elements per element-array-set number.

And that number of unselected cores per element-array-set, is chosen to put wafer yields where they want them.

Trading a generous fixed wafer under-utilization for high yields, uniform configuration and processing efficiency, despite variable element failures.

Comment by dnautics 2 days ago

[dead]

Comment by plasma 4 days ago

A few years old now, but their co-founder whiteboards how their chips work https://www.youtube.com/watch?v=7GV_OdqzmIU

Comment by wmf 4 days ago

The EDA tools that are used to design chips do not support diagonal wires so that kind of rules out hexagonal grids. With only horizontal and vertical wires you end up with rectangular/square grids or rings.

Comment by ithkuil 4 days ago

Do you need diagonal wires for a topological hexagonal tiling?

Wouldn't a rectangular brick layout be topologically a hexagonal tiling

Comment by xpct 4 days ago

If that is so, would it be entirely to decrease search space for layout algorithms? I believe there isn't anything in the fabrication process that couldn't model diagonals.

I could see that square packing is more simple, but by shifting elements per row I don't see why it couldn't propagate at odd angles.

Comment by yvdriess 4 days ago

That's also my assumption. Place and route has to work out an NP hard problem on billions of wires.

Comment by colordrops 4 days ago

Are the next generation of Cerebras chips designed by the current generation? Maybe it won't be recognizable as a particular layout but rather a chaotic but functional vibe layout.

Comment by octoberfranklin 3 days ago

Oh the tools support it. The process doesn't.

Diagonal wires worked everywhere at 180nm and on a few fabs at 90nm if you rotated a whole metal layer (i.e. all metal-3 wires are diagonal to the metal-2 wires) and basically nowhere beyond 90nm.

OPC killed it. Too hard to OPC the diagonal stuff. And even if you rotate an entire metal layer, the contacts between a rotated and un-rotated layer would require enormous metal enclosures.

Comment by eulgro 4 days ago

At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?

Comment by drum55 4 days ago

For a fairly weak comparison, modern Bitcoin mining chips are around 13J/TH, so around 0.32TH per token, or 320 billion double-sha256 mining operations per token served by this system. Peanuts are also about 4200 Joules, or almost exactly 1000 tokens per nut assuming you could efficiently turn it into electricity to run this chip. Just as a frame of reference, I don't think this actually serves to make any point.

Comment by taneq 4 days ago

Pay peanuts, get Shakespeare?

Comment by shrubble 4 days ago

Each Nvidia rack is 120 kilowatts, so the power consumption is about the same as 10 racks of GPUs, FWIW.

Comment by therein 4 days ago

Extreme speeds are actually interesting if you think about it. We are apparently getting some interesting results from LLMs nowadays. If they would put 100x effort in 10% the time it takes, or imagine 1% where how long tool calls take become the bottleneck, that would actually probably feel like a really special kind of "super intelligence".

Comment by root_axis 4 days ago

It would absolutely be a phase change in practical utility, but it still wouldn't be "super intelligence".

Comment by esperent 4 days ago

I think that just like there's tons of different kind of "superheroes" and nobody disputes whether they're genuinely deserving of the name or not, there could be many valid types of super intelligence.

If something is roughly as intelligent as a normal human expert in a given area, but 100 times faster, that would be a valid example of super intelligence.

I'm sure that by some definitions like this we've already discovered narrow super intelligence.

For example - breadth of knowledge. Is knowledge a kind of intelligence? If so then current agents absolutely have a breadth of knowledge which is superhuman.

Comment by msm_ 3 days ago

Is computational ability a kind of intelligence? If so then 20 year old calculators absolutely have a computational ability which is superhuman.

I'm not sure what my (or yours) point is, but like with all of AI it's a moving target - I still remember how image recognition was deep in researchy Artificial Intelligence territory. Nowadays it's such an obvious building block, nobody thinks of image classifiers as "intelligence".

Comment by esperent 3 days ago

> Is computational ability a kind of intelligence? If so then 20 year old calculators absolutely have a computational ability which is superhuman

100 years ago there's not a single person who would deny that computational ability is intelligence.

It's only since calculators because really good at it that you'd even think about asking the question.

Comment by root_axis 3 days ago

I don't necessarily object to your definitions, but the most widely used one in the popular media describes an AI that is so intelligent that it can trivially wrest control of the world away from humans - that's the definition I was referring to.

Comment by esperent 3 days ago

> widely used one in the popular media

I'm not really interested in popular media definitions of things.

Comment by root_axis 3 days ago

The popular media is driving the discourse around these topics, nobody is interested in the arbitrary definitions of anonymous randos on niche internet forums.

Comment by lifeisloving 4 days ago

Or you'd just be making the same mistakes experts do but at a 100x clip.

Tortious really does win the race, and there is no such thing as a free lunch.

Comment by yababa_y 3 days ago

https://www.lesswrong.com/posts/semvkn56ZFcXBNc2d/superintel...

5 Forms of Superintelligence (2018, review of Bostrom's Superintelligence from 2014)

Featuring .... the SPEED SUPERINTELLIGENCE! It really has been considered that way, even in scifi prior to the current AI epoch.

Comment by imtringued 4 days ago

It would be a super intelligence in terms of thinking speed, not necessarily be super intelligent, mind you.

Comment by lifeisloving 4 days ago

More like lighting a fire in your codebase at the speed of light.

Nobody needs this, and the quality of models they're deploying, or all llms for that matter, require heavy human review (if you care at all about maintainability).

One developer can barely keep up with the daily output of multiple llm sessions writing code at 50 tps.

Im convinced anyone confidently thinking 2k tps is a no brainer is probably not concerned about the quality of their work (many such cases).

Comment by Tade0 4 days ago

Occasionally it takes me at most 10 minutes to figure out that a solution the LLM took an hour to churn out is not up to par and I need to throw it away entirely.

There are diminishing returns, but half of the rate you mentioned would already help me keep focus.

Comment by drob518 3 days ago

You’re thinking about this the wrong way. Yes, any individual developer doesn’t need this much power all the time. But any individual developer would like to get an answer quickly, which this does. You share the speed and capacity of this over a whole organization. Any one developer only uses a fraction of its capabilities at any given time. Getting an “instantaneous” answer is quite helpful. Think of it like a shared compiler; a developer is not constantly compiling, but having an “instantaneous” compiler is quite useful.

Comment by cyanydeez 3 days ago

most of the time developers arn't looking for answers. They want to implement some feature. And I'd hazard most developers arn't using these things as a crafter implementer but a frankenstein of architect, implementer, tester, builder and deployers.

And the dev, if they're sane, wants to actually evaluate what it's doing in those arenas. Speed might be a psychology killer then. Actually being able to see the development process as it does it's good think and it's wrong think is a benefit for local development.

I don't think speed of tokens is all that magical if there's a huge imbalance of ability to "human in the loop" it's processs.

Comment by drob518 3 days ago

Code is an answer to a coding question or direction. Getting an answer quickly is always better than getting an answer slowly. You can evaluate the answer at your leisure. The human can take as long as the human wants. Meanwhile, the infrastructure is generating answers quickly for other people. Faster is always better, even with a human in the loop.

Comment by lifeisloving 3 days ago

No, faster is not always better, there's a reason why we have cliches and literal nursery rhymes about it.

You're larping too, can we bring NFTs back so the the dunning kruger vibe coder commenters on the internet find something else to fixate on?

Comment by drob518 2 days ago

1. You don’t know anything about me, so don’t presume.

2. Which nursery rhyme? The tortoise and the hare is a fable, not a nursery rhyme. Even if you think the lesson was “faster is not always better,” which it wasn’t, faster is sometimes better. And in particular, when a machine is involved, all things being equal, faster is better. Would you rather use a fast compiler or a slow compiler, all things being equal except cost to keep the analogy with the original? Compiler A produces the same binary with the same optimization levels a compiler B, but in 1/10th the time. Assuming you can afford it or someone above you in your organization is willing to buy it for you, you take compiler A. If you can’t afford it, you lust after it. Why, because a slow compiler results in a slow iteration loop no matter what. With an instantaneous compiler, the loop is as slow or fast as the human part of it. You want to think some more, rearchitect this or that? Fine. Your code is always ready to run.

I get the sense that you’re anti-AI. That’s fine. I’m not sold myself, though I’m increasingly trending that way. For those that choose to use AI, I submit that a fast AI, all things equal (same output), is better than a slow AI for the same reason that a fast compiler is more preferable. Note that I’m not saying that faster overall development at all costs is preferable, or that we should just throw everything at the AI and stop thinking.

Comment by 8note 3 days ago

speed gets you to have flow though, rather than breaking your concentration with various waits that aren't actually times where you can stop and think because its 7s surrounded by tiring reading.

a milisecond response, that you can then spend 30s focused on is better.

its the same as a build tool. just slow enough build tools really suck, compared to either an instant build or a 30min build

evaluating the areas isnt the same as watching the llm harnes say "thinking" between submitting a request and getting a response

Comment by cyanydeez 3 days ago

I thiml of it as driving. If youre in kansas and on q highway with no features, go ahead, speed away. But the reality is, a human in the loop system needs a variable speed car. Like all driving, sure, some will go faster.

But the opinion is about psychology and expectations.

If you expect the singularity, then of course this sounds crazy. To christians, ghosts also sound crazy. It dont mean a think if yoire biased to expect AGIM

Comment by imtringued 4 days ago

At those token rates you could treat the LLM as a compiler and rebuild your software from scratch every week. That would mean the end for AI slop because people who spend the effort to build a process that creates high quality software from scratch can take full advantage of the one shot performance of an LLM without having to release the process itself.

Comment by lifeisloving 4 days ago

Youre completely missing my argument. Speed is not the bottleneck, its the review burden.

You need human review when working on anything not table stakes. You cannot write shitty code for an avionics package on an airplane under the assumption that if we encounter bugs, its no big deal because we can rewrite it super fast..

It seems like all these bullish takes on AI are coming from people who think all software runs in a browser, and simply exists to take stripe payments or serve ads. Most software is actually very high stakes and lives depend on it and have high compliance burdens.

Just because it can "one shot" (which actually means 90 tools calls and a whole lot of brute forcing against tests/linter/compiler, and possibly reward hacking) to something that runs, does not mean it will continue to run well.

People treat llm made software like disposable code. You're the only one using it, typically on a single machine with no real world multi-user testing, but for some reason we derive from the fact it worked well the 3 times we used it, on a single machine with a single user that it is evidence of llm solving programming. Its bogus.

Comment by visarga 4 days ago

You can lessen the human review burden by pitting multiple agents against each other. Ask 3 agents to implement the same solution, then set them up against each other in differential tests. This can find bugs in them or clarify the spec. When tokens get cheap it is worth having multiple replicas, taking advantage of different programming languages, libraries and agents. You don't keep one version, you have a population, and it tests itself. It's easier to implement again than to imagine testing scenarios anyway.

Comment by lifeisloving 3 days ago

[dead]

Comment by alchemist1e9 4 days ago

This is why I think there will be selective pressure for fast tools. It’s like an evolutionary process and fast tools will be preferred and created. It’s already happening and also why I think computer use is a dead end.

Comment by villgax 4 days ago

Lol these guys have another stupid quantization scheme they had mentioned earlier. Most models are stupider at least from them gateways.

Such a weird company with a terminally online on linkedin ceo, garbage devrel on X. Cant really be friends with CSPs coz everyone has their own chip, no consumer solution either, no neocloud would want to onboard them either coz its baggage no one wants to risk dev time/customer money with.

Comment by dannyw 4 days ago

As far as I understand, they have more demand and commitments than their production and scaling capacity. Maybe they're doing something right.

Comment by villgax 3 days ago

sure, we'll see

Comment by tgtweak 4 days ago

Citing full precision benchmarks then just going with 4 bit quants

Comment by IsTom 4 days ago

It's natively Q4.

Comment by hellohello2 3 days ago

I though 4 bit (mixed) quant is pretty standard, at least for local models that basically what everyone runs? Unless I'm mistaken it can really be a negligible difference in perf.

Comment by n_u 4 days ago

> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.

What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right

Comment by bubblethink 4 days ago

It's describing a different type of parallelism - pipeline parallelism. The network is laid out spatially on the wafer in CB's case. So weights are stationary and activations flow. A GPU is going to process the network one layer at a time. GPUs also use other types of parallelism (tensor and expert mainly in inference). Pipeline parallelism doesn't make much sense on a GPU for inference.

Comment by mmoskal 4 days ago

I think the point is they need several wafers to fit the model. Data parallelism between requires too much bandwidth (they only have so much edge real estate for io) so you have to do pipeline parallelism with say one layer per wafer. No one does it on gpu unless you can’t fit the model in one fast network domain (which is 72 GPUs nowadays for Nvidia). There is a Dwarkesh episode from like two months ago that explains this reasonably accurately.

Comment by edg5000 4 days ago

From reading xpct's comment in this thread it seems like they use effectively a very deep and wide pipeline by tiling the cores without central clocking or dispatch. In theory if a GPU is centrally organized it could simulate this behaviour, but I imagine that creates transfer overhead. Just speculating though.

Comment by pat64 3 days ago

Having dealt with them professionally my view is they’re a company with truly remarkable technology and a bafflingly inadequate team. They need to hire better. Almost stopped me using their stuff wholesale. Groq are somehow worse.

Comment by xnx 4 days ago

Cerebras is the Transmeta of the AI wave.

Comment by lifeisloving 4 days ago

Not sure why people would downvote this. Your not allowed to criticize any AI company on this site it seems.

Comment by Sabinus 3 days ago

With the utmost respect, if you've only been here 20 days you might not be fully cognizant of the site norms and culture. It's not about being pro or anti AI, it's low effort comments.

Comment by nl 3 days ago

Gatekeeping by someone who has only been here for 11 years isn't acceptable either.

Comment by Sabinus 2 days ago

Who's gatekeeping? Also "only 11 years"? That's quite a span of time.

Comment by embedding-shape 4 days ago

"X is the Y of Z" might not be insightful enough to prevent being downvoted. It's basically a saying, and the author likely spent like 20 seconds coming up with it. Probably I spent more time just writing this comment, than parent spent thinking+writing theirs.

Insightful critique of AI companies get upvoted here, more so than insightful cheering, I'd say.

Comment by kristianp 4 days ago

They've shown the calculation for number of concurrent requests limited by kv cache size. I guess for each new request loaded into the queue, the kv cache for that context will need to be loaded from a supporting storage server, that will need to have a lot of ram to store the active caches of multiple requests?

Comment by ACCount39 4 days ago

Less "storage server" and more "attention server".

A server that both holds KV cache in its own memory, and uses its own compute to perform attention operations. That way, you don't have to transfer cache entries server to server for every attention sweep - only the "requests" and "responses". Attention bandwidth stays server-local.

Comment by fastwafer 2 days ago

[dead]

Comment by ismailmaj 4 days ago

> For ultrafast latency, only activations need to be passed between the wafers, and this can be done via RDMA over Ethernet.

I believe it for decode but the article says nothing about how they make prefill fast on that interconnect story :/.

Comment by cfn 4 days ago

I tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.

Comment by scottcha 3 days ago

We can push 1k tps on our speed lane with the right request parameters with deep seek v4.1 flash. Is 2k a hard requirement?

Comment by cfn 3 days ago

I will check it out! I am using that exact model and it is very verbose when compared with the latest Muse, for example, increasing the completion time a bit too much.

Comment by draxen 4 days ago

are you evaluating using OpenRouter.ai or other routers/gateways?

they have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.

Comment by cfn 3 days ago

I am using opencode and Inception.ai (the fastest so far)

Comment by villgax 4 days ago

And the fact that you will never be benefited from the batching they’ve explained either, you’ll be paying for every token sequential or parallel unlike nvidia or amd gpus

Comment by cubefox 4 days ago

What's your use case?

Comment by system2 4 days ago

I bet it is large-scale data crunching for labeling or categorization.

Comment by cubefox 4 days ago

Probably not because this is usually not a time sensitive task.

Comment by amelius 3 days ago

High frequency trading? :)

Comment by cubefox 3 days ago

Fast LLMs seem to be mostly useless, otherwise it wouldn't be so hard to come up with realistic use cases.

Comment by system2 3 days ago

You think too negatively about its use. My company definitely needs it desperately.

Comment by cubefox 3 days ago

If it did, you would be able to specify for what, you can't specify for what, so it doesn't.

Comment by system2 2 days ago

I literally said it in my first comment, which is large-scale data crunching for labeling and categorization.

Comment by cubefox 2 days ago

Why is it time sensitive?

Comment by cfn 3 days ago

I need to have the answers faster it is that simple, I don't want the users to wait for more than a few seconds.

Comment by cfn 3 days ago

Sorry, can't say just yet but nothing as dramatic as what was guessed below :)

Comment by cyanydeez 3 days ago

what is a project that requires 2k tokens per second?

Comment by cfn 3 days ago

It's not launched yet and 2k isn't a hard requirement but faster inference of relatively simple prompts make a big difference for us. Cerebras also failed because it has very small context window.

Comment by jdthedisciple 3 days ago

Well, a weekend project that requires 345'600'000 tokens — you only get so many seconds in two days ;)

Comment by iugtmkbdfil834 3 days ago

I am invested, because I see this hardware as inherently.. neat. But did anyone do a comparison that is a little more.. human ( not purely benchmark based ). You know what I mean. There are things that make me pick chatgpt vs anthropic sub. Not all of it is absolute capability.

Comment by BatchJob 3 days ago

This conversation seems chock full of comments from ML experts. Im really impressed.

Comment by phkahler 3 days ago

59 wafers for kimi?!?! That sounds like some kind of architecture problem to me. I suppose if they're after a high tokens per second and this allows them to run a lot of agents at once and at high speed it might be ok.

Comment by latchkey 3 days ago

44GB

Comment by fastwafer 2 days ago

[dead]

Comment by xlayn 3 days ago

Imagine be able to buy this 2000t/s Qwen 3.8 27B card... you can put it in the express port of this really old laptop, or bluetooth and connect to all your devices... salivating...

Comment by jubilee33 4 days ago

Ok I'm sold, where to buy? :D

Comment by lisplist 4 days ago

If you made the mistake of buying their stock the past few weeks, you might wish you hadn't (or maybe it's an opportunity to buy low yada yada)

Comment by donutdonot 3 days ago

What's the reason for the massive dip? It seems their product works?

Comment by cute_boi 3 days ago

It was massively overhyped that we would see chatgpt sol at 750 tps.. But that never happened.

Comment by lisplist 3 days ago

For the handful of models you can actually use through Cerebras, they are incredibly fast. The problem is that they aggressively rate limit their APIs to the point where if you just run a single coding agent, your coding session will constantly stall out just due to how fast its able to hit those rate limits.

Comment by system2 4 days ago

Maybe it is the bottom now?

Comment by amelius 3 days ago

When will we see some speed improvements? Opus 5.5 is smart enough now, speed is what we need.

(though the agents also spend a lot of time in testing and building, which cannot be improved in speed a lot, I suppose)

Comment by russli1993 4 days ago

could push kv-cache to off chip cheaper storage like dram. Inference engine manages decode for a single user to run to completion before decode a different users so kv-cache fetch from dram to sram is limited.

Comment by lostmsu 3 days ago

What's the cache input discount relative to the regular input?

Comment by risphere 3 days ago

The issue is the models they offer are extremely outdated.

Comment by mlmonkey 4 days ago

A wafer looks pretty big. I'm surprised it has only 44GB of memory available. Doesn't that seem a tad .... low? I'm not a hardware person, but I would have assumed much more memory would fit into a wafer?

Comment by bubblethink 4 days ago

It's SRAM. It's the equivalent of 44 GB of CPU cache.

Comment by stingraycharles 4 days ago

Exactly, which is why it’s so incredibly fast. And expensive.

Comment by morphle 4 days ago

traditional SRAM memory has 6 transistors per bit. Dense SRAM can have two transistors.

Wafers can have trillions of transistors, 10-16 times more than the 4 trillion of the WSE-3.

1 Terabyte of SRAM per wafer will be the upper limit for a few years [1].

[1] https://youtu.be/vbqKClBwFwI?si=gbBeZ1B66_eGI81Z

Comment by GeekyBear 4 days ago

SRAM doesn't shrink as well as other types of circuitry do when process nodes advance.

Comment by Tuna-Fish 4 days ago

SRAM is not very dense.

IMO the next big shift in AI hardware is when some of the upcoming logic-compatible memory types mature. Inference inherently likes many small local pools of memory, with their own interface each. This is impractical with currently available memory other than sram.

Comment by imtringued 4 days ago

You mean it is unimaginably massive.

Comment by timsuchanek 3 days ago

This is amazing, and I'd love to see K3.1

Comment by noahcroghan 3 days ago

asdf

Comment by toccter 4 days ago

[flagged]

Comment by agentic-affairs 4 days ago

[flagged]

Comment by 840754587 3 days ago

[flagged]

Comment by coffecup25 3 days ago

[flagged]

Comment by R1ck404 4 days ago

[flagged]

Comment by happybox2016 4 days ago

[flagged]

Comment by zhyblife99 4 days ago

I'd want to know is what a 59-wafer pipeline costs in failure domain. The GPU deployment can lose a node and degrade; Because, here the model is the rack, so one bad wafer takes the whole thing down, and a pipeline that has to be live end to end doesn't get a rolling upgrade. How do you drain one?