Petals: Run LLMs at home, BitTorrent-style

> For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines

But isn't what everyone want almost the complete opposite of this? For smaller LLMs you can already run them with consumer hardware, sometimes consumer hardware 2x, but it's the rest of the stuff where distributed inference could actually "democratize" access, as almost no one has 500GB VRAM available at home, or whatever the latest SOTA local models take nowadays.

Relevant: Why Switzerland has 25 Gbit internet and America doesn't https://news.ycombinator.com/item?id=47652400

Not a neural network though ;) https://biologyinsights.com/do-fungi-have-a-nervous-system-t...

Too early too in that later (hopefully) each node can serve a full model so it is then a simpler share tit for tat of complete models.

A planetwide hallucination – sounds awful.

Too early? the project has been going for years now...

What? LLMs are perhaps the least bandwidth-constrained thing you can do over the network.

If you run a local model at a good pace, you're emitting tokens at, what, 60 tokens/sec? That's, very roughly, 300 bytes/sec. You can get that speed with dialup from the 80's! And when it takes a half second just to ingest the input tokens (to get to the part where you start generating the reply), you're not gonna care about even a crazy 250ms latency.

Unless you think that the model should be downloaded on-the-fly in response to a user request, there's zero reason for an LLM to need any sort of fast connection.

Can't even scroll the website on mobile, too much going on

That explains the very old models mentioned on the page. There are models you can run locally on anything that would be more useful than llama 2 70B. e.g. the new Prism Bonsai 27B ternary model is 6GB, runs on any modern Mac, any PC with a gaming GPU bigger than 8GB (12GB or 16GB would be better), and probably runs on CPU faster than this runs any of those models. Gemma 4 E2B or E4B or 12B in the QAT quantization are fantastic for their size, and run on a lot of stuff. Models at every level have gotten a lot better since 2022.

Maybe it's time to refresh this idea then. There's a clear demand for running LLMs at home, but the good ones are too big, and you're not using it most of the time. Sharing with others sounds like a perfect solution to me.

Can't wait to run a Sybil farm of subsidized proxy nodes and log all the juicy prompts.

Distributed training is much harder than distributed inference but not impossible. See the recent development of DiLoCo at Nous Research and Prime Intellect.

I've occasionally written comments about the difficult of achieving proof-of-useful-work mechanisms and I should probably find or write a standard post about this question.

In order to be as decentralized as Bitcoin and related PoW mechanisms, a proof-of-work mechanism should have

* the ability to create an unbounded number of instances of the problem deterministically from numeric seeds

* the ability to scale difficulty up and down using a difficulty parameter

* in a way where anyone can easily confirm that a given problem instance corresponds to a given seed, and anyone can easily confirm that a given solution correctly solves a given problem instance

I'm not aware of a proof-of-useful-work mechanism that meets these criteria. (I'm also not aware of an argument that it's impossible to have one. Zero-knowledge proofs might actually go far toward making it possible in the future. but it's kind of complicated.)

As we've seen, that's a solution looking for a problem.

You can easily build a localized, access-controlled compute grid using Apple Silicon hardware. Haven't tested but as far as I can see, the key to making it performant would be keeping full models resident on individual nodes rather than attempting to distribute the tensor processing across the network.