Running a 28.9M parameter LLM on an $8 microcontroller

Posted by boveyking 2 days ago

Comments

Comment by titzer 1 day ago

It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:

The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!

Comment by brucehoult 1 day ago

Don't forget the 128 bit vector ISA with 32 registers, supporting up to 64 bit int and FP, and with LMUL=8 you can process 1024 bits with a single instruction (at 3 cycles per 128 bits for most operations). Fully supported by GCC and CLANG (xTHeadVector) and compatible with RVV 1.0 with just a command line switch if you use the C intrinsic functions. (a lot of code working on 8 bit elements is binary compatible with RVV 1.0 too e.g. typical memcpy(), memset(), memcmp(), strlen(), strcpy(), strcmp())

When I bought my 64 MB Duo they were $3!

Then for a long time they were $5 for the 64 MB, $7 for the 256 MB, and $10 for the 512 MB.

Sadly, like everything else, they've gone up considerably this year.

https://arace.tech/products/milk-v-duo

https://arace.tech/products/milkv-duo-s

Comment by pjmlp 1 day ago

Meaning, minus possible required bug fixes, all the frontend languages supported on a full instal of either GCC or LLVM.

Comment by titzer 1 day ago

The wild thing is that this SoC is a heterogenous compute device. It has three different kinds of cores: a pretty beefy arm64 core and two different RISC-V cores: a 1Ghz one for running Linux and a 700Mhz one dedicated to running a real-time operating system. The arm64 core can also run its own OS.

Comment by brucehoult 14 hours ago

Also a user-programmable 300 MHz 8051 (8 bit) primarily there to manage timers and interrupts and wake up the big 64 bit cores.

Note that the application-processor 1 GHz C906 and Arm A53 are either/or, you can't run both at the same time — similar to the Arm M33 and RISC-V Hazard3 core pairs on the Raspberry Pi RP2350.

Comment by pjmlp 1 day ago

I haven't noticed it, rather cool!

Comment by skippyfish 1 day ago

You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task.

And I hope it stays that way, I don't want MCU shortages...

Comment by thomascountz 1 day ago

Many artificial environments are useless to tasks they were not designed for until somebody experiments, tests, redesigns, and iterates.

I'll give a specific example apropos of TFA. Computer vision models were never run on MCUs because they were constrained to the point of being useless for this task, but then someone tried the impractical, and now it's trivial[1]

Regarding MCU shortages, you should be worried about the supply chain, but I don't see the impact really being from running LLMs on ESP-32s, of all things.

[1]: https://github.com/espressif/esp-vision

Comment by dannyw 1 day ago

Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits.

I think most of my embedded projects aren't that useful, but they've taught me a lot.

Comment by skippyfish 1 day ago

I'm not disagreeing with you, I think this goes beyond impractical, it's doomed from the get go.

In my book, impractical means "I built a cuckoo wristwatch". Beyond impractical: "I built a cuckoo wristwatch but there was no room for a working mechanism".

Comment by rufasterisco 1 day ago

I don't see the relationship between "fun and learning" and "(beyond) impractical".

I enter most of my learning project from the assumption that I could just buy/install whatever I am building and save time/money.

Please consider seeing things from a different perspective than "your book".

Not saying this to antagonise you, but because a lot of time reading comments like this makes other posters less willing to share their impractical efforts, and would love to read more of those, not less.

Comment by dannyw 1 day ago

Learning how to make a cuckoo wristwatch and everything that involves helps you acquire skills, even if what you build is fundamentally useless.

Comment by jeroenhd 1 day ago

There's a neat project out there that can read your water meter into home assistant using an ESP32+camera+computer vision. I imagine a small TPU like this could be very useful for similar projects. More compute means higher resolutions and more reliability.

Comment by r0b05 1 day ago

Interested in this. Link please

Comment by solarist 1 day ago

AI on the edge https://jomjol.github.io/AI-on-the-edge-device-docs/

Comment by titzer 1 day ago

I want to run music models on the Milk-V and have them jam in realtime with me.

Comment by imtringued 1 day ago

How is it useless? If it has an NPU it is literally built to run models.

You're just extremely biased in what you consider to be a useful ML model. For example, for some strange reason you think only LLMs exist. The model must be as big as possible or else it is pointless.

Training custom non-LLM models for specific tasks so they run on a resource constrained device? You must be insane.

Comment by NooneAtAll3 1 day ago

isn't floor price 0.10$ for last couple years?

Comment by skippyfish 1 day ago

Well, that's sub-$0.50. But yeah, CH32V003 is in that ballpark, and some of the cheapest Microchip and Infineon products are around $0.20.

It's almost never worth it to buy the cheapest chip unless you're making a million of something, but there are very good ones around $1-$2, and $5 is the upscale stuff.

Comment by londons_explore 1 day ago

There are 2 cent microcontrollers.

Used in the cheapest devices you wouldn't even imagine to contain any software - eg. To control a touch sensitive on-off switch.

Comment by NooneAtAll3 1 day ago

where can I learn about those?

Comment by RetroTechie 1 day ago

Dave "EEVblog" Jones did a review of a (then) $0.03 microcontroller (Padauk) a while back (sorry, I don't know the exact episode).

Iirc an important caveat was that it was a one-time programmable (OTP) part. So you buy a bunch of them, programming failure or firmware-under-test doesn't work? -> toss the part. Of course that isn't an issue for a $0.03 part. But it can be an issue in terms of a board you want it on. Either that means discarding (breakout) boards too, or for development you'd need some kind of adapter to put bare ICs in.

Such annoyances only make sense for high-volume, low cost applications. Which is eactly where parts like that go in.

Comment by NooneAtAll3 1 day ago

looks like Padauk PMS150C is back up to 0.08$, oh well :(

it's fun reading all the reactions to it (like "it’s cheaper to program a Padauk PMS150C to be a logic-level converter than to just buy a logic level converter"), but the magic is gone more-or-less

but some ideas inside it and padauk's ecosystem as a whole seem impressive https://jaycarlson.net/2019/09/06/whats-up-with-these-3-cent...

edit: PMS152 seems to still be 0.03$ !

Comment by RALaBarge 1 day ago

Not trying to be snide, but I copied and pasted their entire comment verbatim into google and it came up with Padauk PMS150C or Puya PY32 series

Comment by onion2k 1 day ago

I bought 5!

What are you going to do with 120 of them?

Comment by moffkalast 1 day ago

A compute cluster for generating bad math jokes.

Comment by pjmlp 1 day ago

Which is why folks should stop thinking that coding in Assembly, or C is the only way, as if microcontrollers were stuck in the 80s hardware.

Heck, that one would have no issue running Xerox PARC OSes.

Naturally there are still use cases were a PIC with 4KB would be the best option, but lets not behave as if there isn't anything better in most cases.

Comment by moffkalast 1 day ago

It really puts into perspective how much of a leech ARM has been on the entire industry. What being a monopoly does to a mfer.

Comment by titzer 1 day ago

I think that is pretty ungenerous. Before ARM, ISAs were not a commodity, and there were only closed, proprietary implementations of them (usually from a single vendor). Arm licensing its IP and actual designs was hugely beneficial for the broader ecosystem and led to their prevalence in the embedded space. The toolchain and software network effect made it a no-brainer to either reach for a completed Arm design, contract a customized one, or build your own.

The arm experiment ran its course though; the power one vendor had in the marketplace started to be abused for the benefit of the IP holder and detriment to others. Now RISC-V is going one step further with a completely open ISA and also completely open designs. This is an excellent development in the nick of time.

I wonder how Apple feels about arm64 and RISC-V now. They could have probably bought ARM at any point but maybe never considered it to avoid anti-monopoly blowback.

Comment by pjmlp 1 day ago

Sure, because the jungle of 8, 16 and 32 bit microcontrollers stuck on 1980's mindset was so much better.

Also you can reach out to ESP32 as alternative, plenty of maker projects using them.

Comment by helsinkiandrew 1 day ago

Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.

Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!

Comment by mschild 1 day ago

> tips about dental hygiene

They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app.

> advertise toothpaste

God please no. I don't want to have to look for an adblocker for a toothbrush AI.

Comment by left-struck 1 day ago

My goto for what I hate about modern tech is toothbrushes having Bluetooth and needing apps. Not that I hate all modern tech but if it needs an app I probably will.

Comment by EtienneK 1 day ago

And this is why I am extremely bullish on Apple.

Comment by walrus01 1 day ago

PLEASE DRINK A VERIFICATION CAN

Comment by rao-v 1 day ago

This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!

Comment by Lerc 1 day ago

One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.

PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.

The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.

Comment by hgoel 1 day ago

Wouldn't a layer per MCU be heavily bandwidth constrained?

Comment by monocasa 1 day ago

Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.

I can see that being cheaper to bitbang with PIO than to actually compute.

There's certainly some latency stack up, but throughput should be remarkably good.

Comment by NuclearPM 1 day ago

Run the numbers before you waste time here. I doubt this will work.

Comment by Lerc 1 day ago

PIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for.

They have a single cycle double multiply per core, and the interpolators give you a heap of ability

The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins

You are obviously not going to get big TOPS from it because a Trillion is a ridiculous amount anyway. But never underestimate the power of controlling the whole pipeline.

Ultimately none of the other things I'm doing with MCUs are practical, why would this to be any different.

Comment by JK-Swizzle 1 day ago

Running some quick numbers shows you should be able to get >1Gbps. But I seriously doubt you could get those speeds in reality. You would need to get them perfectly in sync, which would likely take a dedicated board and some great knowledge of the oscillator.

As someone who has done a reasonable amount with PIO, I do not think this is possible. However, that should not stop you. If you get it to work, please ping me.

Comment by chrishynes 1 day ago

Why can't this scale to run much larger models on CPU backed by flash with good access patterns?

Comment by AussieWog93 1 day ago

Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD.

I think they got something like 10 _seconds per token_ (not tokens per second).

EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri

Comment by 3eb7988a1663 1 day ago

That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient.

As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.

[0] https://shir-man.com/tokens-per-second/?speed=1.8

Comment by qingcharles 21 hours ago

I used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning.

I could see where you could set up some coding task and let it churn all night.

Comment by kamranjon 1 day ago

So while SSD streaming is interesting I'm not sure it's exactly the same thing as the per-layer embedding that is being utilized in tandem with streaming here. To utilize per-layer embedding, it would have had to be trained that way, which GLM 5.2 was not.

Comment by DoctorOetker 1 day ago

more interesting would be using some kind of FPGA to logic glue each RAM socket interface bitplane to a hard drive (so a collection of hard drives with ridiculous collective bandwidth). Perhaps a single RAM socket contains actual RAM and the linux kernel would have to be modified to only use the real RAM memory region for OS and inference software, with the inference software rewritten to stream LLM weights deterministically from the hijacked RAM slot physical memory regions. Obviously the FPGA can't truly achieve the CAS latencies over the HDD (unless the HDD firmware was rewritten so it can predict the next deterministic token sufficiently in advance to cache the result and stream it just in time to FPGA then "RAM" socket...) but even if the HDD firmware can't be reprogrammed for some reason, the FPGA knows what memory address will be deterministically fetched next, so it can make the requests to the parallel array of HDD's ahead of time.

Comment by npodbielski 1 day ago

Rig with 6x RTX 5090. I would not say it is RAM only

Comment by 3eb7988a1663 1 day ago

Unless I am mis-reading the benchmarks, some of those are run as CPU only, no GPU involvement.

Comment by Rohansi 1 day ago

My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.

Comment by monocasa 1 day ago

It's also mapped into the address space so there's very little extra latency in grabbing the embedding as opposed to something like nvme that will have to setup a command list, submit it to the drive's microcontroller, wait for the op to be processed, etc.

Comment by spacedoutman 1 day ago

>esp32-s3

This microcontroller is a beast, currently using it to do dev work on a pi4.

Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise

Comment by raphlinus 1 day ago

If you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO.

This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.

Comment by Rohansi 1 day ago

$8 ish gets you an ESP32-S3 board with PSRAM, flash, and two USB-C ports. The PSRAM and flash are specifically used for this LLM project. I can't find anything like that with the RP2350 for $1.

Comment by raphlinus 1 day ago

Entirely fair, $1 is just the chip, not the board. No question the ESP32-S3 is incredibly good value.

Comment by NooneAtAll3 1 day ago

While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights

Comment by DoctorOetker 1 day ago

It's quite sad people collectively behave as if leaderboards have served their time.

In the small parameter regime there is no room for benchmaxxing, so instead of leaderboards becoming useless, their utility was merely reduced to establishing ever smaller models with similar performance on the benchmarks, forcing compression or redundancy to be recognized and eliminated at the modeling level.

Comment by anonymous344 1 day ago

i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?

Comment by Rohansi 1 day ago

Run a smaller LLM. It won't be as smart but the one shown here isn't either.

Comment by kamranjon 1 day ago

Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.

Comment by althea_tx 1 day ago

This is a really cool project. Thanks for sharing!

Comment by caruasdo 1 day ago

How accurate is this quantized model?

Comment by cr125rider 1 day ago

9.7 tokens/sec actually seems like a lot! That’s fun!

Comment by 2 days ago

Comment by ReactiveJelly 1 day ago

Finally, an LLM that can run on my i5

Comment by smy20011 1 day ago

Super cool, thank you for sharing!

Comment by mcgin 1 day ago

How lovely j

Comment by rvba 1 day ago

Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive

Comment by vitally3643 1 day ago

Reads are effectively infinite. I'm not aware of any upper limit shorter than "end of the universe" timescales.

Flash wears out from writing and the answer is in the tens to hundreds of thousands of writes per cell.