Ask HN: Who is using FPGA for ML inference?

Posted by softwarewright 4 days ago

With RAM price inflation, I wonder if FPGAs can be used to offload inference processing without keeping weights in RAM? The available RAM would be for activations, KV Cache, context but not static weights. Weights could be streamed from disk. This approach is not for tokens/second but throughput at a lower cost. Possibly better answers/kHh? I've started researching this, but wonder if others have considered/tried this?

Comments

Comment by sphyrna-029 2 days ago

For small models it is very do-able on entry level FPGA. With a PYNQ-Z2 at int8 you could potentially fit 1m params. It would be a fun exercise in writing a inference driver and implementing systolic arrays on a FPGA.

Comment by wmf 4 days ago

FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.

Comment by softwarewright 4 days ago

Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could achieve parallelism. Won't be fast, but, many requests could be processed in parallel, as each request shares the large static weights.

The reason for this is, model weights do not need to be randomly accessed. So why store them in expensive RAM.

Cerebras and Qrok seem to be using a very different approach than NVIDIA to get orders of magnitudes speed ups. I'm trying to explore other alternative approaches.

Comment by wmf 4 days ago

I predict that the FPGA adds no value in this scenario. Just process inference on the CPU.

Comment by softwarewright 4 days ago

I am doing the initial work on many core dual CPUs. I already have systems with large system RAM and older GPUs with less VRAM, and large SAS HDs, fast SSDs running large open weight LLMs that do not really fit any GPU I own (they offload and/or run on CPU and system RAM, but slowly).

I have the MCUs and FPGAs (in a drawer) and I am retired, and this is my idea of fun.

I am trying to generalize an approach to use large MoE models (with possibly small quants) to run many agents in parallel without spending more on more or bigger GPUs.

I am also doing some edge ML (bird species recognition near the camera) using NPUs (in design phase, yet untested). I have an electronics lab, and I've emulated soft CPUs and built software that runs on FPGAs and in my emulators.

Instead of assuming my approach won't work or is too expensive, I choose to be optimistic. Also, failures are educational. I'm trying to gain more FPGA experience.

Comment by addag 4 days ago

I thought the same when I saw that the financial industry was hiring FPGA people for low-latency algorithms.

My understanding (as a non-FPGA expert) is that currently FPGA beats generic hardware (CPU,GPU) for "small size algorithm" (i.e that do not need GB of weights), while enabling a certain flexibility vs ASIC.

My guess is that you cannot bake all the weights into the circuit topology, so you are still bound by the memory transfer speed (to be double checked).

Comment by softwarewright 3 days ago

The value proposition I am pursuing is not faster speed, but increased throughput (and possibly reduced kWh).

I'm trying to understand the benefits of streaming expert weights through hardware that offloads the math and avoids storing all of the weights in RAM at once.

But that's not the only thing that can be streamed and offloaded. '

Finally, I'm trying to come up with approaches to reuse old hardware, old GPUs, old RAM instead of paying today's prices for GPU VRAM or unified memory. Even if I do not end up showing any particular FPGA benefit, I might be able to better run very large models on systems without GPUs or without unified RAM.

Comment by bschup 3 days ago

ML inference using FPGA's is an active area of research, https://arxiv.org/html/2512.01463v1 and have been successfully used in LHC/HEP detectors such as CMS and ATLAS.

Comment by softwarewright 15 hours ago

This IBM Research looks very promising: https://research.ibm.com/blog/northpole-llm-inference-result...

which I found after watching this video: https://www.youtube.com/watch?v=G4nEpY1LYzM - The AI Chip With No Ram (and still 22x Faster)

Comment by softwarewright 3 days ago

This looks extremely helpful, thanks!

Comment by b89kim 4 days ago

You could build small ML model like talos-v2. But larger model/LLM requires much more engineering cost than you expected. Optimizing HDL has too many control knob/param to solve by RL.

Comment by softwarewright 3 days ago

Understood, and I like a challenge. I've also solved some tough engineering problems in the past (before AI). With AI it seems (and I emphasize seems) that I can tackle much harder problems. That's why I am proceeding to measurable experiments and not just taking AI chats as accurate.

I have a homelab, electronics lab, and software dev experience (electronics is a hobby, and an early career before I pivoted to O/S development). Before the recent RAM/GPU price jumps I had invested in a lot of used ECC RAM and many older GPUs (and some new GPUs) on many older servers that I refurbished and upgraded. It is hard to justify the prices of new GPUs/RAM going forward. It is cheaper to upgrade my CPUs (and I have) to have more cores.

Yes, my older hardware is slow by today's standards, but it is at least affordable. Yes, old hardware is power inefficient, but I justify that by using solar panels. I cannot do anything about the current supply/demand problems, but I can perhaps help with reuse and upcycling. Maybe my work will help students and junior programmers learn on old/used hardware.

Comment by pugfugly 4 days ago

Train circuits, not weights; then use an FPGA

Comment by softwarewright 4 days ago

Can you suggest what circuits? examples?

Comment by mathisfun123 4 days ago

No one because FPGAs are 10-20x less dense than ASICs actually designed for purpose. The tooling is also complete shit. Signed someone that did (part of) their PhD on this topic.

Comment by softwarewright 4 days ago

But I can afford my already purchased drawer full of cheap FPGAs for experiments.

I guess I am too risk adverse to bet $10,000 on an ASIC run (having no experience doing that and no desire to go that route).

If I can demonstrate a proof-of-concept in a reproducible research way, others can then advance to the ASIC level.

This is just a hobbyist experiment looking for other hobbyists who can afford a cheap FPGA and have some free time and interest.