GPU-unlock: FP/BF16 instruction issue rate on Nvidia 3000's cards

Posted by fragebogen 5 days ago

Comments

Comment by Zagitta 4 days ago

Hey this is my work, although the exploits etc aren't my ideas by any means. This is just a way to wrap the whole thing up in a much less annoying package than having to patch the nvidia driver. I originally got inspired by the CMP90/50 windows unlocks that run as standalone UEFI modules that chainloads into the OS.

Happy to answer questions!

Comment by aunty_helen 4 days ago

Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.

Comment by xenonite 4 days ago

great to hear, which quant + KV quant + KV size are you on? is it llama.cpp?

Comment by aunty_helen 4 days ago

Here's the writeup from the experiments that claude did: https://claude.ai/artifact/X6WJVtKdL1LrhYuCMqdXvG#b9a4d3e2-8...

Comment by SSLy 4 days ago

it's got a loginwall

Comment by razster 3 days ago

I just copied and pasted it, can't do markdown.

https://pastebin.com/FAdBbGUd Qwen3.8-27B on an RTX 3090: Inference Options and Benchmark Results

Comment by aunty_helen 2 days ago

Thank you!

Comment by jacquesm 4 days ago

Super well done, I wished all mods took your approach. Patched drivers are a nuisance, especially if you have more than one kind of card. Thank you.

edit: now do the 4090 ;)

Comment by Zagitta 4 days ago

Yeah I hate the patched drivers too, I have a CMP170 as well I want to see if I can find a way to get fully working the same way without or minimal driver patching.

4000s is WIP, forks has already unlocked it but I'm doing some refactoring to better support different families and after that I'll port it over :-)

Comment by jacquesm 4 days ago

Can't wait, I have two 48G modded ada 4090's and 4 stock Blackwell 4000's (though I doubt those will be as easy). The 4090's alone would be an amazing thing to have boosted, they're my main workhorses and effectively any improvement there will immediately make my days more productive. If I can help out with testing, my email is in my profile.

Comment by Zagitta 4 days ago

Blackwell and hopper have a whole new boot process so those are unlikely to fall over anytime soon from what I've seen.

If you can't wait for 4090 someone has a fork with a seemingly working unlock :D

Comment by jacquesm 4 days ago

Yes, but it's pinned to a driver version that I unfortunately can not use. That's precisely why I love your solution, it is the best way to do this.

I'm running 8 GPUs attached to a single CPU and that driver is a pretty essential part of getting this to work with P2P between as many cards as possible. The modded 4090 is a problem because the BAR1 it advertises (32G) is smaller than the VRAM so not all software is able to use it (you need two memory segments rather than a single 48G one to make this work). That driver is part of the solution for this problem.

Comment by Tiberium 4 days ago

FWIW the gpu unlock ada repo doesn't pin to a driver version, Codex for me just compiled it, and it works with my 615.xx driver even though the fork is for 610.x. I think the repo just uses the firmware from that version. My GPU is 4070ti super.

Comment by jacquesm 4 days ago

Ok, trying it. I'm using a pretty heavily modified 590.48.01 to be able to use those 48G 4090s, they work but they do require a bit of handholding. The main problem is the BAR1 just isn't large enough, it's capped at 32G.

Heh, got it working, thank you so much!!

Overall 30% reduction in wall clock time, that's pretty neat (and I'm still not running this at full power but 450 W is a bit much for that puny connector).

Comment by SlavikCA 4 days ago

I have two of modded RTX 4090 with 48GB VRAM. But I'm using default drivers. I'm using models, which uses almost full 48GB.

Now i'm curious - am I missing some settings? Do I need to check any of these BAR things?

Comment by jacquesm 4 days ago

You're probably not using peer-to-peer then. Try the p2pbandwidthtest that comes with the NV development package to see if you are or not.

Comment by Zagitta 4 days ago

If they are forked from my repo whatever driver version they pin doesn't matter, the pin is just clanker stuff really. What it's used for is just extracting the right booter firmware and loading it, but the whole booter part is reset before handing over to nvidia driver so the versions dont have to match at all.

Comment by jacquesm 4 days ago

Apparently they were, it compiles and installs but I don't see any improvement just yet. I'll keep plugging away at it.

Comment by marcosscriven 4 days ago

https://github.com/JustAResearcher/gpu-unlock-ada

Comment by jacquesm 4 days ago

That's a fork of GP's work.

Comment by marcosscriven 4 days ago

But it contains the 4090 variant.

Comment by modeless 4 days ago

Is this relevant for 4090? It has FP8 which is already double the rate of FP16. Can this hack double the rate for FP8 too?

Comment by modeless 2 days ago

Answering my own question, FP8 with BF16 accumulate is double rate already, but FP8 with FP32 accumulate is half rate, and this hack doubles it to match BF16 accumulate.

Comment by mietek 4 days ago

What about the 5090?

Comment by Zagitta 4 days ago

Hopper and blackwell have a very different security chain so those are out of scope for this approach at least.

But I would suspect China has figured something out given the reports of them buying 5090s in droves and seemingly modifying them.

Comment by jacquesm 4 days ago

I've looked a lot at that and it seems to be mostly related to packaging rather than performance, as far as I know there are no 'improved' 5090's for sale in any kind of volume, and the ones that they made that I know of (a 64 and a 96 G version) never made it to production for some reason. I'd love for that to happen though.

Comment by Zagitta 4 days ago

Maybe they got hold of the vbios for those higher vram versions similar to the 48gb 4090s?

Comment by jacquesm 4 days ago

There is a lot of speculation about this but nobody seems to really have hard insight. As long as the hardware supports it it should in theory be possible to unlock it but like with those 48G 4090's that can lead to weird side effects. For instance, you would never know that these cards are modded until you start using p2p and then suddenly it all falls apart. The reason is that for p2p to work they map the memory space of one card into the memory space of another. But if the BAR1 is < than the total VRAM that trick will work only until you run past the TOP of that block and then you'll have a crash.

I suspect that modding a 5090 in such a way that you can sell it is a lot harder than doing the same with a 4090.

Comment by az226 4 days ago

48gb 4090 uses stock vbios

Comment by marcosscriven 4 days ago

https://github.com/JustAResearcher/gpu-unlock-ada

Comment by bilalq 3 days ago

I have a 3090 founders edition sitting on a shelf. The FE cards had VRM cooling issues, IIRC. Would this unlock have any uplift even if I undervolt and don't increase the power draw limits much? The use-case I'm thinking is to run a model to power a local only voice assistant for my family. I'm not really familiar with what instructions are used most heavily by LLMs, but is my read correct that this would be expected to help with TTFT? I'm curious if there's anything about this that helps in particular with audio processing models.

And maybe I'm overthinking the cooling issue. It's not like I'm hammering it 24/7. Probably fine to burst high on power draw on use.

Btw, I always love moments like this where library or project owners stumble on their work and chime in on the comments. I really appreciate your work here. It's given me motivation to actually do something with parts that have been sitting on a shelf.

Comment by oceanplexian 3 days ago

You can trivially fix most of the VRM cooling issues btw by pulling the heat sink off, cleaning thoroughly and re applying fresh thermal pads. At this point almost all of 3090s are degraded in some way since those pads have a lifespan. The good news is it has nothing to do with the GPU itself.

Comment by bilalq 3 days ago

I know it's probably not that difficult, but I always feel anxious doing things like that with expensive components. The thought of being slightly misaligned and causing issues gives me pause. I guess now that it's out of warranty, I have a bit less to lose.

Comment by crspatel 1 day ago

really cool! would love to talk more if possible, sent a donation through ur github but if u can send me a dm on @chrs3775 on discord that would be appreciated

Comment by verdverm 4 days ago

Any idea if something like this is happening on GB10 (DGX Spark) and needs similar treatment?

edit: seems you have provided some clarity that even if, unlikely soon unless China has and does the world a favor

Comment by jacquesm 4 days ago

DGX Spark is going to be very hard.

Comment by userbinator 3 days ago

Did you use AI tools to do it? The README reads like Claudish. It's amusing to see Nvidia-powered AI used to unlock their GPUs.

Comment by RachelF 4 days ago

A semi related: a patch for those with multiple cards. You can unlock around 20% increased card to card speed on consumer GPUs using this library:

https://github.com/aikitoria/open-gpu-kernel-modules

The hardware supports it, but then Nvidia disabled it if the driver detects cheaper cards. This bypasses the NVidia slowdown.

Comment by jacquesm 4 days ago

Indeed, I already posted it earlier but this definitely deserves a top level comment.

Comment by jacquesm 4 days ago

This one is real, but there are some caveats:

- you will need to run your card hotter than you might be comfortable with

- the real pay-off (on some cards, apparently, see downthread) requires increasing the maximum power to 450W

- This is well above what the stock cooler was designed for so a bit more cooling would help

What's nice is that you can stack this mod on top of (or rather, below of, this one loads first) the p2p mod that has been out for a while, they don't seem to interfere with each other.

Comment by Zagitta 4 days ago

It's quite nuanced. In a nutshell the unlock just flips the 2 bits that halves how often the core can issue the specific instructions Nvidia locks to full rate on the professional and datacenter cards.

Obviously doing more compute requires more power but what it does is shift the card down on the frequency curve to keep the wattage in check. So you're right that at stock power limits it hits a cap where the full issuing rate isn't reached but it depends heavily on the exact workload how much juice is left.

The frequency curve also takes heat into account so in general you end up with a nice perf boost for a slightly hotter card.

I've been running this on my 2 3090s for a month now with no issues but they are also water cooled :-)

Comment by jacquesm 4 days ago

Ah, that's a twist I didn't see coming, all my GPUs are aircooled. I can see how with watercooling you'd be able to get more performance out of the GPU before hitting the limits.

Comment by BoorishBears 4 days ago

There 3090 is from the era of overclocking, anything but the reference design should be fine even on air cooling (especially if you have a 3 connector model)

My EVGA came with stock firmware that started at a 420W and allowed up 500W, of course 500W did nothing for gaming

Comment by Tiberium 4 days ago

> - you will need to run your card hotter than you might be comfortable with

Can you explain this one? On my GPU I didn't have to change the power limit. Obviously the card might spend more time at max power usage, but it's still within the limit.

Comment by jacquesm 4 days ago

If you increase the power limit on the card you will be able to get much more mileage out of this modification. At 450W the difference is much, much larger but that is not something I'm comfortable with doing for longer runs. Note that the stock settings are already pretty extreme for that cooler, and that ambient temp and free airflow are going to be impacting the resulting GPU temp in a very direct way. I have my 3090's in an external box (with its own separate PCIe backplane) so they can talk p2p and have 'fresh air' rather than in the enclosure.

Comment by Tiberium 4 days ago

Yes, but you can get up to a 40-50% increase in some workloads even with the same power limit, and some cards have it locked anyway. I just felt like it was a litle misleading for you to mention the power in your original post, since the unlock works regardless of that.

Comment by jacquesm 4 days ago

Interesting, I have 7 3090's here of different plumage and they all require a power increase to get > 25% gain with this modification, would you mind sharing the type of card you have and how it is cooled?

Obviously, if there are 3090's out there that respond differently then I will be happy to retract that.

Comment by unsnap_biceps 4 days ago

Is your external box something you put together or is it a product you purchased?

Comment by jacquesm 4 days ago

I put it together myself. It was a lot of work and I learned a lot of nasty lessons along the way. The short version of the recipe:

- get the largest PSU you can find (1500W at a minimum, more is better)

- get the ADT PLX Quad PCIe backplane. Run 4 thin (single slot) GPUs inside your main enclosure, stick the ADT interface board in between two of the thin boards (don't stack them close to each other or they won't be able to breathe).

- ensure you have as much or more RAM as you do VRAM

- Ensure your CPU has enough PCIe lanes to run at least five boards in x16 (four for the GPUs in the main enclosure, another for the PLX PCIe switch)

- In the BIOS set BAR1 to 'resizable' and > 4G decoding should be enabled.

- ditch the cables that come with the ADT board, they may work but there are much better ones out there and they're not expensive, this will save you days of troubleshooting

- Build your own enclosure rather than trying to adapt a standard one, it won't work anyway (the board pitch is wrong, as is the placement of the mounting holes).

- Put another four double or triple boards on the backplane.

- Add another 3x 140 mm fan to the front of the case so you get good airflow without the feeling of being next to a Harrier at take-off

- I used 3D printed parts to mount the PSU and the board as well as the fans (see below).

- Make sure the board is properly grounded to the enclosure

- If you're running 3090's or other cards that are P2P capable that were hobbled by NV then look at this https://github.com/aikitoria/open-gpu-kernel-modules driver module.

- Power on the ADT board, wait three seconds then power on the main machine. Enjoy your VRAM :)

If you have more questions please email me and I'll be happy to help out.

Comment by augment_me 4 days ago

These number claim 240W locked power draw on 4070:

https://github.com/JustAResearcher/gpu-unlock-ada/releases/t...

Comment by jacquesm 4 days ago

This is about the 30' series in principle, I don't have a 4070 here to test with (I do have a couple of modded 4090's) but I suspect that at 240W you are not getting the maximum out of that card. At the same time: that is probably pretty close to optimal in terms of tokens/W.

Comment by Tiberium 5 days ago

I've tested myself and the results are legit, this is incredible, even if a little risky (it relies on exploiting firmware): https://github.com/Zagitta/gpu-unlock/blob/main/docs/How%20A...

> Unfortunately for Nvidia the booter is vulnerable to a classic buffer overflow that writes straight into the stack which gives us the ability to execute code via Return Oriented Programming (ROP).

Repo for RTX 4090 and 4070 Ti Super: https://github.com/JustAResearcher/gpu-unlock-ada

Comment by qingcharles 4 days ago

2080 Ti also getting some love:

https://github.com/KohakuBlueleaf/KohakUnlock

Comment by jacquesm 4 days ago

Those work but don't play nice with other such mods, in contrast to TFA. They require a pinned version of the driver.

Comment by wewewedxfgdf 4 days ago

There is also the unlock for Nvidia encoding which "removes restriction on maximum number of simultaneous NVENC video encoding sessions"

https://github.com/keylase/nvidia-patch

So yeah, buy an Nvidia card then download and run all the software needed to make it work to its full potential so Jensen can buy his 500 millionth leather jacket.

Comment by RandyOrion 4 days ago

As a person who uses consumer level hardware to do neural network research, this unlock is truly INVALUABLE.

This unlock is super useful as it improves the GPU performance on computation-bounded neural network workloads like LLM prefilling, image/video generation, and general neural network training, in a way of disabling deliberate software segmentation on the same hardware imposed by Nvidia.

I'm on rtx 50 series, and no free performance uplifting for now. Waiting for more reverse engineering efforts to reveal whether the unlock is possible for 50 series.

Comment by Zagitta 4 days ago

I wouldn't hold my breath for 50 series, hopper & blackwell has a different boot architecture with more security layers

Comment by RandyOrion 4 days ago

I'm aware that long-lasting reverse engineering efforts on nvidia gpus, like cmp gpus, will lead to something impactful. I just didn't see it clearly until your project came out. Good job on that.

For now, I think reverse engineering efforts on 50 series gpus are just beginning. There are projects like reading sensor temperatures directly from the gpu itself instead of using deliberately DOWNGRADED software like nvidia-smi. However, I don't really see these efforts spreading, and people behind them really needs more recognition. Props to these people, too.

Comment by codedokode 3 days ago

What a garbage company, artificially limiting the performance.

This is why "right a reprogram" should exist. Anyone should be able to reverse-engineer and reprogram anything that has a CPU.

Comment by Havoc 3 days ago

>What a garbage company, artificially limiting the performance.

That's just modern corporate capitalism.

Manufacturing economics favour making everything the same for scale, while sale economics favour market segmentation.

Comment by Ey7NFZ3P0nzAe 4 days ago

Also exists on turing apparently: https://github.com/nlzy/turing-unlock

Comment by chromaton 4 days ago

It worked for me, and I'm getting about 6% faster training on nanochat with my 3060 card. Thanks for this!

However, it required a few extra steps, as the repo is set up for Linux kernel 6.x and I'm on 5.15. Claude Code was able to get me past the hurdles, including code modifications and startup scripts.

Right now, I'm clearly power limited at 170 watts and I have no desire to brick my card or set it on fire.

Comment by blagui 4 days ago

The real gains are far less than the advertised. Tests ton TFTOPS show 25% gains at 200Watt cap. While mostly prefil benefit from that less decoding which make the real gains 7-8%. Using at full power:

This is a benchmark test with Qwen 27B Int4 with 4x3090 and using full batching on VLLM to squeeze every bit of the GPU's.

Comment by triwats 4 days ago

Here are the claimed numbers for reference:

[1] https://flopper.io/gpu/nvidia-geforce-rtx-3090-24gb [2] https://flopper.io/gpu/nvidia-geforce-rtx-3090-ti-24gb

Comment by londons_explore 4 days ago

The next version is sure to have signed firmware which sets these configurations to prevent this sort of thing.

Comment by Zagitta 4 days ago

These firmwares are already fully encrypted but nvidia has been publishing encrypted debug images that turns out could be decrypted with one of their public test keys. And those debug firmwares turned out to be close enough to the production ones that they could be used to construct exploits from.

In this case it's a DMA buffer overflow that writes to the stack which in turn can be exploited with Return Oriented Programming. So no unsigned code is ever executed but we still have code execution control.

Comment by pixl97 4 days ago

Honestly if Nvidia is going to be short on cards for years there is no reason to keep this locked. People are going to buy what they can get and this in no way is going to influence sales at this point by making it official.

Comment by xrd 4 days ago

So if I'm reading this chart correctly, my 3090 card can get a 65-80% boost in speed generally? Or, is this only for certain models and quantizations?

Comment by Sharlin 4 days ago

25%-80%.

Certain (mostly AI) workloads. Depends on how hot you're comfortable running your card. The real-world improvement is considerably less due to other things being bottlenecks.

Comment by Scene_Cast2 4 days ago

Really interesting.

On the other hand, this only applies to a limited set of workloads (training is often memory bandwidth bound for example).

I also wonder how this affects power consumption.

Comment by kingstnap 4 days ago

Training is compute bound.

If its memory bandwidth bound you need to raise your batch sizes to leverage more parallelism.

Comment by Zagitta 4 days ago

It's also useful for prefill in inference, VLLM sees up to 30% boost depending on the exact kernels and prefill size

Comment by whatever1 4 days ago

Can we unlock FP64 so we can do serious work on second order methods? Even at enterprise grade the last good fp64 gpu was the H100. B100 focused on lower fp due to llms.

Comment by jasomill 4 days ago

I don't believe so. As I understand it, the only Ampere GPU with a full complement of FP64-capable cores is the GA100 used in the A-series server cards (which, as you suggest, still underperforms the H-series for FP64).

Comment by mkoryak 4 days ago

Where's the 5090 version?

Comment by superkuh 4 days ago

Uh, no. This headline is wrong. It's FP32 unlocked from being halved, not 16 bit ops. And to be clear, over the last week I've looked many places and I've not seen any reports of this unlock being applied to any 3xxx card except the 3090/3090ti series. Even if it might apply more generally in theory.

Comment by Zagitta 4 days ago

It's the FP/BF16 FMA to FP32 instructions.

And also it does work on most 3000 series cards as they all run the same GA102 booter that's vulnerable :-) There are some reports of cards with weird vbios layouts having issues but i'll try to get that fixed

Comment by augment_me 4 days ago

The repo has fp16 numbers.

Also this one for 4070s shows that its for fp8 as well:

https://github.com/JustAResearcher/gpu-unlock-ada/releases/t...

Comment by Tuna-Fish 4 days ago

it's the widening (for example: fp16 * fp16 = fp32) instructions, which are useful on some loads.

Comment by augment_me 4 days ago

I assumed the comment above is referring to operand dtype and not accumulator dtype since they mentioned ops. In general even for nvfp4 you use fp32 accumulators because you are usually reducing over large Ks which mean large values.

Comment by 4 days ago

Comment by Tiberium 4 days ago

The repo has a lot of reports of people applying it to other 3xxx card, see https://github.com/Zagitta/gpu-unlock/issues