Firecracker VM Escape: PwnEscape Part 2

Posted by SerCe 6 hours ago

Counter176Comment77OpenOriginal

Comments

Comment by scoodah 4 hours ago

I really wish they would have expanded on this paragraph in their article further:

> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.

If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.

Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.

Comment by tptacek 4 hours ago

That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.

Comment by rvz 1 hour ago

Thanks to frontier reasoning agents, Linux is completely riddled in ways you can discover 900+ LPE vulnerabilities. [0]

Another wave of vulnerabilities like this are yet to be discovered in Linux. We are just seeing how weak C and C++ based OSes and kernels are when put under hundreds of AI agents.

[0] https://openai.com/index/a-thousand-fewer-paths-to-root/

Comment by sscaryterry 2 hours ago

> This is an issue in the kernel networking stack.

Exactly, its not a firecracker issue. Using firecracker without a jailer is like setting your password to "password".

Comment by tclancy 1 hour ago

Hang on, what are you getting at?

Comment by sscaryterry 42 minutes ago

They ran it without a jailer, and exploited a bug in the linux socket code. Firecracker was just how they invoked/worked with /dev/kvm. It is not a firecracker bug. Its sensationalist nonsense.

Comment by tptacek 8 minutes ago

It's very much a Firecracker thing. KVM is part of the Firecracker stack, and getting code execution behind the jailer is still effectively game-over; it's just a somewhat higher degree-of-difficulty game-over. It's a drastically worse place than you would have been last year, when there wasn't a known plausible way to get even that far.

This is a really good bug and nobody should downplay it.

Comment by amluto 3 hours ago

Nope. The exploit compromises the host kernel. The researchers built an unconventional exploit chain in which they used the host kernel compromise to compromise the host firecracker userspace code, but I see no fundamental reason they couldn't have compromised something else in the host kernel.

Comment by tptacek 3 hours ago

Wait, this is also presumably true, come to think of it. I'm a little fuzzy because they document the memory disclosure primitive carefully but not the memory writes they also got from it --- but whatever those are, they're hitting kernel memory? But the exploit they're describing here gets code execution out of Firecracker itself, not the kernel.

Comment by monocasa 1 hour ago

Yeah, they're not saying a lot there. And then the proof of concept code is just being passed the physical addresses of the pages they talk about as command line args, and is just opening /dev/mem to do brain surgery on them.

Which like, isn't really a proof of concept of anything being described.

"That was always allowed".

Comment by tptacek 31 minutes ago

It's a super good bug! I just don't follow the exploitation details all the way. I know more than one cloud provider panic-bounced their fleet in response to it, and I think that was the right call.

Comment by vluft 2 hours ago

yup, the other firecracker escape recently (https://www.coinspect.com/blog/firecracker-escape/) popped the jailer via io_uring (tl;dr firecracker was filtering syscalls appropriately with seccomp, and was filtering the uring _they_ initiated, so attacker just got them to pop a new unfiltered uring, which is rather nice).

Comment by tptacek 2 hours ago

Just for what it's worth, that escape is much more situational than PwnEscape (there have been other escapes that depended on nested KVM, as well). Pretty much everyone seems to have been exposed with PwnEscape, because everyone using Firecracker is also using virtio-net.

Comment by a32patt23qfpatt 30 minutes ago

[dead]

Comment by lrvick 6 hours ago

I am more convinced by the day that the Nova Microhypervisor approach is correct.

Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.

https://hypervisor.org/

Comment by eddythompson80 5 hours ago

How is this different from Xen? It's the same approach as Xen or Hyper-V where the hypervisor is its own component, then you boot into a virtualized Dom0 or Root Partition (in Xen and Hyper-V terminology respectively) that can manage the hypervisor underneath it.

This is different from KVM where your hypervisor runs in your Dom0/Root Partition.

Comment by markasoftware 5 hours ago

Xen had a generic guest to host vulnerability disclosed in the last few days: https://codeant.ai/security-research/xenbleed-a-grant-copy-c...

Comment by eddythompson80 5 hours ago

Yeah, all security design guarantees are as good as the implementation at the end of the day. All the fancy hashing and encryption math in the world isn't gonna save you from an erroneous `"" == ""` check or an invalid reference check.

I was replying to OP saying it's the same "approach" as Xen, if your preference is "approach" based. It's fair though to point to Nova's smaller code base. It's becoming an important metric for me in general (how big is the code base). It's not a bullet proof thing, but having a smaller code base is a positive sign in 2026 for me.

Comment by markasoftware 4 hours ago

Novas core seems to be formally verified which rules out a vast amount of bug categories that can happen with xen, including this one

Comment by eddythompson80 4 hours ago

Interesting, I'll have to learn more about Nova.

Comment by lrvick 4 hours ago

See Udos most recent talk at 3mdeb Boot Mastery Conference.

He is an ex Intel CPU engineer and the depth of his thought into the security design had me questioning my life choices.

Comment by skinfaxi 3 hours ago

https://youtu.be/RbL65gVzhNk for anyone else looking for it

Comment by eddythompson80 3 hours ago

Thank you. I'll definitely check that out.

Comment by ThePowerOfFuet 4 hours ago

Now compare to Linux, which is the point raised. Nobody said alternatives like Xen are perfect... but my oh my, some are much better in this regard.

Comment by lrvick 5 hours ago

This is a much lighter weight (~100k) alternative to Xen focused on baremetal TPM anchored remote attestation, formal verification, and trusted computing applications.

Anyone considering Xen should probably also consider Nova depending on requirements.

Comment by josh-wrale 33 minutes ago

Xen is formally verified?

Comment by tptacek 7 minutes ago

No.

Comment by tptacek 5 hours ago

I don't know much of anything about Nova but since the late 1990s this has also been the pitch for L4; there are several mature L4's.

Comment by lrvick 5 hours ago

Nova is L4 inspired, and interchangable with L4 kernels in Genode.

Comment by tptacek 5 hours ago

Yep, very similar, but L4 is a little bit more general purpose (despite being microscopically tiny) and a little more battle-tested.

Comment by lrvick 5 hours ago

Is there a well tested L4 hypervisor solution with hardware accelerated virtualization and raw TPM support formally verified to a similar or greater level than Nova with similar remote attestation and security features?

SeL4 is the closest contender on my radar atm.

Comment by tptacek 5 hours ago

Most of the L4s have hypervisor situations. I don't know about TPM support. Generally the vibe I get from systems security people is a bit of eye-rolling about the formal verification of things like L4; the real win is just drastic attack surface reduction.

And then, of course, L4s live in millions and millions of shipped products, which is a reason to consider L4 instead of something novel, but I'm not saying you should or shouldn't.

Comment by lrvick 4 hours ago

I for one do not want any code in the TCB that is not formally verified, tiny, and easy to audit with every line of code justified.

SeL4 is the only open, mature, and formally verified L4 solution I am aware of that can run on modern servers, but IIRC given vCPUs are not first class objects and any low level CPU virtualization features are handled in user space, it comes at a real performance cost making it only suitable for single-core or non-performance-critical applications.

Nova by contrast was built to handle the expensive and trust critical TPM and vCPU handling directly for high performance remotely attestable multi-core applications without having to reach for something as heavy as Xen, though formal verification is admittedly not as complete as SeL4 yet AFAICT, though lots of recent progress here.

Those are pretty big tradeoffs, but if you or anyone else knows other viable xen alternatives this general shape with a tiny easily-auditable footprint, please chime in!

Comment by tptacek 4 hours ago

Like some of what I'm saying is just that if you tell a systems security person "we're using seL4 because of its formal verification", you're going to get eye rolls. Whatever else you want to say about it, the formally-verified configuration of seL4 doesn't actually do much, meaning whatever system you're building on top of it is going to be where all the security drama is anyways.

I'm certainly not telling you to use Xen. I wouldn't use Xen for anything untrusted.

Comment by lrvick 4 hours ago

> the formally-verified configuration of seL4 doesn't actually do much

As a systems security engineer myself I would disagree and know several others that would as well. All depends on your circles, applications, and threat model I suppose.

Xen, Hyper-V, and KVM have all had major security flaws in the form of logic violations, spacial memory corruptions, use-after-free, and pointer corruptions that allowed hypervisor trust boundary violations. I expect most if not all of those classes of bugs could have been prevented by formal verification.

Xen likely has a bad reputation with both of us because of a history of preventable flaws like these.

In my applications every single daemon gets its own sandbox, which is a VM whenever possible. Some with network, some without network, different filesystem and vsock access for maximum damage control and minimal trust between daemons. The hypervisor is the most important security boundary to me especially when isolating user-controlled workloads. The more confidence I can get that it is correct the better.

Comment by tptacek 4 hours ago

You're responding to a comment where I wrote that I would never use Xen with an argument not to use Xen. All I'm saying is that "formal verification", at least as it exists in current systems, is not why I wouldn't use Xen.

Comment by lrvick 3 hours ago

> an argument not to use Xen

Not my intent but perhaps I communicated poorly. It was an argument that the -reason- Xen has had so many bugs is because it was not built for correctness.

Formal verification should be a bare minimum for a TCB component to know many classes of bugs that have plagued the most popular hypervisors cannot happen.

We both clearly agree not to use Xen.

Anyway, I appreciate the engagement on this all the same.

Few even know alternatives to xen exist!

Comment by tptacek 3 hours ago

Yeah, all I'm saying is that I agree with you about Xen, but that I don't agree about the centrality of formal methods --- at least on current projects with current verifications, things could get different now that AI lets us be more ambitious with verification --- in determining attack surface.

Generally, the less your OS can do, apart from strictly hypervising guest operating systems, the safer you probably are. Hence why people do things like seL4.

Comment by lrvick 26 minutes ago

> the less your OS can do, apart from strictly hypervising guest operating systems, the safer you probably are

100% agreement here and I -would- take something tiny I can audit myself over something huge and formally verified if forced to choose. Thankfully I do not have to choose.

> AI lets us be more ambitious with verification

Exactly. For TCB code formal verification is a minimum, not sufficient, IMO. Too cheap these days to not have very complete test coverage, formal verification, risc0 execution proofs, seccomp tracing. For security critical code we should demand all of it now.

Comment by iberator 4 hours ago

[dead]

Comment by traceroute66 5 hours ago

> the Nova Microhypevisor approach is correct

To save people the time:

"This code is experimental and not feature complete. If it breaks, you get to keep both pieces."

Comment by lrvick 5 hours ago

The actual main open source repo has had that warning there for 14 years in spite of commercial forks that are deployed in production government and high risk use cases.

See his latest talk which represents a lot of impressive recent work: https://cfp.3mdeb.com/media/bsmconf/submissions/KW3SYT/resou...

I am working on a production OS using it as a foundation.

Comment by lwansbrough 5 hours ago

[flagged]

Comment by dexter0 5 hours ago

That phrase predates LLMs.

Comment by lrvick 5 hours ago

Considering the commit that added that text was 14 years ago, I am going to assume it was written by a human.

https://github.com/udosteinberg/NOVA/commit/9e35d89c0e7014f1...

Comment by zX41ZdbW 5 hours ago

I have a demo service that entirely relies on Firecracker VMs: ClickBench playground, https://benchmark.clickhouse.com/playground/

It hosts hundreds of database engines - many of them have vulnerabilities, and some of them, like Pandas, are basically open RCEs regardless.

We can PoC the Firecracker VM Escape on that service.

Comment by gavinray 4 hours ago

  > "run SQL against 110+ databases"
This is one of the most useful web tools I've ever seen.

I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.

Comment by markasoftware 5 hours ago

This is not a new vulnerability, it's just a description of how to exploit the PwnEscape vulnerability in practice on firecracker VMs (disclosed multiple weeks ago and discussed in some less upvoted posts on HN)

Comment by YetAnotherNick 5 hours ago

> already discussed at length on HN

Couldn't find it. Care to provide link.

Comment by tptacek 5 hours ago

The actual vulnerability got very little discussion. It's clever. Because Linux networking people are obsessed with checksum performance, virtio-net plumbs checksum offload metadata through the host/guest driver interface. Guests can thus deliver the host packets along with instructions on what to checksum. The vulnerability is a missing sanity check that allows checksum instructions to instruct the host to overwrite the header length field in the IPv4 header, rather than the checksum field.

Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.

Comment by ytxm 5 hours ago

Fixed a month ago upstream

https://github.com/torvalds/linux/commit/6785011f8b16abf40b1...

https://www.pwn.ai/blog/pwnescape

Comment by tptacek 5 hours ago

Additional color: per the commit, the bug was found by Paulos Yibelo, who runs PwnAI (the author of this piece), and worked on security at AWS for 4 years prior.

Comment by ytxm 4 hours ago

OP asked for a link

Comment by markasoftware 5 hours ago

Ah maybe not on HN, I forget where I learned of it. Either way, multiple weeks ago: https://news.ycombinator.com/item?id=50108845

Comment by denysvitali 5 hours ago

Dupe: https://news.ycombinator.com/item?id=50381397

(Doesn't HN usually deduplicate links? I'm confused)

Comment by tptacek 5 hours ago

Duplicates are generally accepted when there was no significant discussion on the original submission.

Comment by denysvitali 4 hours ago

I seem to remember there's a system in place to prevent duplicates (where the submission of a duplicate link results in the upvote of the original story), but maybe that's limited to the first 24h? Idk, HN works in mysterious ways

Comment by kvemkon 4 hours ago

This has negative impact on searching older discussions. Instead of 1 result for 1 URL (within few month) getting 2 or even more.

Comment by tptacek 4 hours ago

It's a settled enough part of HN that you'd do better emailing hn@ycombinator.com about it than you will litigating it in public. I don't care either way, I'm just explaining the behavior to you.

Comment by nwilkens 4 hours ago

I've been building Firehyve, which is a firecracker like microVM solution for illumos and FreeBSD.. https://github.com/TritonDataCenter/rshyve

Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.

Comment by tptacek 4 hours ago

As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.

Comment by cyphar 3 hours ago

I would've thought that you could actually have an incredibly strict seccomp policy for the jailer which would require you to find an LPE in something like open(2), which seems unlikely even in the "agent swarm" era.

Comment by tptacek 3 hours ago

That's basically what the Firecracker jailer does. Always my thoughts turn to the waitid(2) vulnerability; for a little while, you could pass kernel addresses through it and whack arbitrary kernel memory, and nobody was ever going to seccomp "waitid" out. But yes: it's a meaningful defense, if not one you'd want to bet your service on.

Comment by clhodapp 5 hours ago

I posted a similar query on the Replit sandbox engine article, but..

How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?

Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).

Comment by Betelbuddy 5 hours ago

https://aws.amazon.com/security/security-bulletins/2026-137-...

Comment by 5 hours ago

Comment by thisisauserid 5 hours ago

Firecracker VMs run on AWS. Odd that the article doesn't mention that.

Comment by whizzter 5 hours ago

Firecracker is open source and possibly others use it, AWS sells services with it though.

Comment by tptacek 5 hours ago

(A more complete way to say this is that Firecracker is the engine built to run AWS Lambda and AWS Fargate).

Comment by tcdent 5 hours ago

It has notable adoption in the agent sandbox space where this type of vulnerability is especially volatile.

Comment by thisisauserid 5 hours ago

> The target was Firecracker v1.16.1 on x86-64, with host kernel 7.0.0-1011-aws.

Comment by sscaryterry 5 hours ago

Firecracker v1.17.0 was released 10 September, so the version is ancient.

Comment by iberator 4 hours ago

Do you even know what ancient means? 2 months is ancient for you? Are you like 6?

Comment by sscaryterry 2 hours ago

Yes, 2 months is ancient if 0 days is the game.

Comment by pier25 4 hours ago

Fly uses it too afaik

Comment by tptacek 4 hours ago

Basically everyone in cloud-land uses it at this point.

Comment by wrongquery_dev 5 hours ago

[dead]

Comment by sscaryterry 5 hours ago

The article is light on details, it is LLM slop.

Comment by aghadge 6 hours ago

it's really informative. At least someone are break it.

Comment by sscaryterry 5 hours ago

This is pure LLM slop.

Comment by tptacek 5 hours ago

You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.