A thousand fewer paths to root

Posted by dangelosaurus 2 hours ago

Counter81Comment54OpenOriginal

Comments

Comment by Plasmoid 1 hour ago

How did we get to the point where there are a thousand privilege escalation exploits in the kernel?

Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?

Should we assume that all software is more compromised than the kernel?

Comment by pianopatrick 12 minutes ago

Because the large corporations that fund Linux development and use Linux value speed over security.

Speed requires a lot of code in the kernel, where that code can run fast.

There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.

For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.

OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)

You could also use like seL4, but that's slow too

So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.

Comment by mikestylz 1 hour ago

> Should we assume that all software is more compromised than the kernel?

Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.

The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.

I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.

Comment by asveikau 1 hour ago

For all of my time being aware of this industry, it has generally been understood that privilege escalation bugs are common. This has always been a thing. A lot of people say, commonly and for many years, that if your security depends on there not being privilege escalation bugs, you don't have security.

Comment by chrismarlow9 43 minutes ago

This. Privilege escalation locally is almost a given in most cases.

Comment by bombcar 1 hour ago

The dirty secret is multi-user died in the 90s and privilege escalation has basically been considered table-stakes since then.

Comment by hibikir 7 minutes ago

Back in the 90s, it's not as if any semi competent cs student couldn't slice your typical college server like swiss cheese. People thought null terminated strings were fine back then, and patching everything immediately was rare. So multi user was already dubious.

Comment by tptacek 45 minutes ago

I think this is basically true, but it didn't "die" in the sense that we gave up on it; it just never worked. Systems software was much worse in the 1990s than it today; you could breathe on something with a string of AAAA's and get a 0x41414141 register core dump.

Comment by Muromec 29 minutes ago

>Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?

There is nothing magical about kernel. Look at the code anywhere and it has bugs, lots of them.

Security bugs are just bugs. More often than not they are silently there and don't prevent you from using the software and you can't react and fix them because of that.

Give it time and they accumulate

Comment by NooneAtAll5 39 minutes ago

> How did we get to the point where there are a thousand privilege escalation exploits in the kernel?

it's been rule of thumb basically forever that there is 1 bug per ~10KLOC for any project

linux kernel is millions of lines, so thousands of bugs being hidden there is about on point

(tho obviously far smaller serious escalation ones, obviously, but you're exaggerating as well)

Comment by p-e-w 1 hour ago

How can it be surprising that a software made from 40+ million lines of memory-unsafe code has hundreds of critical bugs?

Comment by wahern 56 minutes ago

More than half of those lines are from drivers, drivers which wouldn't be relevant to a VM environment. Core kernel code for any particular architecture is less than 10 million, but some of that is filesystems or subsystems not in use or available to an unprivileged user on a typical server, so possibly maybe 5 million or less. And the article says they only targeted 2.8 million SLoC.

Comment by hoppp 1 hour ago

There are always more left.

Comment by ChickeNES 1 hour ago

And memory safety doesn't prevent logic errors (or using 4 as a private key for that matter)

Comment by 0xbadcafebee 1 hour ago

Funny how OpenBSD is "memory-unsafe" yet doesn't have these issues. Could it be that the programmer might have something to do with it?

Comment by icepush 59 minutes ago

As an OpenBSD lover and frequent user I will tell you it does not support but a significant fraction of the hardware Linux does, and has many features that are reduced in functionality or just gone compared to Linux.

Is it bulletproof? Definitely! But you need the genius of Albert Einstein as well as the patience of Mother Theresa to actually use it in production.

Comment by Muromec 25 minutes ago

Can't have a bug in a feature you never implemented and can't have another arising from unexpected interaction of two features you didn't implement.

It's the one security practice that actuuuualy works

Comment by wahern 48 minutes ago

For most server setups OpenBSD's hardware support is adequate, and this is especially true for VMs. It's laptops, peripherals, and bleeding edge, high-performance or specialized devices where driver support is a PITA. But that's not something I've encountered running my own private servers. It would be nice to have something like ZFS, but mostly that's a peace and convenience thing--would be nice to be able to stream backups somewhere rather than push around dumps and tarballs, though I have a script for that.

Comment by cyberax 59 minutes ago

Is anyone spending millions of dollars on checking OpenBSD?

But it also got its share of use-after-free and OOB access vulnerabilities this year: https://feedly.com/cve/vendors/openbsd

Comment by esseph 59 minutes ago

> Funny how OpenBSD is "memory-unsafe" yet doesn't have these issues.

First of all... Do you really believe that to be true?

Comment by AnimalMuppet 1 hour ago

I recall reading that every IBM mainframe OS release had 1000 bugs in it. (This was in the 1970s/80s.) Sorry I can't supply a source, after 4-5 decades. It was probably an article in Datamation magazine.

Now, I don't know how many of those bugs were privilege escalations, or could be leveraged to be. I'm not sure that people tried to exploit in the same way back then (or, if they did, if they said anything publicly about the results). Still, even if the majority of those bugs were not exploitable, OSes were much smaller then.

So that's my conclusion of how we got here: We've always been here.

Comment by esseph 58 minutes ago

The surrounding systems that connect to mainframe also expect and support those bugs. For 40+ yr now :)

Comment by antonvs 1 hour ago

> How did we get to the point where there are a thousand privilege escalation exploits in the kernel?

We developed AIs that can find them.

Comment by spartanatreyu 59 minutes ago

> How did we get to the point where there are a thousand privilege escalation exploits in the kernel?

> > We developed AIs that can find them.

We could already find them before AI.

We already knew they were there, and we already knew how to prevent them in the first place.

See previous discussion: https://news.ycombinator.com/item?id=49705794

Comment by esseph 58 minutes ago

> We could already find them before AI.

Not at this speed and cost.

Comment by spartanatreyu 7 minutes ago

> Not at this speed and cost.

I'd argue that the resources (i.e. time, money and effort) needed to create the tooling that would have prevented all of these issues would have been orders of magnitude less than the resources that have gone into "AI".

Comment by est31 1 hour ago

The kernel has the problem that basically every bug in it is also a security vulnerability. All software has bugs. Had Linux been written in Rust, and I strongly recommend a rewrite, it would still have bugs, and thus would still have vulnerabilities. But they'd be way less.

Comment by anonymous908213 39 minutes ago

> and I strongly recommend a rewrite,

Oh, you strongly recommend rewriting tens of millions of lines of code? Why don't you do it yourself? Just a prompt away, right?

Comment by jkelleyrtp 28 minutes ago

fwiw at work we've been looking at what the options are more memory-safe kernels out there. Not many options. If there were a "safe linux" maintained by like canonical or redhat or whatever we'd use it. For datacenter workloads like sandboxes and virtualization I think it'd do well.

Comment by Muromec 27 minutes ago

Kernel will eventually move to rust, but not the dramatic and fast way

Comment by shoobiedoo 11 minutes ago

Won't AI's ability to discover these bugs reduce the need to rewrite things in rust? If the major benefit to rewriting them is memory safety, I mean...

Comment by ventedmouse 25 minutes ago

“Claude rewrite this project, its 40 million LOC, make no mistake”

Comment by dangelosaurus 2 hours ago

This is my team’s work. I’m happy to try to answer questions or take questions back to the team.

And we're hiring! https://openai.com/careers/search/?q=cyber

Comment by BoppreH 1 hour ago

It's a bittersweet result, like finding asbestos in your house. I'd rather know than not, but I would much prefer not having it in the first place.

I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?

Comment by ky1ebot1337 1 hour ago

(I am the author the pipeline, just created an account to answer some questions :D) My goal is to work towards a vuln-free Internet. What I observed is that stronger models can find many more vulnerabilities that weaker models cannot. If compute permits (there are other projects to secure besides Linux), I plan to scan Linux again with every newer generation of cyber models.

Comment by konaraddi 1 hour ago

As models get more advanced and generate an increasing throughput of increasingly higher quality results, how are you determining when to trust its results and when human review is needed on what?

E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results

EDIT: nvm i shouldve kept reading, answered

> At this scale, manually investigating every report is impractical for our research team. We therefore focused on validating findings that a) were reachable by a low-privileged user, b) required no special capabilities, and c) produced measurable evidence of memory corruption.

Comment by ahsillyme 1 hour ago

Thanks. I'm not a programmer so I can't understand what is happening in the kernel by looking at the changelog, so I'll ask: Are the problems remediated structurally so that nothing like it can occur in the future, rather than just patching it up? Are there "lessons learned" that are being applied and documented here (like, "this problem looks like it's shape transports to other subsystems as well, let's fix them all in a disciplined way")?

Comment by esseph 54 minutes ago

I can't comment specifically on the Linux kernel piece, but I want to point out, broadly, that security has generally been an afterthought, and very little gets invested into it in terms of time and money - as a general rule.

Multiply that behavior times the number of commercial and mobile and b2b apps over 50+ years, and that's where we are.

There will likely be many more leaks, and hacks, and outages. And then in trying to be proactive to fix old systems, that will cause new proactive outages, and then eventually many things will be better.

But, it's going to take a lot of time and investment, and a lot of (majority) of currently unsupported devices will have to be complete replaced / scrapped / ewaste. On the scale of Nation-State initiatives.

This is across all operating systems, firmware, software and devices of all kinds, etc.

LLMs to security are like a bomb went off. We still have to wait for things to fall over before the dust will even clear enough to see what we have to work with.

Comment by emmelaich 50 minutes ago

I'd like to see a break down by age of code and part of kernel (drivers, networking, core ,etc ..)

Comment by basilikum 57 minutes ago

The high number is really impression to the point of making me skeptic. Are these distinct vulnerabilities or are a lot of the related in the way vulns often are?

The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?

Comment by ky1ebot1337 46 minutes ago

These are distinct vulnerabilities counted by the number of patches. The patch generation system performs variant analysis. So, each patch can potentially stop multiple exploits. The raw exploits number is higher than what we reported in the post. We don't want to cause the confusion why #patch != #exploit, so we used the smaller number there. These vulns are all kernel specific. Userspace software running as root was not in scope of the analysis.

Comment by farisght 1 hour ago

I wish these blog posts included statistics on the categories of deficiencies that cause these vulnerabilities, and proposed measures to avoid them

A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects

Comment by jethro_tell 1 hour ago

It sounds like they can’t go through them all enough to have that data yet.

Comment by esseph 12 minutes ago

Kernel is far too big to throw at a single LLM, just like a human. It is.. I think somebody said 40mil lines of code.

So you have to chunk it down to workable pieces over time, and then if you collect enough data of that process you can look at higher level stats.

You're also talking about hundreds of thousands (millions?) of individual contributors to the Linux kernel over 30+ yr. That process is likely to take years.

Same for every other existing piece of software, btw.

Comment by shalom1112 3 minutes ago

[dead]

Comment by IcyWindows 20 minutes ago

I think this is one area where Windows has an advantage.

ACLs, Appcontainers with capabilities, etc have been something in the OS for more than a decade.

Comment by jasonjayr 19 minutes ago

There are a lot of LPE's in Windows too. Remember that it started out as a single user operating system, it took a long time to get software on board to not always running as admin.

Comment by nas 1 hour ago

I've always considered local escalations to be possible in Linux (i.e. having user access almost certainly means you can get root). However, 948 vulnerabilities is kind of shocking. Hopefully we are getting to the end of the part where it "gets worse before it gets better".

Comment by getcrunk 1 hour ago

> For the near and medium term, having access to a low-privileged account on an operating system, that isn’t up-to-date on its latest available patches should be considered functionally equivalent to root access for a well-resourced threat actor.

*buries head in sand*

*cries*

Comment by kccqzy 42 minutes ago

LPEs are frequent enough that people no longer use them to sandbox potentially malicious software. They are only used to control access for well-behaving human users without any intent to subvert the system, not attackers.

Comment by rubinlinux 1 hour ago

This has been true for DECADES.

Comment by basilikum 1 hour ago

Regular LPEs are not novel. The frequency has just increased a lot.

Comment by zahlman 1 hour ago

> We've developed an autonomous vulnerability discovery, exploitation, and patching pipeline. In one run with our cyber model, it validated and generated patches for 948 local privilege escalation vulnerabilities in Linux at an API cost of $1.2 million.

Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.

Comment by 10 minutes ago

Comment by tamimio 1 hour ago

These are becoming more and more marketing materials than actual findings, because OAI probably hit the wall in making their models smarter (just like how google try to specialize their models in legal or whatever), but they need to remain relevant in the news. Sol 6.1 sucked so bad in reasoning that either it’s actually a bad model or users behind the scenes get degraded outcome while using all the resources for these marketing stunts. Stop, it’s embarrassing, make better models or be gone, at least we will get cheaper RAM to advance the development of open weights models.

Comment by hoppp 1 hour ago

Well at least they are doing public service while hacking the planet.