Securing Services with Rootless Containers

Posted by speckx 5 days ago

Comments

Comment by raesene9 8 hours ago

Rootless helps, but less now that it used to (pre-2026). There have been a lot of local privilege escalation vulnerabilities in the Linux kernel (dirtyfrag, fragnesia, CIFSwitch et al) and several of those can be repurposed as container breakouts.

As a result, if you're looking for good security isolation, I'd say a (Micro)VM is a better option. The other route is hardening down your container runtime with seccomp/AppArmor/SELinux but that can be a tricky game.

Comment by merpkz 5 hours ago

That is what I have been thinking too about recent linux vulnerabilities in context of container escape, but upon brief research I am not convinced it's all that straightforward. For example here https://github.com/Percivalll/Dirty-Frag-Kubernetes-PoC relies on sharing same container layers with other privileged workloads, which is quite a stretch to find in the wild and moreso it says that having a seccomp enabled breaks the exploit - "The default seccomp policy disables the unshare syscall." Other thing is that temporary remedy to lot of these exploits is to blacklist esp4, esp6, algif_aead modules, but how on earth are they going to be loaded in host kernel, which they are not by default, from unprivileged container in first place?

Comment by raesene9 5 hours ago

So one of the factors in this is that Kubernetes disables the default seccomp policy provided by the container runtime, by default (you can re-enable it ofc, but you have to know to do that).

As a result I reckon there's more vulnerable containers that you might expect.

Also depending on the environment there's things like dirtyclone https://github.com/raesene/vuln_pocs/tree/main/CVE-2026-4350... which can be triggered where the attacker can start new containers.

Comment by 3 hours ago

Comment by seemaze 19 hours ago

The '--userns=auto' argument is a useful isolation method in both rootless and rootful Podman containers. This allows rootful Podman to orchestrate privileged capabilities while running the container processes in an unprivileged namespace.

See the discussion here:

https://github.com/podman-container-tools/podman/discussions...

Comment by lugoues 17 hours ago

I've been doing this for a while now and it is by far much easier to work with and just as safe if configured properly. I think people just cargo cult thier way into wanting rootless containers for their long running services with no real look into if it's actually needed.

Rootless containers have their place, actual userspace workloads. Things like dev containers or normal users running containers but if one is creating users to run system level services then they are just adding extra steps imo.

Comment by samhclark 2 hours ago

Funny timing! I just finished migrating my NAS containers all to be rootless last weekend.

"Migrating" because secrets management was a bit of a pain. I ended up with pretty close to the same design as this article though. (I put an extra layer of complexity in mine, we'll see if I regret it)

Along with the other folks in this thread, I don't think we can stop at rootless in 2026. But I'm optimistic that dropping in libkrun as the runtime will be an easy next step, given its integration with podman (--runtime=krun).

If the author tackles that next, I'd be interested in reading about it

Comment by kayson 19 hours ago

I'd still rather use docker. I don't mind that the daemon runs as root because there are some things that you need root for anyways! Like binding to privileged ports or setting up networks (use `internal: true` and the daemon will automatically set up iptables rules that limit traffic).

I deploy docker compose files with ansible so everything comes with built in security defaults like rootless, dropped caps, no new privileges, etc. I wish more containers supported running read only (its usually pretty easy to add, just overlooked) and distroless (common for go apps, less so otherwise).

There was a pretty good comment on reddit a while back with a list of hardenings for compose files [1]

1. https://www.reddit.com/r/selfhosted/comments/1pr74r4/comment...

Comment by drnick1 17 hours ago

Rootless is definitely the way to go. You can forward ports manually on the host if you really need to use privileged ports. I generally expose my containers through a reverse proxy, running bare metal on the host, and that completely bypasses the privileged port issue.

Comment by merpkz 5 hours ago

That privileged port thing is not true anymore for Docker created containers, since it lowers that limit and unprivileged containers can listen on any port.

  docker run -ti --rm --user 1000:1000 --privileged=false alpine:latest
  ~ $ cat /proc/sys/net/ipv4/ip_unprivileged_port_start
  0
  ~ $ id
  uid=1000 gid=1000 groups=1000
  ~ $ nc -lvp 80
  listening on [::]:80 ...

Also the iptable rules Docker creates is for routing traffic to container with destination NAT, to actually limit traffic you have to do it yourself by inserting rules in DOCKER_USER chain.

Comment by nine_k 15 hours ago

Give your binary CAP_NET_BIND_SERVICE capability, and run it as a regular user. Almost no daemons need to run as root.

Comment by LelouBil 16 hours ago

I get the savings of Distroless, but when you have an issue it's a pain to debug.

Comment by capecodes 13 hours ago

copy the single busybox statically linked binary into a container, and you can create a bunch of symlinks for its tools that you want to use (this is how busybox container itself works)

makes it trivial to do the dhi/rootless thing but do ad-hoc debugging and then remove the debugging tools entirely, leaving no trace/bloat behind (just one file)

Comment by latchkey 18 hours ago

just curious why you'd bind to a priv port inside of a container.

Comment by zerof1l 18 hours ago

This whole privileged port thing is an outdated convention from the time when Linux ran on mainframes. Depending on your use case, it can be perfectly fine to lower it. I have set to `net.ipvX.ip_unprivileged_port_start = 80` on my server so that I can run rootless containers without extra privileges and have them bind to ports 80 and up.

Comment by bityard 17 hours ago

Nits: It's a Unix convention (not just Linux), and Unix traditionally ran on minicomputers (not mainframes). Mainframes had their own specialized OSes, although some could run Unix in a partition or virtual machine.

Comment by nine_k 15 hours ago

I think that "Linux ran on mainframes" was said in jest. Even though Linux runs on mainframes pretty well even now.

Comment by bubblebeard 10 hours ago

I've read that privileged ports was created as a trust model, so users connecting to a service using a privileged port would know that service was started by root on that server and therefore could be trusted. I always felt this was a bit wierd though, maybe I was missinformed, or maybe priviliged ports was already a thing and was just leveraged this way?

Comment by dboreham 17 hours ago

lol the before times when Linux ran on mainframes.

It does run on IBM 370 type mainframes, but that port was done years after it first ran on the 386, and isn't in any way relevant to low port numbers.

Comment by Narushia 17 hours ago

The .container units get the job done, but if you want to manage a stack of containers and are familiar with Kubernetes, or just like YAML, I recommend using the .kube units instead.[1]

[1]: https://docs.podman.io/en/stable/markdown/podman-kube.unit.5...

Comment by Lucasoato 16 hours ago

Is there any other reference regarding how safer we are by using rootless vs rootful containers?

Comment by firasd 18 hours ago

Interesting... yeah if you think of Docker as an easy way to setup environments that's one thing but if you are intending airtight isolation so processes inside Docker can't "escape" most conventional use cases / discourse haven't really focused on that I think

Comment by ranger_danger 19 hours ago

Personally I still think this is not enough, and we really need full generalized (not AI-only) microvm support built into docker/podman, like yesterday.

Currently it's difficult to even get a hold of a properly configured minimal kernel (or time-consuming to try to build one) and all the right command-line incantations to even start a one-off microvm using say, qemu, with all the proper storage/networking/etc. bits one needs for production environments. Plus you need to keep that kernel updated very regularly.

I know there's projects like smolvm that try to make this simpler, but I've had some major problems with those solutions as well, and I just feel like the big boys need to step up and support this directly by now.

Comment by teravor 17 hours ago

this has already been done.

runsc (gvisor)

Kata Containers (virtualization)

both drop in replacements for runc (default OCI runtime)

there are others too. one based on libkrun : https://github.com/containers/crun/blob/main/krun.1.md

and these are just runc replacements.

with some minor scripting you can roll a CLI solution using any of gvisor, cloud-hypervisor, firecracker, muvm (libkrun).

Comment by burakemir 18 hours ago

Maybe this here helps (I have not tried yet): https://github.com/virtkit-dev/virtkit

Comment by ranger_danger 13 hours ago

I wanted to try it but it seems it requires krun which does not have a readily-available package on ubuntu that I could find, and I didn't feel like compiling it.

Comment by binsquare 13 hours ago

I'm all ears, would love to improve smolvm into the right solution for this.

Comment by ranger_danger 11 hours ago

Well for starters, all the single binaries I made of various distros a month ago, simply don't work anymore for no obvious reason. Trying to execute the process silently exits after about 10 seconds with a 0 exit code and no output, shell or anything at all, and no errors. I have absolutely no idea what to do now.

Comment by dagss 17 hours ago

BETA: I have been following https://github.com/superradcompany/microsandbox a bit, very simple to get started with at least.

Comment by jorams 9 hours ago

Have you tried podman with the crun OCI runtime and `--annotation=run.oci.handler=krun`?

Comment by ranger_danger 2 minutes ago

The problem I ran into is that krun does not have a readily available package for ubuntu, and I'm too lazy to build it.

Comment by eyberg 19 hours ago

Containers and security are oxymorons. The flood of page cache cves (which can always be escalated/weaponized to an escape) from the other month is making deploying containers to prod untenable.

As for orchestration - a lot of folks think you need a completely new orchestration system for dealing with vms but we just simply re-use the existing infrastructure that already exists - the public clouds. Those companies have tens of thousands of engineers that are much better than the average engineer at this stuff, custom hardware, custom protocols and close to several decades of existing deployment.

I can build and ship a vm from my laptop/ci to prod on AWS/GCP in ~tens of second. Granted I come from the camp that thinks deploying full blown general purpose operating systems to prod is an increasingly incredibly risky practice.

Comment by dboreham 16 hours ago

Many such things have been built. Problem afaik is that the underlying hypervisor setup to support VMs isn't present, or isn't enabled by default, or is hard to figure out, on many machines. Plus you need nested virtualization when the thing you're using is a VM itself (also comes with possible pain). Docker runs because it has "ROAC" properties (Runs On Any Computer). That's not today true of VMs (of which micro-VMs, firecracker etc are a subset).

Comment by FooBarWidget 16 hours ago

Rootless containers have privilege escalation risks of their own. The problem is the unprivileged_userns_clone sysctl, which rootless containers rely on, has historically been a big source of local privilege escalations. The most recent attack I'm aware of is CIFSwitch.