A whirlwind tour of systemd-nspawn containers (2025)

Posted by ibobev 1 day ago

Counter55Comment12OpenOriginal

Comments

Comment by ectoloph 1 day ago

I have been using systemd-nspawn recently through NixOS containers[1].

The combination feels quite powerful if you are in the ecosystem.

Though it is slightly ironic that I find Nix more approachable with the help of LLMs and then Nix has become one of the ways I manage the environments for the LLMs.

[1] https://wiki.nixos.org/wiki/NixOS_Containers

Comment by chasil 1 day ago

Eleven years ago, I wrote an introduction to nspawn from a busybox perspective.

https://www.linuxjournal.com/content/infinite-busybox-system...

I used that for a number of designs before my retirement.

The quality of my writing certainly did improve after that was finished.

Comment by gnabgib 1 day ago

(2025)

Comment by noopprod 1 day ago

I found a bug with nspawn that it can't bind mount while being in the mode which lets you delegate namespaces or whatever

Aka

If you have some kind of mount that matters like NVIDIA, you can't also create containers from containers.

So I just switched to LXC but I guess I should submit a slop PR as well. But yeah I guess Id just caution anyone who was going to use it lol

Comment by tredre3 1 day ago

Nspawn has a lot of issues with bind mounts in general. They've never been fixed because pottery doesn't see the point. See for example: https://github.com/systemd/systemd/issues/26897

Comment by PeterWhittaker 1 day ago

Your comment intrigued me so I read the issue, but I don't follow how his comments re a potential security concern and being able to overmount from inside the container lead to a conclusion of not seeing the point.

As a cybersecurity guy, my interpretation is that the requested change could introduce a vulnerability and that there is another potential path to achieve the same result, which should likely be explored first.

Any change that can compromise security needs significant justification, and should come with guards to allow it to be used in those cases where no other alternatives exist, without impacting other cases.

I've mostly used podman and am far from expert, and have long been intrigued by nspawn, so I really would like to understand the rationale.

Any other examples of nspawn mount limitations impacting operations without "workarounds"? Thanks!

Comment by notme43 21 hours ago

Passing /dev/dri/renderD128 or /dev/kfd to nspawn doesn't provide separation from the host like podman does. It's possible to run GPU workloads with nspawn, but this behavior makes it a bit trickier than simply passing --gpu or --device to podman.

Comment by PeterWhittaker 20 hours ago

From what I can tell, those options make the GPU available to podman, but without any form of host isolation/separation: the GPU is fully available without restrictions.

Since none of the existing namespaces treat GPUs directly (though perhaps this could be accomplished with cgroup?), this lack of isolation doesn’t surprise me.

While this may be useful for some workloads, the apparent lack of separation gives me pause from a security PoV.

One would hope that podman containers accessing GPUs are otherwise properly constrained to prevent their workloads from using GPU access to gain undesirable access to other hosts resources.

So I am left with my original question: what equivalency to podman, other than invocation convenience, is missing from nspawn? If none, it seems to me that having an awkward invocation in nspawn helps alert the user to the risks hidden by podman's convenience.

Comment by lifeisloving 1 day ago

Per the other blog post on docker. Docker Iptable bs has been annoying me for the last 48, hours. It actually makes no sense.

Comment by binsquare 1 day ago

What are you trying to do?

Docker shares the host kernel + networking stack, so container traffic goes through the host's iptables. So if you're running into some weirdness because of that, happy to help if you can share your setup details.

disclosure: dealt with container weirdness for last 5 years, nothing to sell.

Comment by akerl_ 17 hours ago

I’ve been using docker since the early days. One of the early choices I made and stuck with is to never let Docker touch iptables.

I make a bridge for Docker to use and configure iptables rules myself via parallel config management.

Comment by grono 1 day ago

You might want to try out nftables backend: https://docs.docker.com/engine/network/firewall-nftables

Comment by nottanjune 1 day ago

[flagged]

Comment by 1 day ago