Show HN: Holos – QEMU/KVM with a compose-style YAML, GPUs and health checks

Posted by zeroecco 12 hours ago

Counter44Comment21OpenOriginal

I got tired of libvirt XML and Vagrant's Ruby/reload dance for single-host VM stacks, so I built a compose-style runtime directly on QEMU/KVM.

What's there: GPU passthrough as a first-class primitive (VFIO, OVMF, per-instance EFI vars), healthchecks that gate depends_on over SSH, socket-multicast L2 between VMs with no root and no bridge config, cloud-init wired through the YAML, Dockerfile support for provisioning.

What it's not: Kubernetes. No clustering, no live migration, no control plane. Single host. Prototype, but I'm running it on real hardware. Curious what breaks for people.

Comments

Comment by frabonacci 10 hours ago

it reminds me of https://github.com/dockur/windows with its compose-style YAML over QEMU/KVM. The difference i'm seeing is scope: dockur ships curated OS images (Windows/macOS), while holos looks more like a generic single-host VM runner. Is that a fair read? also curious any plans to support running unattended processes for OS installs?

Comment by zeroecco 9 hours ago

True and decent read. Holos is a generic runner underneath, and I'm not trying to compete on the curated-OS-image front; the assumption is you bring a cloud image and cloud-init does the rest. Unattended installs for Linux work today through cloud-init (that's most of what the YAML drives). Windows/macOS would need a different path.. autounattend.xml injection for Windows, custom payload for macOS; I haven't built that. If there's interest, a provisioner: type block that picks the right unattend mechanism based on OS isn't crazy, but it's not on the near-term list.

Comment by spidermonkey23 11 hours ago

I like it! is there a way to use both virt-manager and virsh with this or do we have to pick one. I'd be keen to migrate my VMs to this compose based system if it was a simple import job.

Comment by zeroecco 11 hours ago

Thanks! Honest answer: nope. Holos talks to QEMU directly and skips libvirt entirely — that's deliberate (no XML, no daemon, state lives in a single directory per project), but I like the idea of being able to import vm's. I will try to find some time to add that feature.

Comment by tiernano 12 hours ago

Interesting. Wonder could it be modified to work with proxmox?

Comment by gmuslera 11 hours ago

With Proxmox it could be as well Docker compose for LXC.

Comment by johnny22 11 hours ago

sure has been awhile since i thought about vagrant

Comment by zeroecco 11 hours ago

Fair — the YAML shape is similar. The differences that matter to me: GPU passthrough works without fighting the provider plugin ecosystem, healthchecks gate depends_on instead of you sleeping in a provisioner, and there's no Ruby, no vagrant reload, no box format.

Comment by yjftsjthsd-h 9 hours ago

Honestly, if you just sold it as vagrant but with all the weird stuff fixed and actually made to play nice with the rest of the stack that I'd actually want to use, that would be a pretty compelling package to me

Comment by tadfisher 8 hours ago

Stop generating comments, please.

Comment by zeroecco 7 hours ago

I did use an LLM to workshop my comms; I tend to go on and on so I put my my comments through a LLM to clean up my replies. the opinions and the project are mine, but the polish isn't. I'll knock it off for the rest of this thread.

Comment by la64710 3 hours ago

What is it using to create the vm without libvirt?

Comment by imiric 11 hours ago

Very cool, thanks for sharing.

I built something similar recently on top of Incus via Pulumi. I also wanted to avoid libvirt's mountain of XML, and Incus is essentially a lightweight and friendlier interface to QEMU, with some nice QoL features. I'm quite happy with it, though the manifest format is not as fleshed out as what you have here.

What's nice about Pulumi is that I can use the Incus Terraform provider from a number of languages saner than HCL. I went with Python, since I also wanted to expose a unified approach to provisioning, which Pyinfra handles well. This allows me to keep the manifest simple, while having the flexibility to expose any underlying resource. I think it's a solid approach, though I still want to polish it a bit before making a public release.

Comment by khimaros 10 hours ago

check out https://github.com/lnussbaum/incant

Comment by imiric 1 hour ago

Ah, that's neat as well.

I took a slightly different approach in that I don't want to use YAML as the authoritative source. Many projects abuse it, and end up creating a DSL on top of it with all sorts of hacks to achieve the flexibility of a programming language. Pulumi and Pyinfra already provide user-friendly primitives and idempotent(ish) APIs that work much better than YAML. I simply want to expose some (opinionated) building blocks to make them easy to use, and allow users to customize them and add their own as needed. E.g. I definitely don't want to write any shell scripts inside YAML. :)

BTW, Pulumi already supports YAML[1], which can be used with any provider. But to me it's too verbose and generic, and of course, it lacks the provisioning primitives.

[1]: https://www.pulumi.com/docs/iac/languages-sdks/yaml/

Comment by zeroecco 10 hours ago

Your path makes a ton of sense too! You get typed languages, state management, and the Incus team's work on the QEMU layer. The tradeoff I wasn't willing to make is the daemon + state store: Incus wants to own the VM lifecycle the way libvirtd does, and once you have that you're back to "two sources of truth" if you ever shell out. Holos is deliberately stateless on the host; Everything lives under one directory per project, rm -rf is a valid uninstall (though it will abandon the VMs if running). Different answer to the same frustration. Would genuinely like to see your thing when it's public.

Comment by d3Xt3r 10 hours ago

This is exactly what I dislike about incus. But what I do like about incus is how I can easily spin up and configure VMs directly using the CLI, without preparing a config first (I hate yaml).

So would be nice if holos could replicate that docker/incus CLI functionality, like say "holos run -d --name db ubuntu:noble bash -c blah".

Comment by zeroecco 10 hours ago

Docker does have both docker run and docker compose... Holos is compose-only right now. A holos run on top of the same VM machinery isn't a huge lift; mostly it's deciding what the sensible defaults are (image, cloud-init user, port, volume scope); Noted. it's not in 0.1 but I hear the ask.

Comment by zeroecco 7 hours ago

[dead]

Comment by imiric 1 hour ago

> Incus wants to own the VM lifecycle the way libvirtd does, and once you have that you're back to "two sources of truth" if you ever shell out.

That's true. But I didn't want to reinvent what Incus or any hypervisor abstraction does. I simply wanted to add some sugar on top that allows me to easily declare infra using small abstractions, and to tie in the provisioning aspect along the way. I still use Incus directly, and can benefit from their work, as you say. State is also managed by Pulumi, so really, there are 3 places for it to exist. There are some challenges with this, of course, but I think the tradeoff is worth it.

Good luck with your project, I'll be keeping an eye on it. I'll probably make a Show HN post when I release mine. Cheers!

Comment by ranger_danger 8 hours ago

My biggest issue with QEMU is trying to keep up with seemingly arbitrary changes to the command-line options format over time, and keeping up scripts that can launch an image across different versions for testing.

Originally if you wanted e.g. a SoundBlaster16 device you could use -device sb16. Then it changed to -soundhw sb16. And now it's -audio driver=none,model=sb16; this has been happening with several different classes of options over the years and I haven't found any good documentation of all the differences in one place. If anyone knows, I'd appreciate it.

Comment by khimaros 10 hours ago

another player in this space built on incus: https://github.com/lnussbaum/incant