Why PiVirt Drives QEMU Directly Instead of libvirt

🚀 Introduction

Almost every KVM management tool you've heard of sits on top of libvirt. It's the sensible default: libvirt is mature, it abstracts QEMU's enormous command line into XML, and it ships with virsh, storage pools, network management, and security confinement.

PiVirt doesn't use it. Our hypervisor daemon, pivirtd, starts QEMU processes itself and controls them over the QEMU Machine Protocol (QMP) — the same JSON interface libvirt uses internally.

This isn't a "libvirt is bad" post. It's an honest account of the trade-off: what you gain by removing a layer, and the list of things you sign up to do yourself once it's gone.

📋 Key Topics Covered

• What libvirt actually does for you
• Why a second source of truth was the deciding factor
• How QMP works: a JSON conversation over a Unix socket
• Surviving daemon restarts without touching running VMs
• Live migration capabilities, set explicitly
• What we had to build ourselves — and what you'd have to build too

🧱 What libvirt Gives You

libvirt is a daemon (libvirtd, or the newer modular virtqemud) that manages VMs described as domain XML. For QEMU/KVM it handles:

• Command-line generation: translating XML into a QEMU invocation, across many QEMU versions and their quirks
• Lifecycle and state: defining, starting, stopping and tracking domains, with its own persistent state on disk
• Security confinement: sVirt labelling with SELinux or AppArmor, so a compromised QEMU process can only touch its own disks
• Resource placement: cgroups, CPU pinning, NUMA, hugepages
• Storage pools and networks: a generic model over directories, LVM, iSCSI, NFS, and Linux bridges
• A stable API with bindings for many languages, and support for hypervisors beyond QEMU

That's a lot of hard-won engineering. If you're building a general-purpose tool, or you need to support Xen or other hypervisors, it's the right foundation.

🎯 Why We Removed the Layer

🔀 Two Sources of Truth

PiVirt is built around a declarative model: a control plane holds the desired state of every VM, node agents carry that intent to each host, and pivirtd makes it so. This is the same reconcile-loop design as Kubernetes; the PiVirt architecture docs describe each layer in detail.

libvirt is also a stateful system with its own idea of what a domain looks like. Put it in the middle and every VM has two definitions: ours and libvirt's domain XML. Every change has to be translated, applied, and then read back to confirm libvirt didn't normalise or reject part of it. Drift between those two definitions was the class of bug we most wanted to design out.

🎛️ One Hypervisor, Full Control

We target exactly one hypervisor: QEMU on KVM. libvirt's biggest strength, abstracting multiple hypervisors, is a cost for us. We want QEMU's newest block-layer and migration features the day we need them, without waiting for an XML mapping or working around one.

🔌 A Native API End to End

PiVirt's interfaces are ConnectRPC/gRPC services generated from Protobuf. With libvirt, every call would be gRPC → our translation → libvirt API → QMP. Without it, pivirtd translates gRPC calls directly into QMP commands, with one less daemon and one less failure mode on every host.

💬 How QMP Works

QMP is a JSON protocol over a socket. Each VM gets its own QMP socket. A session starts with a greeting from QEMU, then a mandatory capabilities handshake:

json
1← {"QMP": {"version": {"qemu": {"major": 9, "minor": 2, "micro": 0}}, "capabilities": []}}
2→ {"execute": "qmp_capabilities"}
3← {"return": {}}
4→ {"execute": "query-status"}
5← {"return": {"status": "running", "running": true}}

From there, everything is a command (execute) with a return or error, plus asynchronous events that QEMU pushes whenever something happens: the guest shut down, a device was removed, a block job finished.

A minimal QMP client in Go is genuinely small:

go
1type qmpMsg struct {
2	Execute   string          `json:"execute,omitempty"`
3	Arguments any             `json:"arguments,omitempty"`
4	Return    json.RawMessage `json:"return,omitempty"`
5	Error     *struct {
6		Class string `json:"class"`
7		Desc  string `json:"desc"`
8	} `json:"error,omitempty"`
9	Event string `json:"event,omitempty"`
10}
11
12func dial(socket string) (*json.Decoder, *json.Encoder, error) {
13	conn, err := net.Dial("unix", socket)
14	if err != nil {
15		return nil, nil, fmt.Errorf("dial qmp: %w", err)
16	}
17	dec, enc := json.NewDecoder(conn), json.NewEncoder(conn)
18
19	var greeting map[string]any
20	if err := dec.Decode(&greeting); err != nil { // read the {"QMP": ...} banner
21		return nil, nil, fmt.Errorf("qmp greeting: %w", err)
22	}
23	if err := enc.Encode(qmpMsg{Execute: "qmp_capabilities"}); err != nil {
24		return nil, nil, fmt.Errorf("qmp handshake: %w", err)
25	}
26	return dec, enc, nil
27}

The real work is everything around it. Events and command replies arrive interleaved on the same stream, so a production client needs a reader goroutine that routes replies to their callers and events to subscribers. And QEMU versions differ: pivirtd checks query-version and adjusts its behaviour to the QEMU in front of it.

🧰 Commands We Lean On

♻️ VMs Survive the Daemon

Each QEMU process is independent of pivirtd. Every VM lives in its own directory holding its configuration, disks, QMP socket, PID file, and logs. If pivirtd crashes or is upgraded, running VMs keep running.

When the daemon comes back, it rebuilds its view from disk: it reads PID files, checks which processes are still alive, reconnects to their QMP sockets, and corrects its in-memory state. A background checker repeats this every 30 seconds to catch VMs that died for reasons the daemon never saw, such as an OOM kill.

The same idea applies to networking. On startup, PiVirt can adopt existing Open vSwitch bridges and TAP devices instead of tearing them down and recreating them, so a daemon restart doesn't interrupt VM traffic.

🚚 Live Migration, Explicitly

Live migration is where direct QMP access pays off most. Before starting a migration, pivirtd sets QEMU's migration capabilities and parameters explicitly instead of relying on defaults:

• auto-converge — always on. A guest that dirties memory faster than the network can copy it never finishes migrating. Auto-converge throttles the guest's vCPUs progressively until the copy catches up.
• multifd — optional. Sends memory over multiple parallel streams, which matters on 10/25 GbE links where a single stream can't saturate the network.
• postcopy-ram — optional. Switches the VM to the target early and pulls remaining pages on demand. It guarantees completion for very busy guests, at the cost of the VM depending on both hosts until the copy finishes.
• Bandwidth and downtime limits. max-bandwidth keeps migration from starving production traffic; downtime-limit bounds the final pause when the VM switches hosts.
• Network announcements. After the switch, QEMU sends a configurable burst of self-announcement packets so switches relearn where the VM's MAC address lives. Without them, the VM is "migrated" but unreachable until ARP caches expire.

Daemon-to-daemon traffic between nodes runs over mutual TLS against a cluster-wide CA, so only nodes in the cluster can initiate or receive a migration. The operational side (prerequisites, commands and troubleshooting) is in the Live Migration guide on pivirt.hu.

⚖️ What You Take On Without libvirt

This is the part to read carefully if you're considering the same design. Removing libvirt doesn't remove its responsibilities; it moves them into your code.

For PiVirt, the trade was worth it: one hypervisor target, a declarative control plane that wants to be the only source of truth, and a team comfortable owning QEMU at this level. For many projects it isn't, and libvirt remains the right foundation.

🏁 Conclusion

libvirt solves real problems, and for multi-hypervisor tools or teams that don't want to own QEMU internals, it's the right choice. But it's a full stateful system, and putting one stateful system inside another creates drift.

By talking QMP directly, PiVirt keeps a single source of truth, gets QEMU's newest block and migration features without waiting for a translation layer, and keeps VMs running through daemon restarts and upgrades. The price is owning everything libvirt used to own, security confinement especially.

If you're evaluating KVM as a VMware replacement, start with our VMware to KVM migration playbook, or read more about PiVirt.

---

❓ Frequently Asked Questions

Q: Is QMP a stable interface?

A: QMP is QEMU's supported machine interface; libvirt itself uses it to control QEMU. Individual commands and arguments evolve between QEMU versions, and some carry an x- prefix while experimental, so a production client should check the QEMU version and handle differences.

Q: Do VMs stop if pivirtd crashes or is upgraded?

A: No. Each VM is an independent QEMU process. When pivirtd restarts, it rediscovers running VMs from their PID files, reconnects to their QMP sockets, and reconciles its state, so upgrades of the management daemon don't require VM downtime.

Q: Can I still use virsh with PiVirt?

A: No. PiVirt doesn't run libvirt, so virsh won't see its VMs. Management goes through PiVirt's own CLI tools (pivirtctl, pvcli) and its gRPC/ConnectRPC API.

Q: Does PiVirt support live migration without shared storage?

A: PiVirt's live migration is designed around shared storage (iSCSI with OCFS2), so only memory and device state move between hosts. Moving a running VM's disk is a separate operation using QEMU block jobs such as blockdev-mirror.

Q: What's the difference between pre-copy and post-copy migration?

A: Pre-copy (the default) copies memory while the VM runs on the source, then pauses briefly to send the last dirty pages. Post-copy switches the VM to the target early and fetches remaining pages on demand, which guarantees completion for busy guests but makes the VM depend on both hosts until the copy finishes.