Fast Kubernetes Workers on KVM with QCOW2 and cloud-init

🚀 Introduction

Kubernetes is happiest when nodes are cattle: when a worker misbehaves, you drain it, delete it, and let a fresh one take its place. That only works if a fresh node is cheap — seconds to provision, identical every time, and joined to the cluster without anyone touching it.

On bare KVM, the two tools that make this possible have been around for years: QCOW2 backing files for instant disk provisioning, and cloud-init for first-boot configuration. PiVirt builds its Kubernetes worker pipeline on exactly these. This post explains how they work, how to wire them together, and the guest settings that keep kubelet stable on a VM.

📋 Key Topics Covered

• Why copying full disk images doesn't scale
• QCOW2 backing files and overlays
• cloud-init with a nocloud seed ISO
• Joining a worker to the cluster on first boot
• Guest tuning for Kubernetes: CPU, machine type, memory ballooning, disk cache
• Treating workers as disposable

🐢 The Problem with Copying Images

The naive approach is to keep a "golden" 20 GB disk image and copy it for every new VM. That costs 20 GB of storage and 20 GB of I/O per node. On shared storage, provisioning ten nodes at once saturates the SAN for minutes and slows every other VM on it.

Most of each copy is identical: the same OS, the same packages, the same container runtime. Only a few hundred megabytes ever change per node.

🧅 QCOW2 Backing Files

QCOW2 supports backing files: a new image starts empty and points at a read-only base image. Reads of untouched blocks fall through to the base; writes go into the new image (the overlay). Creating an overlay takes milliseconds, whatever the base image's size.

bash
1# One-time: the base image, kept read-only in a shared image library
2qemu-img info /images/base/ubuntu-24.04-k8s-1.33.qcow2
3
4# Per node: an empty overlay pointing at the base (-F pins the backing format)
5qemu-img create -f qcow2 -F qcow2 \
6  -b /images/base/ubuntu-24.04-k8s-1.33.qcow2 \
7  /vms/worker-07/disk.qcow2 40G

The overlay can be larger than the base (40G above): the guest sees a 40 GB disk, and it can grow its root filesystem on first boot.

On PiVirt, the image library lives on clustered storage (iSCSI with OCFS2), so every node sees the same base images and an overlay created on one host can run on, or migrate to, any other.

Keep chains short. Each overlay layer adds a lookup on read misses. One base plus one overlay is ideal for workers. If you snapshot heavily, periodically flatten with qemu-img commit or qemu-img rebase.

🌱 cloud-init with a Nocloud Seed ISO

The overlay gives every node an identical disk. cloud-init makes each one unique on first boot: hostname, SSH keys, network settings, and the commands that join it to the cluster.

cloud-init looks for its configuration in several places. On KVM without a metadata service, the simplest is the nocloud data source: a small ISO with the volume label cidata, containing two files.

bash
1# meta-data: instance identity
2cat > meta-data <<'EOF'
3instance-id: worker-07
4local-hostname: worker-07
5EOF
6
7# user-data: the #cloud-config document shown in the next section
8
9# Build the seed ISO (volume label must be "cidata")
10genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data

Attach seed.iso as a CD-ROM and boot. cloud-init detects it, applies the configuration once, and records the instance-id so it doesn't run again on later boots.

PiVirt generates this seed ISO automatically for every VM from the VM's definition, with SSH keys, hostname, and any custom user-data. Nobody builds ISOs by hand. Building your own base images is covered in the Custom OS Images guide on pivirt.hu.

🔗 Joining the Cluster on First Boot

The base image already contains the container runtime, kubelet, and kubeadm, pinned to the cluster's version. So the user-data only needs to configure the node and run the join:

yaml
1#cloud-config
2hostname: worker-07
3ssh_authorized_keys:
4  - ssh-ed25519 AAAAC3Nza... ops@pilab
5
6growpart:
7  mode: auto
8  devices: ['/']
9
10write_files:
11  - path: /etc/kubeadm/join.yaml
12    permissions: '0600'
13    content: |
14      apiVersion: kubeadm.k8s.io/v1beta4
15      kind: JoinConfiguration
16      discovery:
17        bootstrapToken:
18          apiServerEndpoint: 10.0.10.5:6443
19          token: abcdef.0123456789abcdef
20          caCertHashes:
21            - sha256:<ca-cert-hash>
22      nodeRegistration:
23        kubeletExtraArgs:
24          - name: node-labels
25            value: pool=general
26
27runcmd:
28  - [systemctl, enable, --now, containerd]
29  - [kubeadm, join, --config, /etc/kubeadm/join.yaml]

With the image already on shared storage, most of the time from "create" to Ready is the guest's own boot plus the CNI pods starting, not disk provisioning.

⚙️ Guest Tuning for Kubernetes

A VM that's fine for a web server can be a poor Kubernetes node. These settings matter most.

🧠 CPU and Machine Type

• -cpu host (host passthrough) exposes the real CPU's instruction set, including AES-NI and AVX, to the guest. Container images built for modern CPUs run at full speed. The trade-off: live migration requires hosts with matching CPU models, so keep node pools homogeneous.
• Explicit topology (sockets, cores, threads) so kubelet's CPU manager and the guest scheduler see a realistic layout instead of many single-core sockets.
• The q35 machine type with OVMF (UEFI) gives a modern PCIe topology and UEFI boot, matching what current cloud images expect.
• A qemu-guest-agent channel so the host can freeze filesystems for consistent snapshots and request clean shutdowns.

🎈 Memory Ballooning: Use with Care

A virtio-balloon device lets the host reclaim memory from a guest by "inflating" a balloon inside it. That's useful for stats reporting and for consolidating idle VMs. But kubelet computes node capacity from the memory it sees, and schedules pods against it. Reclaiming memory from under a busy Kubernetes node creates memory pressure that kubelet didn't plan for, and it starts evicting pods.

For Kubernetes workers, size memory deliberately, don't overcommit it, and don't actively reclaim memory from nodes running pods. In PiVirt the balloon device is configurable per VM and can be disabled entirely.

💾 Disk Cache and Discard

bash
1# Disk settings suited to workers on shared storage
2-blockdev driver=file,filename=/vms/worker-07/disk.qcow2,node-name=f0,cache.direct=on,discard=unmap
3-blockdev driver=qcow2,file=f0,node-name=d0,discard=unmap
4-device virtio-blk-pci,drive=d0

• cache=none (direct I/O) bypasses the host page cache. On clustered or shared storage it's required for safe live migration, and it avoids double-caching container image layers in both host and guest.
• discard=unmap passes the guest's TRIM through to the QCOW2 overlay, so space freed by deleted containers and images is returned to the storage pool. Pair it with fstrim.timer inside the guest.

Both are per-disk settings in PiVirt's VM definition.

🗑️ Disposable by Design

The point of this pipeline is that you stop caring about individual nodes:

This is the same immutable-infrastructure model public clouds use for managed node pools, running on your own KVM hosts. For the cluster-side practices that make node replacement safe — PodDisruptionBudgets, graceful shutdown, and topology spread — see our Kubernetes in production guide.

🏁 Conclusion

Fast, disposable Kubernetes workers on KVM don't need exotic tooling. QCOW2 overlays make disk provisioning near-instant and storage-cheap; a nocloud seed ISO gives each node its identity; and a base image with the runtime preinstalled reduces first boot to kubeadm join. Add host CPU passthrough, careful memory sizing, direct I/O, and discard, and you have worker nodes that behave like cloud instances on your own hardware.

PiVirt packages this pipeline (base image library, automatic seed ISOs, clustered storage) as its standard way to provision VMs. If you're moving off VMware to get there, start with our VMware to KVM migration playbook.

---

❓ Frequently Asked Questions

Q: What is a QCOW2 backing file?

A: A read-only base image that a new QCOW2 image (the overlay) points to. The overlay starts empty; reads of unchanged blocks come from the base, and writes go into the overlay. This lets many VMs share one base image while storing only their own changes.

Q: How fast can a worker VM be provisioned this way?

A: Creating the overlay and seed ISO takes well under a second. Total time to a Ready node is dominated by the guest's boot, cloud-init, and the CNI pods starting, typically tens of seconds with a base image that already contains the container runtime and Kubernetes packages.

Q: Can I update the base image for existing VMs?

A: No. Modifying a base image corrupts every overlay built on it. Create a new versioned base image, provision new workers from it, and drain and replace the old ones.

Q: Should Kubernetes worker VMs use memory ballooning?

A: Not for actively reclaiming memory. Kubelet schedules pods against the memory it sees at startup; shrinking it underneath causes evictions. Size worker memory deliberately and avoid overcommitting it. A balloon device kept only for statistics reporting is harmless.

Q: What's the difference between nocloud and a metadata service?

A: The nocloud data source reads cloud-init configuration from an attached ISO (or disk) labelled cidata; it needs no network. A metadata service serves the same data over HTTP from a link-local address, as public clouds do. Nocloud is simpler on bare KVM and works before the guest's network is configured.

Q: Does -cpu host prevent live migration?

A: It requires the source and destination hosts to have matching CPU models, because the guest relies on the exact instruction set it booted with. Keep each node pool on identical hardware, or use a named CPU model as a common baseline if your hosts differ.