Host storage
Metal stores VM disks in ZFS and keeps records and artifacts under /var/lib/metal. The Atlas app owns image identity and object storage. Metal owns everything on the host. Host setup creates the ZFS pool before metald starts, and the zfs.pool setting names it.
How disks are built
Every VM disk is a copy-on-write clone of an image, so creating a disk is almost instant and costs no space until the VM writes.
A writable VM disk accepts discard, so a guest trim such as fstrim frees blocks in the clone.
| Dataset | Holds |
|---|---|
<pool>/images/<image-ref> | Base volume with a @ready snapshot, the source of VM disks. |
<pool>/vms/<vm-id> | One VM disk clone. The VM ID matches machines/<vm-id>. |
<pool>/staging/<snapshot-id> | Read-only image or sleep disk upload source. |
<pool>/warm/<key>@ready | Warm boot disk for one image and exact VM shape. |
<pool>/state | File system mounted at /var/lib/metal. Downloads, warm memory, and saved state use pool space, not the root disk. |
Files on the host
Metal keeps VM records, image files, and staging under /var/lib/metal. The host layout shows every file and dataset.
Prepare a VM disk
- Download the image and verify architecture and SHA-256 digests.
- Import a base ZFS volume.
- Create a copy-on-write clone of its ready snapshot.
- Grow the clone if the requested disk is larger.
Metal never shrinks a disk. Different content under an existing image reference conflicts.
Snapshots and warm artifacts
| Machine image snapshot | Warm artifact | |
|---|---|---|
| Contents | Disk and kernel. | Disk, guest memory, and Firecracker state. |
| Purpose | Publish an image through Atlas. | Skip boot for an exact image and VM shape. |
| Transfer | Signed multipart upload. | Stays on the host. |
A Machine image snapshot is not a VM rollback point. Metal has no restore or promote operation. Atlas creates a new VM from the published image.
Metal creates the snapshot ID shared with Atlas. A read-only staging clone keeps upload contents stable. Progress is saved, missing parts retry after restart, and upload activity extends staging life.
Limit disk IO
Set disk.throughput_mibps and disk.iops. Each covers reads and writes together. 0 means unlimited.
Metal applies Firecracker drive limiters and can update a live VM. It does not use cgroup IO limits: ZFS schedules its own IO, so block-device limits do not hold per dataset.
Firecracker sends each guest request as one 4 KiB IO, and the IOPS limit counts each request. So a VM at 3000 IOPS reads or writes about 12 MiB/s, even when the workload is sequential.
Host ZFS settings
Storage setup writes the ZFS module options to /etc/modprobe.d/zfs-metal.conf and applies them to the running module. It sets deeper vdev queues and limits the ARC to 16 GiB, because the guests and the host page cache already cache VM data. It also turns on autotrim and trims the pool once, because the Ubuntu trim job skips pools on md arrays.
To apply the settings on an existing host, run Install Metald again.
Keep and remove cached images
Host sync supplies image cache policy. Metal keeps each image that the policy caches.
Metal removes an image that the policy does not cache one hour after its last VM start. This includes an image whose cache_image setting was turned off. A new image that no VM started from ages from its download time.
When a cached image no longer requests a memory snapshot, Metal removes its unused warm artifacts.
A VM disk clone keeps its source image and warm disk until the VM is removed. Metal can build a new warm artifact while an old warm disk stays in use.
Images with the same kernel share one kernel file. Metal keeps a kernel while a stored image or a VM on the host uses it, including a VM that is migrating in, and removes an unused kernel one hour after its download. A VM with an existing disk boots from its kernel alone, so it still starts after its image is removed.
An existing VM clone can keep an image dataset in use after policy stops requesting it.
Limits and recovery
Check free storage before downloads, staging, migration, or warm-artifact work. If a ZFS capacity check fails, Metal does not invent a capacity value.
Metal reports available storage as the smaller of two values: the free pool space, and the usable pool size less the full size of every VM disk and incoming migration. A thin clone can grow to its full size, so placement never gives the same space to two VMs.
The <pool>/state dataset has a 1 GiB ZFS reservation. Metal can still write its records when VM data fills the pool.
Keep partial staging for retry. Before removing a dataset, confirm that no VM or staging clone still uses it.
Experimental
The file-backed ZFS setup script is for development only. Normal installation uses a pool device selected by host setup.
Source code and tests
- Storage specification maps image, disk, snapshot, and pool ownership.
- Storage construction creates the local stores.
- Image store checks artifacts and cache policy.
- VM disk store prepares and releases disks.
- Snapshot staging and upload prepare machine images.
- Experimental ZFS script labels its file-backed pool as experimental.