Sleepy VMs
A sleepy VM is one that Metal saves and stops after an idle period. Metal restores it when new traffic arrives. While it sleeps, it uses no CPU or memory on the host, so a host can hold many more quiet VMs. It suits sites and VMs that are idle most of the day.
A VM is sleepy when sleep_after_idle_seconds is greater than 0. Set it when you create the VM or with a resize. 0 turns sleeping off.
How it sleeps and wakes
An explicit stop ends both states. Setting the timeout to 0 wakes a sleeping VM and turns sleeping off.
Going to sleep:
- Metal watches traffic sent from the host to the VM.
- Every 5 seconds, reconciliation checks running sleepy VMs with no pending change or error.
- After the idle timeout, Metal saves memory and device state under
machines/<id>/saved-state/and stops Firecracker. - Metal checks traffic again. If a packet arrived during the save, it restores the VM.
Waking up. An eBPF hook on the VM's tap0 sees the next packet and tells Metal. Metal restores the saved state, and the VM carries on from where it stopped: open files and processes are intact.
Only real traffic counts: unicast IPv4 or IPv6 packets from the host to the VM, except ICMP. Traffic the VM sends, ICMP such as ping, ARP, and IPv6 multicast chatter do not keep it awake or wake it.
What the state looks like
| Record | While sleeping |
|---|---|
| Desired state | running. Nobody asked the VM to stop. |
| Observed state | stopped. Firecracker is not running. |
| Sleep state | sleeping. See the values below. |
| Atlas list | Shows the last reported state, stopped. |
Metal reports the sleep state as observed.sleep_state. Atlas shows it as Sleep State on the VM. The sleep state tells where Metal keeps the saved state:
| Sleep state | Saved memory | Disk |
|---|---|---|
| Empty | None. The VM is awake. | On the host |
sleeping | On the host | On the host |
deep-sleeping | In object storage | On the host |
comatose | In object storage | In object storage |
Metal sets only sleeping today. A host with an older metald does not send the field, so Atlas shows the VM as awake.
The saved state belongs to one VM and stays on its host. It is not the shared warm artifact that speeds up a VM's first boot.
Placement
Sleepy VMs can live on their own hosts. With Use Dedicated Sleepy VM Hosts on in Atlas Settings, sleepy VMs go only to sleepy hosts, and regular VMs only to regular hosts. Auto-spawn creates a host for the right pool.
Because most sleepy VMs are asleep at any time, the balanced strategy divides sleepy memory by sleepy_vm_overcommit_factor when it ranks hosts. The capacity check still uses real free memory. spread-3 and best-fit pack sleepy VMs onto the fullest sleepy host. See placement.
Limits and recovery
- The first packet can be lost while the VM wakes. Clients must retry, as they would on any network.
- Migration keeps the memory only with a memory snapshot. A memory snapshot migration sends the saved state, and the VM resumes awake on the new host. A disk-only migration wakes the VM, shuts it down, and cold-boots it on the new host.
- Some actions throw the saved state away: an explicit stop, a restart, a shape change, or deleting the VM. The next start is a normal boot.
- A failed restore keeps the saved state for another try. Metal never cold-boots instead, because that would lose the VM's memory without warning. The exception is a migration rollback: Metal cold-boots the source so the migration cannot stay stuck.
- A
metaldrestart resets the idle clock, so a VM can sleep later than expected, never earlier. - A start resets the idle clock, so a VM is never put to sleep before the guest has booted.
To check a sleepy VM, see Automatic idle shutdown in the Metal runbook.
Source code and tests
- Idle watcher decides when to save and when to restore.
- Traffic monitor and eBPF program record host-to-guest traffic.
- Firecracker machine saves and restores the VM state.
- Placement strategies apply the sleepy pools and overcommit factor.