Find and fix a problem
Start with the first failed step. Find its owning component before you retry. Keep uncertain records until that owner confirms the result. A timeout can leave an accepted request in progress. Keep the operation ID, fix the cause, and let the owner retry.
Each component has a runbook with detailed checks: Atlas, Metal, HTTP proxy recovery, and WG Mesh operations. Read the incident reports before you change recovery behavior.
Find your symptom
| Symptom | First check | Next guide |
|---|---|---|
Metal Server stays Pending or Failed | Read status, provider ID, Atlas Error Log phase, and SSH Task result. | Host provisioning |
| VM stays a draft | Read it from the assigned Metal host. Check the draft reconciliation Error Log. | VM lifecycle |
| Atlas cannot choose a host | Check sample age, host pool, image architecture, and draft reservations. | Choose a host |
| VM does not boot | Compare Metal desired and observed generations, phase, error, systemd unit, and ZFS pool. | Metal operations |
| VM list state looks old | Check latest Metal Server Usage sample and the host sync Error Log. | Host sync |
| VM has no private network | Check its Metal namespace, peer set, and mesh location. | Traffic paths |
| Site HTTP or HTTPS fails | Check DNS, proxy readiness, route generation, then mesh reachability. | HTTP proxy |
| Direct or routed public IP fails | Check the allocation request, host route or router VM, and the VM network route. | Public IPs |
| Image transfer stalls | Check Atlas image status and upload IDs, Metal snapshot status, and object storage access. | Images and snapshots |
| Migration stalls | Compare Atlas migration status with source and destination Metal records. | Migration |
Find the logs and records
| Component | Logs and state |
|---|---|
| Atlas app | Error Log, SSH Tasks, and the latest Metal Server Usage timestamp. |
| Metal | VM records under its base directory. JSON logs in the host journal. |
| VM host resources | metal-vm@<id>.service, metal-<id> namespace, and ZFS disk. |
| HTTP proxy | /var/lib/nginx/cluster-state.json. |
| WG Mesh | eBPF forwarding maps. |
Safe recovery rule
- Read the owner's record and operation ID.
- Correct the underlying fault.
- Let the owning reconciler retry.
A lost response can mean the request was accepted. Keep drafts, host IDs, upload IDs, migration records, saved state, and ZFS datasets until the owner confirms cleanup is safe.
For detailed host commands, use Metal operations. For regional jobs and records, use Atlas operations. For packet inspection, use the WG Mesh operations guide and the proxy setup checks.
Source code and tests
- Atlas scheduler queues regional reconciliation jobs.
- Host provisioner records setup failure status.
- Atlas sync job stores host capacity and VM reports.
- VM reconciliation settles uncertain create and delete results.
- Metal VM reconciliation stores host phases and errors.