Pause, suspend, and fork

An allocation normally runs from boot until it is released. Four operations change that without making VMs long-lived or reusable. ADR 0004 explains the design.

A workload controller can build preemption on these: pause or suspend its own low-priority allocations to make room, and resume them when capacity returns. Beachball never chooses which allocation to preempt.

Hosts advertise what they support. bb host get lists pause-v1, snapshot-v1, and snapshot-store-v1 under features. Snapshots need a guest image built with a guest bootstrap that announces restore-v1; rebuild older images with bb image build.

Pause

bb allocation pause ALLOCATION_ID --wait     # state becomes paused
bb allocation unpause ALLOCATION_ID --wait   # state becomes ready

A paused VM gives its CPU to other placements; its memory and disk stay reserved. The client lease and the maximum lifetime keep running. When an unpause finds no free CPU, the allocation stays paused with the resume_waiting_capacity condition and resumes as soon as CPU is free.

Suspend and resume

bb allocation suspend ALLOCATION_ID --mode memory --wait   # state becomes suspended
bb allocation resume ALLOCATION_ID --wait                  # a new generation, restored

Suspend snapshots the VM on its host and removes it. The snapshot keeps disk capacity until the resume, which restores the VM on the same host as the next generation of the allocation: same allocation ID, same private address, processes and files as they were. The guest's clock is set and its random pool reseeded when it resumes.

• --mode auto (the default) takes a memory snapshot unless the host is busy writing other snapshots, then a filesystem one.
• The maximum lifetime counts running time only. The client lease keeps running, so renew it while suspended.
• A suspension may last up to the class's maxSuspendedSeconds (a day by default); then the allocation is released.
• A workload class with preemptible: true reserves disk for a memory snapshot when it is placed, so its suspend never fails for lack of disk.

Connections do not survive a suspend. A resumed VM is a new generation, so clients connect again. bb proxy --auto-resume resumes a suspended allocation and connects once it is ready.

Fork

bb allocation fork ALLOCATION_ID --count 3 --wait

This snapshots the running allocation (it keeps running) and starts three new allocations from the snapshot on the same host. The two steps are also available separately:

bb snapshot create ALLOCATION_ID --ttl 2h --wait
bb allocation create --from-snapshot SNAPSHOT_ID --acknowledge-duplicated-state --wait ready
bb snapshot delete SNAPSHOT_ID

Each copy has its own allocation ID, private address, and machine ID. It also starts with everything the parent had in memory, including credentials its workload was given, which is why a fork must set --acknowledge-duplicated-state. Deliver each copy its own bootstrap payload if it needs its own credentials. A snapshot expires at its TTL; copies restored from it keep running after it is deleted.

Images and workloads

After a restore, the guest runs /etc/beachball/hooks/after-restore if the image has one, with BEACHBALL_RESTORE_MODE set to resume, fork, or boot, and before a snapshot it runs /etc/beachball/hooks/before-snapshot. Each hook has ten seconds. Use them to reconnect, rotate what must be unique, or flush state.

Workloads that cache randomness, UUIDs, or TLS sessions in memory should refresh them in the after-restore hook: the kernel pool is reseeded, but user-space state is copied as it was. The kernel boot_id stays the same across every VM restored from one snapshot.

Boot snapshots

A host keeps one boot snapshot per image and size, taken right after a VM boots and before it is configured or given any payload. The first allocation of an image and size on a host starts a short-lived builder placement beside it. The builder's commands wait while client placements have commands of their own, and it tracks dirty pages from boot so its snapshot writes only the memory the guest touched (about 0.6 s for a 5 GiB guest on the dev host). Later allocations restore from the boot snapshot and are rebound to their own identity, which on the dev host brings a VM up in about 0.2 s of agent time (prepare and start) instead of 1.6 s cold. Boot snapshots hold no workload data. They are rebuilt when an image, size, Firecracker build, or host kernel changes.

Snapshots on another host

With snapshot_store_url set in each host's [firecracker] configuration to a filesystem every host mounts, a snapshot can outlive its host:

head -c 32 /dev/urandom | base64 > snapshot.key
bb allocation export ALLOCATION_ID --key-file snapshot.key     # a suspended allocation
bb snapshot export SNAPSHOT_ID --key-file snapshot.key         # a fork snapshot

The stored copy is compressed, leaves out zero blocks, and is encrypted with the key, which Beachball never stores. If the snapshot's host loses it, pass the key again and Beachball imports the copy onto a host that can run it:

bb allocation resume ALLOCATION_ID --key-file snapshot.key --wait
bb allocation create --from-snapshot SNAPSHOT_ID --snapshot-key-file snapshot.key --acknowledge-duplicated-state

A snapshot loads only on a host with the same Firecracker build, CPU model, and host kernel; others refuse the import.