Boot OCI images

Beachball hosts boot signed image manifests that name a kernel and a root filesystem by digest. bb image build produces such a manifest from an OCI container image, so anything you can docker pull can run as a disposable microVM without an init system, a systemd unit, or an ip binary inside the image. The build runs on an administrator machine; hosts never talk to registries.

Let hosts build images

A host that joined with --oci-kernel VMLINUX accepts container image references directly, so bb run ubuntu:26.04 -- uname -a needs no build step:

sudo bb agent join 'https://scheduler#bbe1.…' --oci-kernel /path/to/vmlinux

The first placement of a reference pulls the image, runs the build below with a host-local signing key (oci-signing.key in the state root, trusted only by that host), and stores the result in the image cache; later placements reuse it. A tag is resolved once per host; pin a digest with a registry path (for example docker.io/library/ubuntu@sha256:…) to choose an exact image. The host needs squashfs-tools, and the agent must be able to set file ownership, as when it runs as root. A shell payload delivered to such an image, which is what bb run -- COMMAND sends, runs with /bin/sh next to the image's own workload.

Build explicitly, as below, when you want to publish a signed image to many hosts or change the entrypoint, command, user, or format.

What the build does

Resolves the reference (alpine:3.20, ubuntu:24.04, ghcr.io/org/app@sha256:...) against the registry, selects the linux/amd64 manifest from an index, and verifies the manifest, config, and every layer blob against its digest. Tags are resolved once; the resulting digests are recorded in the signed manifest.
Applies the layers bottom-up into a directory tree, honouring whiteouts and opaque directories, resolving symlinks the way the guest kernel will, and preserving ownership, modes, setuid bits, and extended attributes such as security.capability. This is why the build runs as root.
Installs the statically linked guest bootstrap as /sbin/beachball-init and writes the image's entrypoint, command, environment, working directory, and user to /etc/beachball/workload.json.
Packs the tree. The default squashfs format produces an immutable, compressed root plus a tiny initrd holding the bootstrap; the ext4 format produces a writable root the agent copies per placement.
Signs a schema 2 manifest with your Ed25519 image key and writes an env file with the values a host or the smoke harness needs.

Build an image

You need mksquashfs (squashfs-tools) or mkfs.ext4 (e2fsprogs), a Firecracker-compatible uncompressed vmlinux, the static beachball-guest binary, and a 32-byte signing key (raw or base64). The tutorial setup leaves all of these under .dev/beachball.

sudo bb image build ubuntu:24.04 \
  --kernel .dev/beachball/ubuntu-assets/downloads/vmlinux-6.1.182 \
  --guest agent/target/x86_64-unknown-linux-gnu/release/beachball-guest \
  --signing-key-file .dev/beachball/image-signing.key \
  --output-dir .dev/beachball/oci-images

The command prints the immutable imageRef for a workload class, the manifest path and base URL, the trusted public key, and the path of an env file:

imageRef=ubuntu-24.04@sha256:786c29e8...
manifest=.../oci-images/manifests/786c29e8....json
manifestBaseUrl=file:///.../oci-images/manifests
trustedPublicKey=UoytwF1H...
rootfs=.../oci-images/ubuntu-24.04.squashfs (Squashfs, 31 MB, 1 layers)
initrd=.../oci-images/ubuntu-24.04.initrd.cpio
env=.../oci-images/ubuntu-24.04.env

Useful options:

Option	Effect
`--format squashfs` (default) or `--format ext4`	Immutable root with an overlay upper on the scratch disk, or a writable root copied per placement.
`--entrypoint ARG` / `--cmd ARG` (repeatable)	Replace the image's `ENTRYPOINT` / `CMD`. `--cmd` alone keeps the image entrypoint.
`--env KEY=VALUE`	Add or override an environment variable.
`--user user[:group]`, `--workdir DIR`, `--hostname NAME`	Override the image config. Users resolve against the image's own `/etc/passwd`.
`--wait-for-bootstrap`	Start the workload only after the first bootstrap payload has been delivered.
`--name NAME`	Beachball image name; defaults to the repository's last component plus tag.
`--platform linux/amd64`	Platform selected from a multi-architecture image.
`--cache-dir DIR`	Layer blob cache (default `/var/cache/beachball/oci`).
`--max-entries N`, `--max-mib N`	Abort images that expand beyond these limits.
`--rootfs-size-mib N`	Minimum ext4 root size.

Private registries use the credentials in ~/.docker/config.json (or $DOCKER_CONFIG); credential helpers are not consulted.

How the guest boots

With the squashfs format the agent attaches the root read-only as /dev/vda, formats the placement's scratch disk as /dev/vdb, and boots the kernel with the initrd. The bootstrap, running as /init, mounts the squashfs lower and the ext4 upper, assembles an overlay, and switches root into it. Every write the workload makes lands on the upper disk, which the agent deletes with the placement. With the ext4 format the kernel mounts the copied root directly and starts /sbin/beachball-init.

In both cases the bootstrap is PID 1. It mounts /proc, /sys, /dev/pts, /dev/shm, /run, and cgroup v2, sets the hostname, raises lo, applies the private IPv6 address and route over rtnetlink when the host sends configure_guest, writes /etc/resolv.conf, and then starts the workload as entrypoint + cmd with the image's environment, working directory, and user. BEACHBALL_BOOTSTRAP_PATH names the file where a delivered bootstrap payload is stored (/run/beachball/bootstrap/payload, readable by the workload user). The console shows BEACHBALL_INIT_MODE=1, BEACHBALL_JOB_START ..., the workload's own output, and BEACHBALL_JOB_EXIT delivery_id=workload status=N when it exits. The VM stays up until the allocation is released; on release the bootstrap terminates the workload, syncs, and powers the VM off.

Images built this way carry no systemd, so guestBootstrapVersion and the vsock protocol are exactly what the Ubuntu image uses; the agent, scheduler, and placement logic do not distinguish the two.

Boot one locally

agent/scripts/boot-oci-image.sh builds an image and boots it through the real placement driver (jailer, cgroups, network namespace, nftables), asserts the console markers, tears the placement down, and writes a proof file. It needs the tutorial assets and passwordless sudo:

agent/scripts/boot-oci-image.sh alpine:3.20 \
  --entrypoint /bin/sh --cmd -c --cmd 'uname -a; id; echo BEACHBALL_WORKLOAD_DONE'

Extra arguments go to bb image build. The proof and a copy of the console land in /var/lib/beachball-oci-smoke/.

To use the image with a scheduler, point a workload class at the printed imageRef, serve the manifests/ directory (and the artifacts the manifest names) from the host's --image-manifest-base-url, and configure the host with the printed trusted public key. pnpm dev:prepare accepts the ref through WORKLOAD_IMAGE_REF.

Limits and choices

The build must run as root to preserve ownership in the image; running it unprivileged fails early with a clear message.
The Firecracker CI kernel Beachball ships mounts ext4, squashfs, and overlayfs. It has no erofs, virtiofs, or device-mapper support, which is why the layered format uses squashfs.
ephemeralDiskMiB still bounds every per-placement file: the scratch disk that becomes the overlay upper, plus the copied root for the ext4 format. A squashfs root, its initrd, and the kernel are hard links to one shared copy on the host.
One squashed root per image is attached, not one device per layer; Firecracker on x86_64 has 19 usable interrupt lines for virtio devices.
Images that need a real init system (systemd services, cron) should keep using a hand-built image such as the Ubuntu tutorial image.