Run Beachball with Firecracker

In this tutorial, we start the local scheduler, enroll a real Firecracker host, boot Ubuntu 24.04, send a shell payload to the guest, and release the allocation. Direct WireGuard tests do not require a packet relay.

Use a disposable x86_64 Linux host with KVM. The agent runs as root and manages cgroups, network namespaces, TAP devices, nftables rules, mounts, and Firecracker processes. Do not run this tutorial on a host whose network or workloads you cannot disrupt.

Run every command from the repository root.

Run the setup script

To prepare the complete tutorial host, run:

pnpm setup:firecracker

The script runs pnpm reset, installs the Ubuntu host tools and development dependencies, prepares the signed Ubuntu image, and enrolls and checks host-1. Agent setup creates the cgroup and network configuration that Firecracker needs. The image download and build can take several minutes on the first run.

The script starts the scheduler only while it enrolls the host and stops it before it exits, then prints the commands that start the scheduler, start the agent, and submit the Ubuntu job. Use the remaining sections when you want to run or inspect each setup step yourself.

Install the host tools

On Ubuntu 24.04, install the build and runtime tools:

sudo apt-get update
sudo apt-get install -y \
  binutils build-essential curl e2fsprogs iproute2 nftables openssl \
  wireguard-tools xz-utils

After setup creates the agent configuration, bb agent doctor checks the host and reports remediation for every failed requirement.

Prepare the checkout and Ubuntu image

Enter the checked-in development environment and install the repository dependencies:

direnv allow
pnpm setup

Create a local Ed25519 signing seed. Keep an existing seed because its public key identifies the trusted development image signer.

install -d -m 0700 .dev/beachball
signing_key="$PWD/.dev/beachball/image-signing.key"

if [ ! -s "$signing_key" ]; then
  openssl rand -out "$signing_key" 32
fi
chmod 0600 "$signing_key"

Build the static guest once so the image preparation step can verify the exact binary that it injects:

RUSTFLAGS='-C target-feature=+crt-static' cargo build \
  --locked \
  --release \
  --target x86_64-unknown-linux-gnu \
  --manifest-path agent/guest/Cargo.toml

guest_binary="$PWD/agent/target/x86_64-unknown-linux-gnu/release/beachball-guest"
export BEACHBALL_EXPECTED_GUEST_SHA256="$(sha256sum "$guest_binary" | cut -d' ' -f1)"
export BEACHBALL_IMAGE_SIGNING_KEY_FILE="$signing_key"

Download the pinned Firecracker 1.17.0 release, the supported kernel, and the Ubuntu 24.04 root filesystem. The script verifies each download, injects beachball-guest, builds an ext4 image, and signs its manifest.

assets="$PWD/.dev/beachball/ubuntu-assets"
./agent/scripts/prepare-ubuntu-24.04.sh "$assets"
source "$assets/runtime.env"

Any OCI container image can be booted the same way. Once the assets above exist, agent/scripts/boot-oci-image.sh alpine:3.20 (or ubuntu:24.04, python:3.12-alpine, ...) builds a signed image with bb image build and boots it through the placement driver; see Boot OCI images.

This step downloads and builds several gigabytes of data. Later runs reuse the verified files under .dev/beachball/ubuntu-assets.

Prepare and start the scheduler

Create the local credentials, scheduler secrets, and deployment manifest. WORKLOAD_IMAGE_REF places the signed Ubuntu image in the local workload class.

source .dev/beachball/ubuntu-assets/runtime.env
WORKLOAD_IMAGE_REF="$BEACHBALL_IMAGE_REF" \
  BEACHBALL_IMAGE_MANIFEST_BASE_URL="$BEACHBALL_IMAGE_MANIFEST_BASE_URL" \
  BEACHBALL_IMAGE_TRUSTED_PUBLIC_KEYS="$BEACHBALL_IMAGE_TRUSTED_PUBLIC_KEYS" \
  pnpm dev:prepare
source .dev/beachball/environment.sh

The preparation command keeps an existing root secret and creates it only when it is missing. The development manifest makes the signed Ubuntu image the default class image and publishes the image settings that hosts read when they join.

In the first terminal, start the scheduler:

pnpm dev

Keep this process open. The public API listens on http://127.0.0.1:8787.

In a second terminal, load the environment and confirm that the API is live:

source .dev/beachball/environment.sh
bb health live

You should see live: ok.

Join the host

Create a one-use join token. The token names the host and expires after an hour:

join_token="$(bb --profile local-operator host enrollment-token create host-1 --token-output -)"

Without --token-output, the command prints a ready sudo bb agent join command instead, which is what you would paste on a separate machine.

Build bb and join. Every setup value that the scheduler knows (region, network prefix, connection verification key, image settings) comes from /v1/.well-known/beachball, and the host's capacity, uplink interface, and WireGuard port are detected. The flags below only keep this checkout's state out of the system paths:

host_state="$PWD/.dev/beachball-host"
config="$host_state/agent.toml"
jailer_root="/var/bbj-$(printf '%s' "$PWD" | sha256sum | cut -c1-8)"
cargo build --locked --release --manifest-path agent/Cargo.toml

sudo "$PWD/agent/target/release/bb" agent --config "$config" \
  join "http://127.0.0.1:8787#$join_token" \
  --runtime-manifest "$PWD/.dev/beachball/ubuntu-assets/runtime-manifest.json" \
  --service-user root \
  --agent-credential-file "$host_state/agent.token" \
  --state-root "$host_state/state" \
  --runtime-root "$host_state/run" \
  --jailer-chroot-base-dir "$jailer_root"

join validates the runtime asset digests, enables the cgroup controllers, enrolls the host, and runs every host check, printing a fix for each failure. The host is active as soon as it joins; the scheduler places work on it after its first healthy heartbeat.

Start the Firecracker host

Start the agent and keep the command open:

sudo "$PWD/agent/target/release/bb" agent --config "$config" run

After a host reboot, run setup again with the same flags before run (the join token is no longer needed): it recreates the ephemeral cgroup parent, keeps the host credential, and does not enroll twice.

In a third terminal, check the host:

source .dev/beachball/environment.sh
bb --profile local-operator host get host-1

The host view reports active, and healthy once the first heartbeat arrives.

Check direct connectivity

The agent publishes its WireGuard public key and local UDP candidate through its normal control session. A client creates a Connection object only when you run bb proxy. The host then applies the signed connection grant and routes only that client's /128 through bbconn0.

For relay fallback tests, create TLS files and copy deploy/relay.example.toml to a temporary configuration. Start the unprivileged relay in another terminal:

agent/target/release/bb relay --config /absolute/path/to/relay.toml run

Stop the relay to prove that the direct path remains independent.

Start an Ubuntu VM for SSH

After the scheduler and the agent are running and host-1 is active, run:

pnpm run:ubuntu

The script creates or reuses a private Network, creates an Ubuntu allocation, and installs a local SSH public key through the one-time bootstrap payload. It then prints the bb proxy, ssh, and allocation release commands. The script does not keep a proxy or SSH process open.

Run a command in Ubuntu

bb run creates an allocation, waits for the guest readiness handshake, and delivers the command as a shell payload. The default class already names the signed Ubuntu image, so no image argument is needed:

bb run --max-lifetime 30m -- sh -c '. /etc/os-release; echo "hello from $PRETTY_NAME"; uname -a'

The command prints the allocation, including its allocationId; keep it for the steps below. The ready state means Firecracker booted the signed image and the scheduler completed both guest authentication handshakes. Pass --cpu-millis, --memory-mib, or --disk-mib to size the VM, or --script FILE to deliver a script instead of a command.

The same steps are available individually: bb allocation create --wait ready followed by bb allocation bootstrap-payload put.

The Ubuntu fixture executes text/x-shellscript payloads with /bin/sh. Read the host-local serial console before releasing the allocation:

sudo find "$PWD/.dev/beachball-host" \
  -path '*/logs/console.log' \
  -type f \
  -exec tail -n 60 {} \;

Look for BEACHBALL_JOB_START, the Ubuntu greeting, and BEACHBALL_JOB_EXIT.

Measure allocation latency

scripts/measure-allocation-latency.mjs creates allocations one at a time, waits for each to become ready, and releases it before the next. It prints where the time went three ways: the client's wall clock, the scheduler's allocation events, and the agent's own steps, which it reads from the agent's log:

node scripts/measure-allocation-latency.mjs --runs 5 --agent-log .dev/logs/agent.log
node scripts/measure-allocation-latency.mjs --class surf-linux-x64-medium --profile surf-local --json

The first allocation of an image on a host also fetches it and fills the host's shared artifact copies, and the first of an image and size builds its boot snapshot; later runs show the steady state.

Inspect and release the workload

The job API presents the same workload and uses the allocation ID:

bb job list
bb job get "$allocation_id"
bb allocation get "$allocation_id"

Release the allocation and wait for the agent to stop Firecracker and remove its resources:

bb allocation release "$allocation_id" --wait

Stop the relay, agent, and scheduler with Ctrl+C.

To discard the local celld state and start with new credentials, run:

pnpm reset

The reset command stops Beachball processes from this checkout. It removes the scheduler state (scheduler/.celld/dev, or the Docker Compose object-store volume in s3 mode), generated scheduler files, and enrolled host state. It preserves the downloaded Ubuntu assets and .dev/beachball/image-signing.key. Run pnpm reset -- --dry-run to inspect the deletion scope.

The scheduler state, signed image, and enrolled agent state remain in place. The next run can start with pnpm dev and the agent run command. If you run pnpm dev:prepare again, pass WORKLOAD_IMAGE_REF="$BEACHBALL_IMAGE_REF" again so the manifest keeps the signed Ubuntu image. Use pnpm reset only when you intend to delete the local celld state and enroll the host again.

This tutorial allows file:// image artifacts on one local host. Production hosts must load signed manifests and image artifacts over HTTPS and must omit --allow-file-images.