Architecture

This page explains how Beachball works from the inside. It assumes no experience with schedulers or with Firecracker. It starts with the problem the system solves and names the pieces. It then follows one allocation from the API request to a running microVM and back to cleanup. Later sections cover the host runtime, private networking, failure handling, and the repository layout.

The normative sources are the architecture specification and the MPR specification. This page describes what the code in this repository does today. Where the broader design and the current release differ, the last section says so.

The problem a scheduler solves

A scheduler answers one question, over and over. Given a fleet of machines and a stream of requests for compute, which machine should run each request, and how do you make sure the machine does it and later gives the resources back?

Beachball narrows that question to one kind of work. Every request asks for a single disposable Firecracker microVM. The VM boots from an administrator-approved image, runs one workload, and is destroyed when the requester releases it or when a time limit expires. Beachball does not know or care what runs inside the VM. A separate program, called a workload controller, owns that knowledge. The first controller, a GitHub Actions runner controller, turns queued CI jobs into VM requests. Beachball itself has no GitHub logic.

That narrowing makes several hard scheduler problems simpler:

• Every VM is short-lived and never reused, so there is no rescheduling, migration, or rolling update.
• Every VM's shape comes from a named workload class, so the scheduler compares fixed resource numbers instead of interpreting arbitrary specifications.
• Every request is idempotent and every VM has an expiry, so retries are safe and abandoned VMs clean themselves up.

The problems that remain are the ones this page is about. Never hand out more capacity than a host has. Keep the host in sync with what the scheduler decided, even when either side crashes. Fence stale work so an old command cannot revive a VM that was already replaced. Connect a client to a VM that sits behind an outbound-only firewall.

System overview

Beachball has three runtime roles plus the clients that use it.

The scheduler is a TypeScript Worker built on SQLite-backed Durable Objects. The specification also calls it the control plane. It runs on celld 0.6.0, a self-hostable implementation of the Cloudflare Workers programming model. It holds all authoritative state and makes every placement decision. It never touches a VM and never carries VM traffic.

The host agent is bb agent run, a privileged Rust process on each Linux machine with KVM. It connects only outbound to the scheduler. It reconciles the VMs on its host toward the state the scheduler asked for, and it runs the host end of the private network.

The packet relay is bb relay run, an unprivileged process that forwards encrypted WireGuard datagrams between a client and a host when the two cannot reach each other directly. It cannot decrypt what it forwards.

The guest bootstrap is beachball-guest, a small static binary in every approved image. It proves the VM's identity to the agent over vsock, applies network configuration, and hands a one-time bootstrap payload to the workload.

One Rust binary, bb, contains the agent, the relay, and the client and operator CLI. The subcommand selects the role. Entering one role never initializes another role's state or credentials.

Vocabulary

Design principles

The core responsibility split is one sentence from the specification:

Controllers submit jobs by requesting allocations. Allocation objects choose candidates. Host objects grant resources. Agents execute grants on hosts.

Several rules follow from it. Together they explain most of the shape of the code.

Only a Host object admits work against its capacity. Directories, caches, and hints can influence which host gets tried first. The reservation transaction inside the Host object is the only place that consumes capacity. At worst, a stale directory causes a rejected reservation or a slightly worse placement.

Intent is persisted before side effects. Every transition writes its intent and an audit event to SQLite in one transaction, and only then performs remote work. If the Worker dies between the two, an alarm resumes the work from the persisted intent. A side effect is never the only record that something happened.

Everything is idempotent under retry. Clients send idempotency keys, commands have deterministic IDs, and observations carry sequence numbers. Delivery is at-least-once. Correctness comes from making duplicates harmless, not from preventing them.

Stale work is fenced in layers. Generation numbers, agent epochs, execution leases, and command IDs each reject a different kind of stale message. The section on fencing lists them.

Capacity is released only after absence is observed. A placement stops consuming capacity when the agent reports it absent for the current generation, or when an operator fences the host. A timeout alone never frees capacity.

Secrets never rest in the scheduler. Bootstrap payloads travel through memory only. WireGuard private keys stay on the endpoint that generated them. The scheduler stores hashes, verifiers, and public keys.

Workload logic stays outside. No scheduler schema, state, or API field is specific to GitHub or any other workload provider.

The scheduler

Worker and Durable Objects

A Durable Object is a single-threaded, named instance of a class. It owns its own SQLite database and can schedule an alarm to wake itself later. Requests to the same name always reach the same instance, so the instance can serialize access to its state without locks or a separate database. Beachball uses that property as its concurrency control. There is no PostgreSQL, Redis, or etcd.

The Worker entry point, scheduler/src/index.ts, is stateless. It authenticates each request, resolves the caller's identity, and forwards the request to the right object with the identity attached as internal headers. Objects that expose internal endpoints verify a shared internal secret before they accept a call from the Worker or from another object.

The Allocation object name is a hash of the client's identity and its clientRequestId. Two identical requests from the same client land on the same object. That is what makes create-allocation idempotent without a separate lock. The Allocation, Host, Network, and Connection objects are authorities. Their SQLite state is the truth for their resource. Directories are caches that can be rebuilt. Losing one delays or misdirects a lookup but changes no authority.

Authentication and identity

Every request carries a bearer credential in the Authorization header. The Worker compares its SHA-256 digest with the deployment manifest and derives the identity from the match. A request body cannot assert an identity.

All configuration comes from one deployment manifest. The manifest names secrets by reference, for example the internal object secret, the audit sink token, and the Ed25519 connection-signing key. The runtime's secret mechanism resolves each reference at use time.

Admission and limits

Before the Worker creates an allocation object, it asks the AdmissionObject to admit the request. The AdmissionObject enforces the controller's requests-per-second and concurrent-allocation limits, the deployment-wide rate and allocation limits from the manifest, and a cap on retained records. A rejected request returns 429 with Retry-After or 409 with a stable quota code, and no allocation object exists. The same object gates host registration and activation against the declared maximum registered and active host counts. That is how the deployment refuses work before it exceeds a limit it declared.

Allocation lifecycle

States

Every allocation moves through a fixed set of states. The scheduler has no running or succeeded state because it does not know what success means for the workload. After ready, the VM stays up until something ends it.

paused, suspending, and suspended come from pause, suspend, and fork: a paused VM gives back its CPU, a suspended one its CPU and memory, and a resume restores the VM as the next generation. released means the VM is gone and the host has transactionally returned its capacity. failed is the same terminal confirmation with a recorded reason, such as NO_CAPACITY or GUEST_BOOT_TIMEOUT. An allocation whose cleanup is incomplete stays in releasing with a CLEANUP_PENDING condition. A failure never hides incomplete cleanup.

The reconciliation loop

Every non-terminal allocation object owns an alarm, scheduled at most 60 seconds out. An API call, an agent observation forwarded by the Host object, an operator action, and the alarm all enter the same single-flight reconcile function. Each pass:

1. Reads the durable state.
2. Applies any pending inbox events idempotently.
3. Chooses at most one next intent.
4. Persists that intent and an event in one transaction.
5. Performs bounded remote work, such as asking a host to reserve or sending a command.
6. Persists the result only if the generation and placement still match.
7. Schedules the next alarm if the allocation is still non-terminal.

The loop is the only place that changes state. So "the client called release", "the host reported the VM absent", and "the alarm fired after a Worker crash" all run the same code path.

Timeouts

Timeouts are policy. These are the defaults the reconciler enforces.

Replacement

A retry after a failed attempt creates a new placement and increments the generation. The allocation ID stays the same. The scheduler replaces an attempt automatically only when the guest never reported readiness and no bootstrap payload was consumed. After that point the old VM may hold external credentials, and a generation number cannot revoke those credentials on a host that is unreachable. So a lost placement after readiness needs one of two things. Either an operator fences the host, and fencing revokes the host credential, or the controller certifies through POST /v1/allocations/{id}/replacement-authorizations that it revoked the workload's external authority. Until then the allocation shows a replacement_blocked condition and no replacement VM starts.

Placement: choosing a host

Placement happens inside the pending → reserving → provisioning transitions.

Advertisements. Each Host object publishes a compact advertisement to the directory after any material change and at least every 30 seconds. The advertisement carries the admin state, heartbeat time, region, architecture, labels, protocol versions, free CPU, memory, disk, and slot hints, and cached image digests. Each advertisement has a monotonic revision, so a late older one cannot overwrite a newer one.

Filtering. The directory keeps only hosts that pass every check:

• The host is administratively active and its heartbeat is healthy.
• The architecture matches and the region is allowed.
• The host has every required label.
• The host speaks a compatible agent and guest protocol version.
• The host reports its isolation, local state, and image cache as healthy.
• The host's clock skew is within tolerance.
• The host's hints leave room for the request.

Scoring. The directory sorts the survivors by these keys, in order:

1. The required image is already cached.
2. The heartbeat is freshest.
3. The least memory remains after placement.
4. The least CPU remains after placement.
5. A stable hash of allocation_id + host_id breaks ties.

Keys 3 and 4 pack hosts tightly instead of spreading load. That is deliberate. Tight packing keeps larger holes free for larger requests.

Reservation. The allocation tries the candidates one at a time, never in parallel, so a request never holds tentative reservations on two hosts. Each attempt uses a deterministic idempotency key, reserve:{allocation}:{generation}:{host}, so a retried call after a crash returns the earlier answer. The Host object rechecks every hard constraint from its own state. It computes used capacity by summing its capacity-consuming placements, which are those in reserved, preparing, running, stopping, or cleanup_pending. It rejects with a stable code such as HOST_NOT_ACTIVE, INSUFFICIENT_CPU, or PLACEMENT_LIMIT_REACHED. When it accepts, it inserts the placement, its first execution lease, and an ensure_present command in one transaction. This release never overcommits CPU, memory, disk, or slots.

No capacity. If every candidate rejects, the allocation returns to pending, records which hosts it tried, emits an allocation.no_eligible_host alert, and asks the directory again after backoff.

Hosts

Enrollment

A host joins the fleet through a one-use token.

1. An administrator runs bb host enrollment-token create HOST_ID, which prints a bb agent join command carrying the scheduler URL and the token. The Worker mints a token bound to that host ID with an expiry of at most one hour (one hour by default), and stores only the token's digest.
2. On the host, bb agent join (or bb agent setup) reads the region, ULA prefix, connection verification keys, and image settings from GET /v1/.well-known/beachball, detects the host's capacity and uplink, validates the packaged runtime manifest, and writes the agent configuration. It then calls POST /v1/agents/enroll with the token and the host's registered capacity, region, labels, and protocol versions.
3. The scheduler consumes the token, derives a unique host credential, reserves a registered-host slot in admission, and creates the Host object in the draining state with credentialVersion: 1.
4. Setup writes the credential to a root-owned file with mode 0600. If you run setup again with an existing credential, it validates the credential and does not enroll again. If you reuse the token with different host configuration, enrollment returns ENROLLMENT_CONFLICT.
5. With hostEnrollment.autoActivate (the default), the Worker activates the host right after its first enrollment, subject to maxActiveHosts; placement still waits for a healthy heartbeat. Otherwise an operator runs bb agent doctor and, once every check passes and a smoke allocation works, bb host activate HOST_ID.

An administrator can instead register a statically provisioned host through PUT /v1/hosts/{id}/register. The request must match a host entry in the manifest.

Administrative state and health

type HostAdminState = "active" | "draining" | "disabled";
type HostHealth = "unknown" | "healthy" | "suspect" | "lost";

active accepts reservations. draining accepts none but keeps existing placements running. disabled fences the host. The Host object clears the credential digest, bumps the credential version so the agent can no longer authenticate, and raises an alert. To reuse the physical machine later, an administrator assigns a new host ID or performs an explicit identity reset.

The Host object derives health from heartbeat age: healthy under 30 seconds, suspect from 30 to 90 seconds, lost after 90 seconds. Only active and healthy hosts receive reservations. Four other conditions also make a host unschedulable: its reported inventory disagrees with its registered capacity, its clock skew exceeds 5 seconds, its isolation self-check fails, or its local state or image cache volume is nearly full.

Capacity accounting

An operator registers capacity when they enroll a host: CPU millicores, memory MiB, ephemeral disk MiB, and a maximum placement count. allocatable = registered − operator reserve. The agent reports physical inventory in every heartbeat, but the registered capacity is authoritative. A mismatch makes the host unschedulable and raises an alert. It never silently changes capacity.

ephemeralDiskMiB in a workload class covers every writable file for that allocation: the copied root disk, the scratch disk, and filesystem overhead. Shared immutable image-cache artifacts do not count against it.

The agent protocol

Transport

The agent initiates every connection and speaks only HTTPS to the scheduler, so a host needs no inbound port for control traffic. This release uses long polling. WebSockets are deferred.

• POST /v1/agents/poll carries the agent's boot ID, current epoch, lease renewals, placement updates, a health report, and a connectivity update. The Host object parks the request for up to 25 seconds, until a command is enqueued or the wait expires. It then returns the current epoch, the authoritative lease expiries, and up to 16 commands.
• POST /v1/agents/messages carries the same kinds of updates plus command acknowledgements, for messages that cannot wait for the next poll. Two heartbeat threads in the agent use it every 8 seconds. One renews leases while a poll is outstanding. The other renews the single placement's lease while a long command such as image preparation runs.

Both routes are rate-limited per host and update the host's heartbeat time. The agent retries after a transport failure. If the scheduler reports an overlapping session, the agent pauses for 90 seconds.

Sessions and epochs

The first poll after an agent starts creates a session with a fresh random agent epoch. The Host object attaches that epoch to every command. It rejects updates, acknowledgements, and lease renewals from any older epoch with STALE_AGENT_EPOCH. A new boot ID while the previous session is still fresh marks the host OVERLAPPING_AGENT_SESSION and suspect until one epoch wins. Rotating the epoch clears stale observations and reissues outstanding commands under the new epoch.

Messages

Every message is a JSON envelope of at most 256 KiB:

interface Envelope<T> {
  protocolVersion: "1.0";
  messageId: string;
  hostId: string;
  agentEpoch: string;
  sentAt: string;
  type: string;
  payload: T;
}

Every command from the scheduler carries the allocation ID, placement ID, generation, agent epoch, execution lease expiry, and a command ID. The command types are:

Command IDs are deterministic, for example {placement}:present:{generation}, so redelivering a command is harmless. Before the agent performs any side effect, it persists the command ID, a digest of the payload, the generation, and the desired state. A duplicate returns the earlier result. A reused ID with different content is rejected as COMMAND_CONFLICT and does nothing.

The agent sends four kinds of observations. placement_update carries the observed state, a strictly increasing per-session sequence number, and a bounded error code. command_ack carries accepted, rejected, or duplicate. The health report carries inventory and self-checks. connectivity_update carries the host WireGuard public key, endpoint candidates, applied grants, and path observations. When the Host object observes prepared, it enqueues ensure_running on its own. When it observes absent, it releases capacity and supersedes any outstanding command for that placement. Every observation nudges the owning allocation to reconcile.

The placement observed states are unknown, accepted, preparing, prepared, starting, running, stopping, absent, and error. Agent errors use stable codes such as IMAGE_DIGEST_MISMATCH, INSUFFICIENT_LOCAL_DISK, FIRECRACKER_START_FAILED, GUEST_BOOT_TIMEOUT, NETWORK_SETUP_FAILED, STALE_GENERATION, LEASE_EXPIRED, and CLEANUP_FAILED. The scheduler, not free text, decides which codes are retryable.

One allocation end to end

The bootstrap payload is the one secret that crosses the whole system, so its path matters. The controller sends it with a delivery_id and the current generation. The Worker forwards it synchronously to the Host object. The Host object keeps it in an in-memory map keyed by delivery, includes it in the next poll response, and resolves the controller's HTTP request when the agent acknowledges consumption. If the guest is not connected, the generation is stale, or 30 seconds pass, the controller receives 409 or 503. The controller still holds the secret, so it can retry. No component writes the bytes to disk. The Host object and the agent persist only the delivery ID and a digest, so a repeated delivery returns the earlier result without serving the bytes again.

Fencing: how stale work is rejected

In a distributed system, an old message can arrive after a newer decision. Beachball uses several independent checks so that each kind of stale message meets one that rejects it.

The execution lease decides what happens during a scheduler outage. Agents keep VMs running only until the current lease expires, then shut them down. The lease trades workload availability for isolation and bounded stale execution. Its length is a policy value. Raising it makes that trade explicit.

On the host

Agent runtime and local state

The agent runs as the beachball system user with a fixed set of ten Linux capabilities, among them CAP_SYS_ADMIN, CAP_NET_ADMIN, and CAP_SYS_CHROOT. A hardened systemd unit grants it access only to /dev/kvm, /dev/net/tun, its state directories, /run/netns, and its cgroup subtree. Firecracker itself runs unprivileged inside jailer.

Local state lives under /var/lib/beachball by default:

/var/lib/beachball/
  agent.lock                    exclusive process lock
  agent.db                      SQLite: allocations, processed commands, bootstrap receipts
  images/
    manifests/  artifacts/      content-addressed, verified cache
    quarantine/                 artifacts that failed verification
  allocations/bb-<digest>/      one directory per placement
    metadata.json  runtime.json  network.json
    logs/console.log            bounded serial console capture
  wireguard/host.key            host WireGuard private key, mode 0600
  wireguard/connections.json

An allocation directory name is a SHA-256 of the allocation, placement, and generation identity, never client-supplied text, so no request value can become a path. agent.db records the desired and observed state, the highest accepted generation, processed command IDs with payload digests, execution lease data, and the Firecracker process identity. Writes use SQLite transactions and atomic renames, so a crash at any point leaves either the old state or the new state.

A starting agent does three things before it accepts any command. It takes the lock. It runs the same prerequisite registry as bb agent doctor. The registry checks binary digests, the KVM and TUN devices, cgroup v2, capabilities, the ip, nft, and mkfs.ext4 commands, a probe network namespace, and scheduler liveness. Then it inventories every Beachball-owned resource it can find: allocation directories, Firecracker processes, jailer directories, network namespaces, TAP devices, cgroups, mounts, nftables tables, writable disks, and sockets. The agent adopts a resource only if its allocation, placement, and generation match the database exactly. It quarantines and reports anything else that carries the Beachball prefix, and never deletes it on its own. A non-empty quarantine marks the host's isolation unhealthy. An unhealthy host receives no new reservations, and its agent refuses every command except ensure_absent until an operator resolves the quarantine.

From command to microVM

Prepare (ensure_present). The agent parses the resolved spec into a machine shape. The vCPU count is ceil(cpuMillis / 1000), and memory and disk come straight from the spec. It fetches the image manifest by digest from the configured manifest base URL and verifies the manifest's Ed25519 signature against the host's trusted keys. It then downloads the kernel and root filesystem into private temporary files, verifies their exact size and SHA-256, and publishes them into the cache with an atomic rename. Anything that fails verification goes to quarantine/. Next it picks an unused UID and GID for the placement from the configured range and builds the network namespace and firewall described below. It creates the jailer chroot with the kernel, a rootfs.squashfs (attached read-only, plus the manifest's initrd) or a private rootfs.ext4 (writable), and a freshly formatted sparse scratch.ext4, which a squashfs guest uses as its overlay upper. Read-only artifacts are hard links to one copy per digest under shared-artifacts/ in the jailer chroot base, owned by root and mode 0444, so a placement copies nothing the guest cannot write. An ext4 root is cloned from that copy, with a reflink where the filesystem supports one and otherwise by copying only its data blocks. The agent logs the time each step took with the prepared placement and started placement lines. Last, it writes runtime.json, which includes a random 256-bit bootstrap nonce, and vm-config.json. If any step fails, the agent rolls back what it created. If the rollback is incomplete, it reports CLEANUP_FAILED.

Start (ensure_running). The agent requires a live execution lease. It then runs the pinned jailer with the placement's UID and GID, chroot, network namespace, and cgroup v2 limits. The limits are cpu.max, memory.max, memory.swap.max=0, pids.max, and file-size limits sized to the ephemeral disk. Jailer drops into the chroot and executes Firecracker with seccomp enabled and no ambient capabilities. The VM configuration has two drives, one TAP interface, and a vsock device. Its kernel command line forces console=ttyS0 reboot=k panic=1 pci=off, adds root=/dev/vda rw for ext4 roots or beachball.lower=/dev/vda beachball.upper=/dev/vdb for squashfs roots booted through the initrd, and injects the guest's identity as beachball.allocation_id, beachball.placement_id, beachball.generation, and beachball.bootstrap_nonce, plus IPv4 and DNS settings. The agent captures Firecracker's stdout and stderr, which include the guest serial console, to logs/console.log under a byte budget.

If an agent restarts while a VM is running, it adopts the process instead of restarting it. First it verifies the PID, the process start time, and the instance ID in the command line, and pings the guest over vsock.

Guest handshake. The agent listens on vsock port 1025. When beachball-guest starts, it binds its own control port 1024 first, then connects to the host and sends GuestReady with its identity, boot ID, guest version, and capabilities. The agent accepts the message only if the allocation ID, placement ID, generation, and nonce all match the current placement. The agent then proves the reverse direction with a ping, sends configure_guest with the private IPv6 address and route, and reports running. If jailer exits before this handshake completes, the agent treats it as a boot failure.

Stop (ensure_absent). Cleanup attempts every safe step even when an earlier one fails:

1. Revoke bootstrap delivery.
2. Ask the guest to shut down over vsock. The guest syncs its disks.
3. Send SIGTERM to the verified Firecracker process group, then SIGKILL after 30 seconds.
4. Unmount the allocation's mounts.
5. Remove the TAP device and the network namespace.
6. Delete the allocation's nftables tables.
7. Remove the cgroup.
8. Delete the disks, sockets, console log, and allocation directory.
9. Persist absent locally and report it.

not found counts as success only after the agent confirms that no conflicting resource exists.

Guest network isolation

Each placement gets its own network namespace and a veth pair between the namespace and the host. Inside the namespace, a TAP device owned by the placement UID connects to the VM's eth0. The guest is dual-stack:

• Every guest uses the same IPv4 /30 (10.0.0.2, router 10.0.0.1) and MAC, so a snapshot restores into any placement. The allocation namespace translates the guest address to an allocation-local host-visible address, and the host masquerades that address out of the uplink interface for controlled public egress.
• An allocation-owned IPv6 ULA /128 from the tenant's Network provides authorized private access. The host routes it into the namespace.

The agent writes the placement's nftables rules before it starts Firecracker. The rules default to deny. They allow DNS only to the configured resolvers and allow connection-tracked IPv4 egress to public addresses. They drop traffic to RFC 1918 ranges, carrier-grade NAT ranges, link-local ranges including the 169.254.0.0/16 metadata range, multicast, and any operator-configured management ranges. They permit IPv6 from the guest only from its exact assigned address and only through the host's WireGuard endpoint interface. Every table and set carries a Beachball-derived name, so the restart inventory can identify and remove it without parsing arbitrary output.

Images

An image is referenced by the digest of its signed manifest:

interface ImageManifest {
  schemaVersion: 1 | 2;
  architecture: "x86_64";
  kernel: Artifact; // url or object key, byte size, sha256
  rootfs: Artifact;
  rootfsFormat?: "ext4" | "squashfs"; // schema 2; ext4 when absent
  initrd?: Artifact; // required for squashfs roots
  kernelArgs: string[];
  agentProtocolRange: string;
  guestBootstrapVersion: string;
  source?: { kind: "oci"; reference; manifestDigest; configDigest; platform }; // schema 2
  createdAt: string;
  signature: string; // Ed25519 over canonical bytes
}

A host trusts only manifests signed by the keys in its configuration, so a compromised object store cannot serve a different kernel. A workload class references an image by digest, and each allocation persists the resolved reference, so a later change to the class never changes a running VM. Cold boot from kernel and rootfs is the default mode. Pause, suspend, snapshot restore, and fork follow ADR 0004.

Two kinds of image share this manifest. The pinned Ubuntu 24.04 build has systemd as PID 1 and beachball-guest as an enabled service; the guest passes a delivered payload to whatever the image configured as $BEACHBALL_WORKLOAD_ADAPTER, and the reference image runs text/x-shellscript payloads with /bin/sh. Images converted from OCI container images by bb image build have no init system: beachball-guest is PID 1, either from the manifest's initrd (squashfs root, the default) or as /sbin/beachball-init inside the image (ext4 root). It provides the mount table a container expects, applies the host's network configuration over rtnetlink, starts the image's entrypoint from /etc/beachball/workload.json, and stores a delivered payload at BEACHBALL_BOOTSTRAP_PATH. Both kinds write BEACHBALL_JOB_START and BEACHBALL_JOB_EXIT markers to the console, and Beachball never interprets the payload. Boot OCI images covers the build.

Private networking

Hosts sit behind NAT or outbound-only firewalls, and the scheduler cannot carry UDP. So reaching a port inside a VM needs its own design. ADR 0003 records the decision. The client and the host hold one end-to-end WireGuard session. The scheduler coordinates that session but never carries it. A stateless encrypted relay is the fallback.

Networks, addresses, and grants

A Network is a tenant-owned routing and authorization boundary. The deployment owns one random RFC 4193 ULA /48. Each Network receives a /64 from it, which the network directory assigns sequentially. Each allocation and each client peer receives a /128 inside its Network. No address is derived from an identifier. An allocation's address belongs to the allocation and survives replacement on another host. The scheduler reserves the address at creation, before it chooses any host, and reuses it only after every grant for the old generation is gone and cleanup has completed.

Membership alone permits no traffic. Access requires a connection grant. A grant is a document signed with the deployment's Ed25519 connection-signing key. It names the grant, tenant, Network, allocation, placement generation, host, both WireGuard public keys and private addresses, the allowed transport and destination ports, the policy revision, and an expiry. Clients and hosts verify the signature against configured verification keys and reject anything that does not match exactly. A grant targets exactly one generation and one host peer. The scheduler never issues a grant that spans a replacement.

Data path

bb proxy ALLOCATION_ID --local-port 2222 --remote-port 22 runs without root and without touching the operating system's routing table. It does the following:

1. Generates an ephemeral X25519 key pair in memory.
2. Calls POST /v1/allocations/{id}/connection-grants with the public key, transport, and ports. The Network object checks policy and fetches the host's current WireGuard public key and candidates from the Host object. It then assigns a client address, signs the grant, signs a relay map, and mints per-role relay tokens and endpoint tokens.
3. Verifies every signature, then binds 127.0.0.1:2222 and runs a boringtun WireGuard tunnel over a plain UDP socket with a userspace TCP/IP stack on top.
4. Publishes its own local and STUN-discovered candidates to the Connection object and learns the host's candidates from it.
5. Forwards each accepted local TCP connection to [allocation IPv6]:22 through the tunnel.
6. Revokes the grant on exit. If the scheduler is unreachable, both endpoints drop the peer when the grant expires.

On the host side, the agent runs a userspace WireGuard endpoint on a bbconn0 TUN device with one UDP socket, on port 51820 by default. The same socket carries WireGuard traffic, STUN candidate discovery, and direct-path checks. Sharing the socket keeps one NAT mapping alive for all three. The agent refreshes candidates every 20 seconds. For each applied grant it keeps a peer with the client's public key and an inner filter. The filter allows only the exact source and destination address pair, the grant's protocol, and its destination ports, and it admits return traffic only for recorded flows.

Coordination and path selection

The Connection object is a rendezvous point. Both sides authenticate with the endpoint token from the grant. They exchange candidate, path_observation, and ack messages through GET /v1/connections/{id}/connect over WebSocket, or through POST …/poll and POST …/messages. Messages carry per-role sequence numbers and are bounded in count and size. The object stores no private key, no packet, and nothing that outlives the grant.

Until a direct path is confirmed, each side sends its encrypted datagrams to every known candidate of the peer and mirrors them through the relay. The first datagram that decrypts pins its source address as the direct endpoint. Switching from the relay to a direct path changes only the transport address. The WireGuard session, the private addresses, and the grant stay the same, so inner TCP connections survive the switch.

Packet relays

A relay is bb relay run. It is a stateless process that listens for QUIC and WebSocket over TLS on port 443 and exposes a health endpoint on port 9443. Each endpoint opens an outbound session and presents a short-lived signed relay token as its first frame. The relay verifies the signature, relay ID, freshness, and limits. It then pairs exactly one client role and one host role per grant ID and forwards frames between them under byte, rate, and idle limits. Dropping a session frees the pair. The relay holds no scheduler credential, no WireGuard key, no route, and no firewall authority. Its systemd unit has only CAP_NET_BIND_SERVICE and cannot see agent state or /dev/kvm. Clients try QUIC first and fall back to WebSocket on networks that block UDP. A relay can observe connection metadata such as endpoints, sizes, and timing. It cannot read inner packets.

Revocation and replacement

Revoking a grant sends a configure_connection revoke command to the host and marks the grant revoked. The host also removes expired peers on its own, without waiting for a command. When an allocation must move to a new generation, the Network object refuses to advance until the host has observed every grant for the old generation as revoked or expired. Until then it answers CONNECTIONS_NOT_FENCED. That rule guarantees that one private address is never routed through two hosts at once. It holds even when the old host is unreachable, because the old host's grants expire on their own.

Failure and recovery

Observability and retention

Every meaningful transition writes a structured event. The event carries the tenant, client, allocation, placement, host, and generation, the prior and new state, a stable reason code, and correlation IDs. Allocation events are readable through GET /v1/allocations/{id}/events and bb allocation events --follow. Host events mirror into the Operations object.

The Operations object durably records events, audit entries, and alerts, deduplicated by event ID, and exports them. Audit records go to the manifest's audit sink. Alerts fan out to every configured alert destination with a runbook anchor. While a backlog exists, export retries from durable cursors on a 60-second alarm. Metrics are counters and gauges with a small allowed dimension set (result, reason, state, workload class, host, and phase), readable at GET /v1/operations/metrics. A raw allocation ID never becomes a metric label.

Retention is alarm-driven and idempotent. A terminal allocation deletes itself after the configured number of days, and first purges its directory row, lookup route, and admission record. Host objects prune events, operator mutations, and acknowledged commands. The host directory drops advertisements older than 24 hours. Enrollment tokens are purged 24 hours after expiry.

Redaction is a design rule, not a filter. Credentials, bootstrap payloads, WireGuard private keys, and raw request bodies are never placed in events, logs, traces, or support output in the first place.

Repository map

scheduler/                 TypeScript scheduler
  src/index.ts             Worker entry point: routing, authentication, request post-processing
  src/config.ts            deployment manifest schema
  src/policy.ts            workload-class resolution
  src/domain/              pure state machines and validation (allocation, host, agent, network, connection)
  src/objects/             one Durable Object class per file
  tests/                   Vitest domain and contract tests
  tests-integration/       tests against a running celld

registry/                  OCI image registry Worker (ADR 0008): blobs in R2, repositories in Durable Objects
  src/index.ts             Distribution API routing, uploads, blob and manifest reads
  src/auth.ts              token realm and scheduler-credential principals
  src/repository.ts        RepositoryObject and the catalog directory
  scripts/dev-registry.sh  local celld instance on 127.0.0.1:8792

agent/                     Rust workspace producing the single `bb` binary
  src/main.rs, lib.rs      entry point; `cli/` dispatches to a role
  src/agent_command.rs     `bb agent setup | doctor | register | status | run`
  src/control.rs           long-poll control client, heartbeats
  src/reconcile.rs         startup inventory, adopt or quarantine
  src/store.rs, state.rs   SQLite authority and on-disk layout
  src/image.rs             signed manifest resolution and verified cache
  src/firecracker.rs       prepare, start, adopt, stop; VM configuration
  src/jailer.rs            jailer invocation and validation
  src/network.rs           namespaces, veth, TAP, routes, nftables
  src/endpoint.rs          host userspace WireGuard endpoint (bbconn0)
  src/candidate.rs         local and STUN candidate discovery
  src/connection.rs        signed grant, relay map, and token types
  src/relay.rs             `bb relay run`
  src/relay_client.rs      QUIC and WebSocket relay links
  src/client_proxy.rs      `bb proxy` userspace tunnel and TCP stack
  src/host_check.rs        prerequisite registry shared by doctor and daemon startup
  src/protocol.rs          scheduler-to-agent envelope and command types
  guest/                   `beachball-guest` bootstrap binary
  protocol/guest/          host-to-guest vsock protocol
  packaging/, scripts/     .deb and .rpm packaging, image preparation

deploy/                    manifest and relay examples, systemd units
adr/                       architecture decision records
docs/                      this site

The scheduler's domain modules do not import the runtime SDK, HTTP routing, or any workload-provider concept. The agent's privileged modules sit behind the agent subcommand boundary, so client commands cannot initialize host state.

Release profile and deferred work

The architecture specification describes a broader system than this release ships. The minimal production release (MPR) profile narrows it, and a deployment may claim only MPR conformance:

Also outside this release: host autoscaling, warm pools, live migration, GPUs, public service ingress, a web console, and any Layer 2 overlay between hosts. Each of these needs a specification amendment that revisits the invariants and the threat model before anyone builds it.

Further reading

• Run Beachball with Firecracker walks through enrolling a host and booting a guest on one machine.
• Connect to a private VM covers bb proxy from the user's side.
• Installation describes the packaged host and relay services.
• The architecture specification, the MPR specification, and the bb command specification are the normative documents.
• ADR 0001 explains the toolchain choices. ADR 0003 explains direct WireGuard connectivity.