Deploy Surf for an internal team

This runbook deploys the current repository sessions, prompt queue, agent runtime, and web UI. It assumes trusted team members, one management machine, a private network, and x86-64 Linux hosts with KVM. Hosting provider, addresses, credentials, and capacity are deployment inputs; the example files are not ready to start.

The first deployment accepts a management-machine outage. Session records and timelines are durable, but the VM workspace is disposable: push commits to Git before idle release, expiry, or shutdown. There is no workspace restore. The default idle release is 30 minutes and the Surf session lifetime is four hours. All users share the deployment's repository credential and model account.

Layout

The scheduler and Surf are two separate celld fleets. Never deploy them to the same bucket root. The templates use separate buckets. celld also supports separate prefixes. A single node per fleet waits for object storage on every acknowledged write; measure prompt and timeline latency during the canary. See celld storage and deployment.

Use a store celld qualifies, such as S3, R2, or Tigris. The development Silo configuration is not a production default. Verify the chosen endpoint with celld diagnose, including its write probe, before using it. See celld storage guarantees.

Keep celld's unauthenticated internal listeners on loopback in this layout. TLS terminates at nginx, which overwrites forwarded host and protocol headers; celld must have CELLD_TRUST_FORWARDED_HEADERS=1. Otherwise scheduler requests behind the proxy can fail its HTTPS check. See celld security.

No relay service is needed when the gateway can reach each host's advertised UDP endpoint. Set networking.relay.required to false and relays to []. The gateway's user-space bb proxy needs neither root nor a host TUN interface. Guest port 7700 stays inside the granted tunnel.

1. Choose the deployment inputs

Record these before provisioning:

• Management machine and host addresses, region, host OS, and KVM availability.
• Private routing from the gateway to every host, including UDP 51820.
• Three names with trusted TLS certificates: surf, scheduler, and images. They can resolve to the same management address. Hosts and the management machine must also resolve and reach them.
• Two object-store buckets, region, endpoint, and scoped credentials; a separate backup destination and recovery procedure.
• One workload class and its concurrency budget. The current medium class uses 4 vCPUs, 8 GiB RAM, and 20 GiB scratch per VM. Reserve additional host resources for the OS, agent, image cache, and VM overhead.
• A repository credential for the allowed team repositories, one model provider account, and a model supported by the image's OpenCode runtime. The example uses a provider API key; change its guest environment variable for your provider.
• Working audit and alert receivers. The scheduler manifest's example URLs are placeholders; setting a URL does not install a receiver.

Use a new RFC 4193 ULA /48 consistently in the scheduler and hosts. Block guest access to management networks, host control endpoints, cloud metadata, and object-store control access using host policy and network firewalls. Preserve guest access to DNS, GitHub, required package registries, and the model provider.

2. Build and record a release

Use a recorded source revision containing all intended changes. Build on Linux x86-64 with the repository's Rust toolchain, Node 24, pnpm 11.18.0, Docker, SquashFS tools, and the package-building prerequisites. Do not run setup:firecracker, surf:prepare, surf:register, or the development launchers on production machines: they manage development state and credentials.

From the checkout:

pnpm install --frozen-lockfile
pnpm check
pnpm test
pnpm --recursive build
cargo build --locked --release --manifest-path agent/Cargo.toml \
  --target x86_64-unknown-linux-gnu --bin bb
RUSTFLAGS='-C target-feature=+crt-static' cargo build --locked --release \
  --manifest-path agent/Cargo.toml --target x86_64-unknown-linux-gnu \
  --package beachball-guest

Also run uv sync --frozen --extra dev and uv run pytest -q in surf/guest. The web build writes surf/control-plane/public; it must finish before deploying Surf. Preserve the built gateway with the release checkout at /opt/beachball/releases/REVISION, owned by root. Point /opt/beachball/current at that release when installing the gateway service.

Install and verify celld 0.6.0, record its SHA-256, and install it as /usr/local/bin/celld on the management machine. Do not substitute the latest celld release without checking compatibility. The local .tools/bin/celld can be a build input after verifying its version and provenance.

Publish one Surf image

Build surf/image/Dockerfile with surf/guest as the context. Push it to a registry accessible to the image builder, record its OCI digest, and scan that digest. Use the scan policy for this release; the development script contains dated vulnerability exceptions that require review.

With SURF_OCI_REF set to the digest reference, SURF_IMAGE_DIR to an absolute output directory, and the kernel, guest binary, and image-signing key paths set:

sudo agent/target/x86_64-unknown-linux-gnu/release/bb image build "$SURF_OCI_REF" \
  --kernel "$SURF_KERNEL" \
  --guest "$SURF_GUEST_BINARY" \
  --signing-key-file "$SURF_IMAGE_SIGNING_KEY" \
  --output-dir "$SURF_IMAGE_DIR" \
  --name surf-linux-x64 --wait-for-bootstrap --user surf --workdir /workspace \
  --no-input

The image key is 32 raw Ed25519 bytes or base64, kept in a mode-0600 file. It is different from the PEM release-signing key and the scheduler connection key. Use a pinned, verified Firecracker-compatible kernel; the repository's current kernel pin is in agent/scripts/prepare-ubuntu-24.04.sh. That script builds an Ubuntu image, so do not use its root filesystem as the Surf image.

bb image build emits a signed manifest containing local file URLs. Before registering it for remote hosts:

1. Copy each kernel, rootfs, and initrd artifact to the HTTPS image server under an immutable path such as /blobs/ARTIFACT_SHA256. Preserve its bytes.

2. In a copy of the generated manifest, change each artifact's url to the corresponding https://images.example.com/blobs/ARTIFACT_SHA256. Leave size and SHA-256 unchanged. Set signature to an empty string.

3. Sign the edited manifest and record the output digest and public key:

   bb image sign "$SURF_UNSIGNED_MANIFEST" "$SURF_SIGNED_MANIFEST" \
     --signing-key-file "$SURF_IMAGE_SIGNING_KEY" --json

4. Publish the exact signed bytes as /manifests/MANIFEST_SHA256.json, after the blobs are available. Use surf-linux-x64@sha256:MANIFEST_SHA256 as the workload class's imageRef. Changing a URL changes this digest.

5. Verify HTTPS downloads and hashes from a VM host. Keep allow_file_images disabled. The image downloader has no HTTP credential-header configuration; restrict access through networking and use stable URLs.

The nginx template serves /srv/beachball-images for this purpose. A CDN or object-store HTTPS origin can serve the same layout. Never upload private keys or runtime credentials with the image files.

Package the host runtime

Complete the scheduler manifest in the next section, then fill release-metadata.example.json with the actual commit, versions, manifest hash, and runtime/image hashes. Install cargo-audit, cargo-cyclonedx, and the native package tools required by agent/scripts/build-native-packages.sh. Generate the report and package:

cargo audit --file agent/Cargo.lock --format json > scheduler/dist/cargo-audit.json
BEACHBALL_RELEASE_METADATA_FILE="$SURF_RELEASE_METADATA" \
BEACHBALL_RELEASE_SIGNING_KEY_FILE="$SURF_RELEASE_SIGNING_KEY" \
BEACHBALL_DEPLOYMENT_MANIFEST_FILE="$SURF_SCHEDULER_MANIFEST" \
  bash agent/scripts/package.sh x86_64-unknown-linux-gnu

All three paths must be absolute; the signing key is an Ed25519 PEM file with mode 0600. Resolve audit failures before packaging. Follow package signature verification and installation on every host and on the gateway machine, which needs bb but does not enroll as a host. Packaging is not evidence that the live deployment passed the canary.

3. Prepare configuration and secrets

Copy these templates to a private configuration directory outside the checkout:

Replace all placeholders, dates, domains, and example identities. For the scheduler manifest:

• Keep admissionsEnabled: false for initial setup.
• Use one controller, for example tenant internal-team, client surf, and allowedWorkloadClasses: ["surf-linux-x64-medium"].
• Use one matching class, with allowedClients: ["surf"], the published imageRef, privateIngress: true, allowedNetworks: ["default"], and requiredHostLabels: ["standard"]. Register hosts with --label standard; the current scheduler requires at least one host label in a class.
• Give the class maxCpuMillis, maxMemoryMiB and maxEphemeralDiskMiB at least as large as Surf's beachball.machine.limits (maxVcpus × 1000 for CPU), so every machine size an administrator saves fits the class. Without them the class's maximums equal its own size and larger sessions are refused with RESOURCES_EXCEED_CLASS.
• Set allowedRegions to the host region, and use that region for the Surf gateway in both manifests and the gateway config.
• Leave Surf's beachball.networkId as null. The scheduler creates the controller tenant's default network on its first allocation.
• Set networking.relay.required: false and networking.relay.relays: [].
• Bound controller and deployment allocation limits to the planned capacity. Keep maxActiveHosts above the enrolled pool size: current scheduler readiness returns ACTIVE_HOST_LIMIT at equality. This limit is not a desired host count.
• Fill runtime metadata, retention, backup, audit, alert, and runbook settings.

Generate independent random bearer tokens for the scheduler operator, Surf controller, Surf operator, and gateway. Hash the trimmed token bytes with SHA-256 for each manifest's sha256; do not hash a trailing newline. Use actual creation times and finite expiry dates, and record when to rotate. The controller token is shared by Surf and the gateway's bb profile, so both act as the same scheduler client.

People sign in to Surf with GitHub (spec section 5.1a):

• In the GitHub App Surf uses (or a separate OAuth App), add the callback URL https://surf.example.com/api/auth/callback/github, grant the Email addresses account permission (read), and generate a client secret. Set auth.publicUrl to Surf's URL and auth.github to the App's client ID and SURF_GITHUB_CLIENT_SECRET.
• Generate SURF_AUTH_SECRET with openssl rand -base64 48. It signs session cookies and encrypts stored OAuth tokens; rotating it signs everyone out.
• List who may sign in in auth.allow (github:<login>, email:*@example.com) and the administrators in auth.administrators. Give each manifest user identities: ["github:<login>"] instead of a token.
• Sign-in state lives in the surf-auth D1 database of Surf's fleet bucket; back it up with the bucket. The Worker creates its tables on first use.
• For a script or CLI without a browser, an administrator operator mints an API key: POST /v1/users/{userId}/api-keys with { "name": "…" }.

To move an existing deployment from user tokens, add auth with legacyTokens: true and the users' identities, let everyone sign in once, then set legacyTokens: false and remove the users' sha256 fields.

Generate the scheduler's root secret with openssl rand -base64 32. Every scheduler secret derives from it: internal authentication, the credential issuer, the bootstrap credential, and the Ed25519 connection-signing key. Print the matching connection verification key for the gateway bb profile and each host's connection-verification key:

node --input-type=module -e '
  import { readFileSync } from "node:fs";
  import { connectionVerificationKey } from "./scripts/lib/derived-secrets.mjs";
  const { keyId, x } = connectionVerificationKey(
    readFileSync(process.argv[1], "utf8").trim());
  console.log(`key_id = "${keyId}"\npublic_key = "${x}"`);
' /path/to/scheduler/root.secret

These are not the image-signing keys.

celld 0.5 and later have no variables file or secret store. Each app's Worker variables are written into a private copy of its Wrangler config and uploaded with celld deploy, so they are stored in that app's deployment record in its bucket. Bucket access already controls the app's state, so this does not widen who can act as it. Keep the variables files readable only by the operator who deploys:

Use the exact secret-reference names in your manifests. Each value must occupy one KEY=value line. The Surf script warns about any referenced secret that the file does not set. The API-key template runs opencode; select a real model and the provider's matching environment variable in agent.credentialReferences.

For example, prepare surf.json and secret files named by the table under secrets/scheduler/ and secrets/surf/. This writes new vars files beside the manifests and refuses to overwrite existing files:

python3 - "$SURF_CONFIG_DIR" <<'PY'
import json
import os
import pathlib
import sys

root = pathlib.Path(sys.argv[1])
for app, names, manifest in [
    ("scheduler", ["AUDIT_EXPORT_TOKEN"], None),
    ("surf", ["INTERNAL_OBJECT_SECRET", "BEACHBALL_CLIENT_TOKEN",
              "SURF_GIT_TOKEN", "SURF_MODEL_API_KEY", "SURF_AUTH_SECRET",
              "SURF_GITHUB_CLIENT_SECRET"], "surf.json"),
]:
    values = {}
    if manifest:
        values["SURF_MANIFEST"] = json.dumps(
            json.loads((root / manifest).read_text()), separators=(",", ":"))
    for name in names:
        value = (root / "secrets" / app / name).read_text().strip()
        if not value or "\n" in value or "\r" in value:
            raise ValueError(f"{app}/{name} must contain one nonempty value")
        values[name] = value
    fd = os.open(root / f"{app}.vars", os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    with os.fdopen(fd, "w") as output:
        output.write("".join(f"{key}={value}\n" for key, value in values.items()))
PY

4. Install the management services

Create system users beachball-scheduler, beachball-surf, and surf-gateway, each with its own group and no interactive login. Create /etc/beachball/ subdirectories scheduler, surf, and gateway, owned by root with their respective service groups and mode 0750. Install:

• runtime.env: root-owned, mode 0600; systemd reads it.
• The scheduler's root secret, manifest, and scheduler.vars, and Surf's surf.vars: readable only by the operator who deploys, mode 0600. They are uploaded with each deployment rather than read by the services.
• Gateway JSON, bb profile, gateway.token, and beachball-client.token: owned by surf-gateway, mode 0600. The bb credential reader checks ownership.
• celld template unit and gateway unit: root-owned under /etc/systemd/system/, mode 0644.

Confirm /usr/bin/node, /usr/bin/bb, /usr/local/bin/celld, and the gateway release path exist. The units create private persistent state directories under /var/lib/. The gateway needs no Linux capabilities.

Install the nginx template, using your names and certificate paths. Apply the network restrictions from the layout section, including access for the management machine itself. Verify with nginx -t before reloading nginx. The proxy settings preserve long polls and streamed responses; see nginx proxy documentation.

Deploy from the release checkout using the storage credentials for each app. In each command, CELLD_BUCKET, S3_ENDPOINT, AWS_REGION, and AWS credentials must identify that app's bucket. Supply them through your secret system; the systemd environment files are not shell scripts to source.

# Scheduler storage environment
CELLD_ESBUILD="$PWD/scheduler/node_modules/.bin/esbuild" \
  /usr/local/bin/celld diagnose --bucket "$CELLD_BUCKET" \
    --endpoint "$S3_ENDPOINT" --region "$AWS_REGION"
# The root secret, manifest, and referenced secrets travel in the deployed
# config's `vars`.
deploy_config="$(node scripts/scheduler-deploy-config.mjs \
  --secret-file /path/to/scheduler/root.secret \
  --manifest /path/to/scheduler/manifest.json \
  --vars-file /path/to/scheduler.vars)"
CELLD_ESBUILD="$PWD/scheduler/node_modules/.bin/esbuild" \
  /usr/local/bin/celld deploy "$deploy_config" \
    --bucket "$CELLD_BUCKET" --endpoint "$S3_ENDPOINT" --region "$AWS_REGION"
rm -f -- "$deploy_config"

# Switch to the Surf storage environment before these commands.
/usr/local/bin/celld diagnose --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" --region "$AWS_REGION"
deploy_config="$(node surf/scripts/deploy-config.mjs \
  --vars-file /path/to/surf.vars)"
CELLD_ESBUILD="$PWD/surf/control-plane/node_modules/.bin/esbuild" \
  /usr/local/bin/celld deploy "$deploy_config" \
    --bucket "$CELLD_BUCKET" --endpoint "$S3_ENDPOINT" --region "$AWS_REGION"
rm -f -- "$deploy_config"

sudo systemctl daemon-reload
sudo systemctl enable --now beachball-celld@scheduler beachball-celld@surf
sudo systemctl enable --now surf-gateway

diagnose may report no deployed application or active nodes on an empty bucket; its storage checks must pass. Repeat it after startup. Inspect logs:

sudo journalctl -u beachball-celld@scheduler -u beachball-celld@surf \
  -u surf-gateway --since '10 minutes ago'
curl --fail https://scheduler.example.com/v1/health/live
curl --fail https://surf.example.com/v1/health/live

Readiness is expected to fail while admissions are disabled. Do not use /v1/health/ready to remove the scheduler from proxy routing: operators need access while admissions are closed. Use /.well-known/celld/health for runtime health, and monitor application readiness separately.

5. Enroll one canary host

Install the verified host package. Create a scheduler operator bb profile with audience = "operator", the scheduler HTTPS endpoint, and the operator token file. From that profile, issue a short-lived, host-bound token:

bb --profile operator host enrollment-token create host-01 \
  --expires-at "$SURF_ENROLLMENT_EXPIRES_AT" --reason 'initial Surf host' \
  --token-output "$SURF_ENROLLMENT_TOKEN_FILE"

Transfer the token securely to the host. Follow host setup, using the actual capacity, image URL and public key, ULA prefix, management blocks, and connection-verification key. Set --connectivity-listen-endpoint PRIVATE_HOST_IP:51820 to a specific reachable address. Direct private networking does not require a candidate-discovery server. Do not advertise 0.0.0.0 as the reachable address.

Run bb agent doctor, start beachball-agent.service, and inspect its logs. Enrollment starts the host draining. When healthy, activate only this host:

bb --profile operator host activate host-01 --reason 'Surf canary'

Set scheduler admissionsEnabled to true in a new reviewed manifest, with a small allocation limit and only the canary user enabled in Surf. Worker variables live in the deployment, so redeploy each changed app with its deploy-config script as in section 4, then restart its unit:

sudo systemctl restart beachball-celld@scheduler beachball-celld@surf
curl --fail https://scheduler.example.com/v1/health/ready
curl --fail https://surf.example.com/v1/health/ready

Host activation and admission must precede an ordinary canary allocation. A draining host or a scheduler with admissions disabled cannot run that test.

6. Pass the launch checks

Run this against the intended host kernel, private network, and model provider:

1. Sign in with GitHub as the canary user, select an allowed repository, and submit a small real task. Verify clone, VM boot, model output, and timeline streaming.
2. Ask the agent to commit and push a test branch. Verify the branch in the remote repository before releasing the VM.
3. Submit a follow-up prompt in the same live session. Then exercise stop and release. Confirm allocation cleanup and restored host capacity.
4. Restart the gateway during a disposable task. Check tunnel recovery and timeline continuity. Kill a disposable VM and verify an explicit failure, cleanup, and the ability to start another session.
5. Restart each celld service. Verify existing session records, fresh gateway polling, and new session creation. A management outage can end a running workload when leases expire; do not expect workspace recovery.
6. Verify rejected credentials, a GitHub account outside auth.allow being turned away, access restrictions, audit delivery, alert delivery, and host disk/capacity monitoring. Check CPU, RAM, disk, and prompt latency at the intended concurrency.
7. Back up the configuration, keys, and both celld fleets using the chosen provider's recovery procedure. Restore into isolation with admissions off and production hosts unreachable. Verify records and record recovery time.

Enable the remaining user identities, enroll and activate the remaining hosts, and raise the reviewed concurrency limit only after these checks pass. Keep a record of the revision, image digest, package signatures, checks, and operator.

The existing local E2E report does not establish these results for this deployment. The broader MPR verification and release evidence describe the full production qualification; several runners still require infrastructure hooks.

Portals (optional)

Portals serve a session's sandbox port at a public URL, like Amp's. They run on Beachball public Services and an edge (ADR 0005). They need a separate registrable domain for portals, never a subdomain of Surf's own, and a small edge node with a public address.

1. Scheduler manifest. Add the edge and the domain:

   • edges: [{edgeId, domains: ["portals.example.com"], sha256, …}];
   • edgeDomains: [{domain, tenantId: "<Surf's tenant>", access: {loginUrl, keys, identityHeaderPrefix: "x-surf-"}, publicWarning: true}];
   • publicIngress: true on the Surf workload class.

   loginUrl is https://<surf>/portal-login. keys holds the base64 public half of Surf's portal signing key.

2. Surf manifest. Add portals: {domain, keyId, signingKeyReference: "SURF_PORTAL_SIGNING_KEY"}. Put the Ed25519 PKCS#8 private key, base64 without PEM armor, in that secret.

3. Edge node.

   • Install bb, then deploy/beachball-edge.service, deploy/edge.example.toml as /etc/beachball/edge.toml, and deploy/edge-bb.example.toml as /etc/beachball/edge-bb.toml.
   • Run bb edge dns-sync once to point *.<domain> at the node.
   • Start the service. It obtains the wildcard certificate through ACME DNS-01 on first start.
   • Run bb relay on the same node, and list it in the relay map. Hosts behind NAT then reach the edge without inbound ports.

4. Public Suffix List. Submit the portal domain, so portals cannot share cookies with each other. Until it lands, the edge's cookies are host-only.

Check: open a session, ask the agent to start a dev server and share a portal (surf-portal open <port>), and open the link. You should go through /portal-login and see the app.

Updates and recovery

Deploy Worker code with celld deploy from a tested checkout. Rebuild web assets before the Surf deploy. Update gateway code by switching the release path and restarting surf-gateway. Change a manifest or secret by regenerating the deploy config, redeploying, and restarting the matching beachball-celld@ unit; celld 0.5 and later have no variables file to reload.

Before a disruptive update, disable admissions and let active work finish or save its commits. Drain each host and verify no placements remain before an agent upgrade. Keep the prior signed image available for new allocations if the new image fails.

Roll back application code only when it understands the current stored schema. The Surf API cleanup removes schema columns, so do not assume a build from before that change can read a new database. Never restore a backup over a live fleet. A backup destination in the manifest does not create a backup job.

If measured availability or write latency requires another management machine, add a second node to each celld fleet, using each fleet's existing bucket and reachable private peer addresses. Move the internal listeners off loopback only onto a trusted network. Two application processes on the same machine do not provide redundancy for either fleet. Recheck gateway recovery and routing as part of that change.