Running

This page covers running The Test Cabinet locally — on your own machine, for development or to exercise the whole flow end to end. Execution is now a single, unified path: every launcher — the CLI (tcab), the Tauri desktop app, and the web console — enqueues a run at the backend and watches it; none of them runs a test case on its own machine. So launching a run locally means standing up the service-driven stack:

The full service-driven flow — the backend (which owns the run queue), the auth service, the dispatcher, the driver, the artifact service, the arena service (adversarial matches/tournaments), and the web console, running exactly as a deployed environment runs them. A launcher enqueues a run at the backend; an in-cluster dispatcher claims it and creates a per-run Kubernetes Job running the driver, which executes that one run. Because execution is a cluster concern, the local service-driven story runs on a k3d cluster (k3s-in-Docker) from the same manifests a deployment uses. (The auth service is what lets you register, log in, and launch / review / publish; without it the read-only flow still works, but mutations are rejected 401.)
A single run from the CLI or desktop still targets that same stack. tcab run and the desktop app are thin enqueue + watch clients, so they need a reachable backend (TCAB_BACKEND_URL) and an account — not a host container runtime. The fastest local path is to bring the k3d stack up (below) and point tcab (or the desktop app) at the forwarded backend; the quickstarts walk through it and Building covers producing the binaries.
- Developing on the desktop app? Set TCAB_BACKEND_URL and it behaves as the thin client described here. Left unset, the shipped app instead stands up its own bundled k3d cluster from the published images (see Self-contained cluster): great for end users, but it ingests the bundled catalog, so during development point it at a manually-run backend you can re-ingest at will.

Running the services on one machine is the local mirror of a real deployment: the same images and the same configuration, only on a throwaway local cluster. When you are ready to put them on real hosts — staging and prod — see Deployment.

Prerequisites

A container runtime (Docker) — needed by k3d, which runs the cluster as containers. In the devcontainer this is the host’s daemon, reached over a bound socket (Docker-outside-of- Docker); on a bare host it is the local daemon. The CLI and desktop app no longer need a runtime of their own (the in-cluster driver creates the run’s sandbox via the Kubernetes API). See Execution and first-time setup.
k3d and kubectl, for the service-driven flow — both ship in the devcontainer (install them yourself on a bare host).
The harness container images built or pullable for whichever harness you intend to run.
The service binaries, built per Building: cargo build -p test-cabinet-backend, cargo build -p tcab-auth-service, cargo build -p test-cabinet-dispatcher, cargo build -p test-cabinet-driver, and cargo build -p test-cabinet-artifacts (or the build-portable-* aliases for static binaries). The web console is a Vite app under apps/web; unlike the services, it does not run in the local k3d cluster — you run its Vite dev server (npm run -w apps/web dev) from source against the forwarded backend, both for the k3d flow and the bare-process path (iterating on the UI this way needs no image rebuild). Only prod/staging serve it in-cluster as the tcab-web image.
A harness API key for the harness you will run (for example ANTHROPIC_API_KEY for claude).

The whole stack on k3d (deployment parity)

For a task-oriented walkthrough of this — bringing the stack up, connecting the console, and enqueuing a run — see the guide Running the Local Service Stack (or the quickstart for just the steps). This section is the reference the guide sits on top of.

Run execution is now a cluster concern: a run schedules as a per-run Job, so the service-driven flow runs the services the way a deployment runs them — in a real (local) Kubernetes cluster, from the same manifests. The deployments/local/Makefile drives the whole thing, and is meant to run inside the devcontainer (which ships docker, k3d, and kubectl and binds the host daemon socket in); it also works on a bare host with those three installed directly:

Before bringing the stack up, export the harness provider API key the run needs — the Makefile reads it from your environment and creates the cluster Secret from it, so no key is ever written to a tracked file:

export ANTHROPIC_API_KEY=…   # for the `claude` harness (or OPENAI_API_KEY for
                             # codex, OPENROUTER_API_KEY for cline/goose/kilo/…)

make -C deployments/local local-up        # create cluster, build+load images, apply secrets+overlay, ingest
make -C deployments/local local-forward   # hold backend→:8787, auth→:8789, artifacts→:8790, arena→:8791, Grafana→:3000 open on localhost (run the console from source, below)
# … develop …
make -C deployments/local local-rebuild   # after a code change: rebuild images + restart
make -C deployments/local local-status    # show the namespace's pods, Jobs, and services
make -C deployments/local local-ingest    # force re-ingest the catalog after editing a case
make -C deployments/local secrets         # re-create the Secrets from the environment (after rotating a key)
make -C deployments/local local-down      # delete the cluster and everything in it

local-up creates a throwaway k3d cluster, builds the backend, auth, dispatcher, driver, artifact, and arena images from deployments/images/, loads them with k3d image import (no registry needed), creates the cluster Secrets from your environment (the harness key above plus a dev service token), applies the deployments/k8s/overlays/local kustomize overlay, and force-ingests the catalog from a read-only mount of this repository. The web console is the one piece that does not run in-cluster locally — you run it from source (below) so a UI edit hot-reloads instead of forcing a full image rebuild + re-import (prod still serves it in-cluster). The dispatcher and driver run in-cluster under their own ServiceAccounts, so a run you enqueue at the backend schedules as a Job in this same cluster — exactly as a cloud deployment runs it. The host no longer runs any worker process.

make local-forward holds the backend on 127.0.0.1:8787, the auth service on 127.0.0.1:8789, the artifact service on 127.0.0.1:8790, the arena service on 127.0.0.1:8791, and — for convenience, since this is the session you keep running anyway — Grafana on 127.0.0.1:3000. Then start the console from source in a separate terminal — npm run -w apps/web dev — and open http://127.0.0.1:1430. Its backend/auth URLs are pre-set to those forwarded addresses via the committed apps/web/.env.development, so there is nothing to configure and no VITE_BACKEND_URL to pass (the backend/auth CORS layers are permissive, so the dev server’s cross-origin requests are allowed). The forwards are needed because the browser runs outside the cluster: it loads the console, and reaches the backend (which the in-cluster dispatcher and driver drain), the artifact service (each run’s build + proof/asset media, as <img>/<iframe> requests), and the arena (adversarial matches/tournaments — the backend reports its URL at GET /config) over these same forwards. tcab run and the desktop app target the same forwarded backend: point tcab at it with TCAB_BACKEND_URL=http://127.0.0.1:8787 (and tcab login first); the desktop app uses the same backend URL in its Connections settings. After editing a test case, re-ingest with make -C deployments/local local-ingest.

Pointing `tcab` at a deployment

tcab is a thin enqueue-and-watch client, so the only difference between the local k3d stack and a remote staging/prod deployment is the URL you point it at and logging in with an account. There are two ways to reach a remote backend:

Over the VPN, at the private hostnames (the prod path). A deployment with the internal ingress serves the backend and auth service at private *.testcabinet.ai hostnames, reachable on the company VPN. Point tcab straight at them — no port-forward needed:
Terminal window
```
export TCAB_BACKEND_URL=https://api.tcab.testcabinet.ai
export TCAB_AUTH_URL=https://auth.tcab.testcabinet.ai
tcab login --username <name>     # authenticate against the auth service
tcab run --test-case carom --version v1.0.0 --variant base \
  --harness claude --model claude-opus-4-8   # enqueue + watch, exactly as locally
```
These hostnames resolve only on the VPN, via the cloud’s private DNS — they are not public. The backend reports the artifact and arena URLs (https://artifacts.tcab.testcabinet.ai / https://arena.tcab.testcabinet.ai) at GET /config, so media and arena views resolve over the same VPN.
kubectl port-forward (off-VPN fallback / debugging, or before the ingress is up). Forward the backend (and auth) ClusterIP services to localhost and point tcab at the forwarded ports:
Terminal window
```
kubectl -n tcab-prod port-forward svc/tcab-backend 8787:8787 &
kubectl -n tcab-prod port-forward svc/tcab-auth    8789:8789 &
export TCAB_BACKEND_URL=http://127.0.0.1:8787
export TCAB_AUTH_URL=http://127.0.0.1:8789
tcab login --username <name>
tcab run --test-case carom --version v1.0.0 --variant base \
  --harness claude --model claude-opus-4-8
```
Note that artifact/arena media still resolves to whatever the backend advertises at GET /config; if those TCAB_*_PUBLIC_URLs point at the private ingress hostnames, you also need the VPN (or matching forwards) for media to load.

Iterating on the backend and auth services as bare processes

You can run the backend and auth services as ordinary host processes — the quickest way to iterate on those two binaries — but note that run execution still requires the dispatcher and driver, i.e. a cluster (the k3d stack above or a remote one). The bare-process path below stands up the two stateful services and the console for read/review work; to actually launch a run, point the console at a backend whose queue an in-cluster dispatcher is draining.

Configure the services

Copy the repo-root example env files and fill them in. These remain the authoritative list of every variable each service reads.

cp .env.backend.example .env.backend

In .env.backend, the only required value is the checkout the backend ingests definitions from — point it at this repository:

TCAB_BACKEND_CHECKOUT=/absolute/path/to/the-test-cabinet
# Leave TCAB_BACKEND_BIND at its default 127.0.0.1:8787 for local use.
# Leave TCAB_BACKEND_DATABASE_URL unset to use the default local SQLite file.
# Leave TCAB_BACKEND_AUTH_URL at its default http://127.0.0.1:8789 so the backend
# verifies bearer tokens against the local auth service.
# R2 + deploy-hook variables can stay blank: with them unset the backend still
# records to its database and regenerates the snapshot on disk (a dev-only mode).

The dispatcher and artifact service take their own env — see .env.dispatcher.example and .env.artifacts.example for the full lists — but they assume the cluster context the k3d overlay wires up (the dispatcher’s Kubernetes API access, the driver ServiceAccount, the artifact volume), so the k3d stack is the supported way to run them.

Start the backend

Run the binary directly from a directory containing .env.backend:

./target/debug/tcab-backend

Once it is up, ingest the repository so the catalog is populated:

curl -X POST http://127.0.0.1:8787/ingest

Confirm it is serving with curl http://127.0.0.1:8787/healthz and curl http://127.0.0.1:8787/test-cases.

After you edit a test case, re-ingest so the backend serves the change. A plain scan skips any version it already holds (the store is immutable per (slug, version)), so force the overwrite — optionally scoping it to the case you touched. The scripts/reingest.sh helper forces the overwrite and streams its per-case progress (a full re-render takes a minute or more). By default it only re-ingests cases whose files changed since its last successful run — it records that baseline in a gitignored .reingest-timestamp marker — so a repeat call after touching one case renders just that case:

scripts/reingest.sh             # re-ingest only cases changed since the last run
scripts/reingest.sh carom        # scope to one case (still skipped if unchanged)
scripts/reingest.sh --force      # re-ingest every case, ignoring change detection

The first run (or after rm .reingest-timestamp) has no baseline and re-ingests everything. Use --force to re-render regardless of what changed.

It is a thin wrapper over the endpoint’s streamed (Accept: application/x-ndjson) progress feed; the raw call is:

curl -X POST http://127.0.0.1:8787/ingest \
  -H 'content-type: application/json' \
  -d '{"testCases": ["carom"], "force": true}'

Backend-driven runs (the desktop and web consoles) resolve their definition from the backend, so without a re-ingest they keep running the previous definition — a newly added spec, proof, or prompt change silently does not reach the model, and new manifest fields read back empty. (tcab validate against a local checkout reads the repository directly and is not affected.)

Forced re-ingest overwrites the stored version in place and is a development-only convenience for iterating on a version no run has been published against. Once a published run references a version it is immutable — revise by creating a new version instead, never by editing and re-ingesting the published one (see Test Cases).

Start the auth service

So you can register, log in, and review/publish, start the auth service. It takes its own bind address and its own database, separate from the backend’s:

TCAB_AUTH_BIND=127.0.0.1:8789 \
TCAB_AUTH_DATABASE_URL=sqlite://./tcab-auth.db?mode=rwc \
  ./target/debug/tcab-auth-service

Both default to the values shown, so a bare ./target/debug/tcab-auth-service works too. Confirm it with curl http://127.0.0.1:8789/healthz, then create an account and log in:

tcab register --username dev --display-name "Dev"

The backend (pointed at it by TCAB_BACKEND_AUTH_URL) now verifies the token the CLI stored, so mutations are accepted. Without the auth service running, reads still work but review/publish are rejected 401.

Start the web console

Run the console’s dev server and open it in a browser:

npm run -w apps/web dev

The console defaults its backend to http://127.0.0.1:8787 and its auth service to http://127.0.0.1:8789 — the forwarded local-stack addresses, pre-set in the committed apps/web/.env.development, so there is nothing to configure. (To aim it elsewhere, set the backend in the UI or override VITE_BACKEND_URL in a gitignored .env.local.) The console enqueues a run by posting it to the backend’s queue; the in-cluster dispatcher claims it, the driver Job executes it, and the console watches its event stream live and reads the produced build and media from the artifact service (the backend reports its public URL to the console via GET /config). There is no worker to register.

Telemetry (optional)

To watch traces across tcab-backend → tcab-dispatcher → tcab-driver locally, the Grafana LGTM stack runs in the cluster (the local overlay’s components/observability), so make -C deployments/local local-up already wires every in-cluster service to it — Grafana is exposed at http://127.0.0.1:3000 by make -C deployments/local local-forward (the session you already keep running); make -C deployments/local local-grafana is only needed when you also want the OTLP collector ports forwarded (to export from a process run outside the cluster). That is fully described under Observability — in particular the endpoint-duality rule: in-cluster pods use http://tcab-lgtm:4318, while a process run outside the cluster uses http://localhost:4318 via that port-forward. Leaving OTEL_EXPORTER_OTLP_ENDPOINT unset keeps everything on plain stdout logging.

When this works end to end, the same service images deploy unchanged to staging and prod on Kubernetes — what changes is the namespace they live in, not how the flow is wired. A run is a per-run Job everywhere: the dispatcher claims a queued run and creates a Job running the driver, which (under the Kubernetes runtime) creates one ephemeral sandbox pod per run. See Deployment for the remote build.