Architecture
The Test Cabinet is built as a headless core with a set of components layered on top of it. The core owns all of the orchestration — resolving a test case version, seeding a run’s repository, executing the run in a container, invoking the agent harness, collecting metrics, running validation, writing the run record, and publishing — and every other component is a thin wrapper that exposes that functionality under whatever interface it is expected to provide (a CLI, an HTTP API, a desktop GUI, and so on).
Keeping orchestration in the core and out of the interfaces is what makes batch runs, automation, unattended sweeps, and remote execution possible: any component can drive a run because none of them re-implement what a run is.
Components
Section titled “Components”The Test Cabinet is made up of the following components.
| Component | What it is |
|---|---|
| Core | The Rust library that implements ~95% of the functionality. Everything else wraps it. |
| CLI | The tcab binary. Exposes the core so runs can be scripted and swept in batch. |
| Dispatcher | A thin controller that claims queued runs from the backend and creates one driver Job per run. |
| Driver | The per-run executor: it runs exactly one test case in a Job, streams its progress to the backend, and exits. |
| Artifacts | A data-plane service that serves produced run trees (playable builds, proof/asset media) off a persistent volume. |
| Tauri app | The desktop GUI — the primary interactive way to launch runs, watch them live, review them, and publish. It enqueues runs at the backend like the web console. |
| Web console | The same launcher/reporter console as the Tauri app, running in a browser, that enqueues runs at the backend’s run queue. |
| Backend | A private Rust server that distributes test case definitions, owns the run queue, and stores run results (stored, reviewed, and published). |
| Auth service | A small standalone Rust server for user accounts: self-registration, password login, and the bearer tokens the backend verifies. |
| Site | The public static gallery at testcabinet.ai where published runs are browsed and played. |
| UI library | Shared frontend code (@test-cabinet/ui): the full routed gallery application all three GUIs mount, the presentational primitives they render, and the backend client interfaces the Tauri and web consoles share. |
| Docs | This documentation site. |
Runners and Reporters
Section titled “Runners and Reporters”Two roles recur across the components:
- A runner is the component that actually executes a test case. There is now exactly one: the driver the dispatcher creates per run. The driver needs a container runtime (the Kubernetes API, which it uses to create an untrusted sandbox pod), resolves the requested test case version from the backend, drives the run through the core, and reports the result back to the backend. The CLI, the Tauri app, and the web console do not run test cases themselves; they enqueue a run at the backend and watch it (see Server-side Run Topology), so none of them needs a container runtime.
- A reporter is any component that displays run results: the Tauri app, the web console, and the public site. Reporters read published results; only GUI reporters let a person interact with the produced implementations.
The Tauri app and the web console are launchers and reporters in one — they
enqueue runs, watch them live, review them, and show results in one place — which
is why the Tauri app is expected to be the primary way The Test Cabinet is used.
The two consoles differ only in delivery (a desktop binary vs. a browser bundle)
and in their host wiring; both enqueue runs at the
backend, which a
dispatcher drains into per-run
driver Jobs. All three GUIs in fact mount the
same routed gallery application from the UI library;
the consoles are that app with the launch surface enabled, and the public site is
the same app with it off.
The Backend
Section titled “The Backend”Earlier versions of The Test Cabinet deliberately had no backend. Run records were committed into the site’s dataset — a “git-as-a-db” design that was chosen for convenience rather than because it was sound. That requirement has been dropped in favor of a single, centralized backend that records run results and serves as the canonical copy of the test case definitions runners need.
The backend stays deliberately small and has no public write surface; it sits on a private network, so reaching it is the first line of access control (see Backend). On top of that, real user accounts — held in a standalone auth service — identify who acts, so that every review is attributed to a person. The backend verifies the auth service’s bearer tokens on the mutating run endpoints (review, publish); reads stay open. The public site remains a fully static, backend-less deployment: publishing exports a public snapshot of the published runs that the site builds from, so the gallery has no live dependency on the private backend.
Local Operation
Section titled “Local Operation”No run executes on the machine that launched it. The CLI, the Tauri app, and the
web console all enqueue a run at the backend
and watch it; the only component that needs a container runtime is the
driver, and the runtime it needs is the Kubernetes
API. So a launcher requires nothing more than a reachable backend (and an
account) — no host Docker or Podman. For local development this means the
service-driven stack must be up: the local mirror runs the backend, dispatcher,
driver, and artifact service on a k3d cluster, and tcab run and the desktop app target that stack just as a console does in production.
See Execution and Running.
Server-side Run Topology
Section titled “Server-side Run Topology”A run launched from any of the GUIs — the CLI, the
Tauri app, or the
web console — does not execute on the launcher’s
machine. The launcher enqueues the run at the
backend, which owns a run queue; a thin
dispatcher claims the queued run and creates one
Kubernetes Job running a driver; the driver
executes the run (creating an untrusted sandbox pod via the Kubernetes API), streams
its live progress back to the backend (which relays it to the launcher), uploads the
produced tree to the artifact service, and reports
the produced record (the backend stores it privately). Each run is one schedulable
Job, so concurrency scales with
the cluster rather than with a hand-sized pool — there is no per-pod registration
and no long-lived worker. Local development runs the same manifests on
k3d, so a run is a Job everywhere. This topology replaces
the earlier worker-pool design; see Kubernetes: staging & prod.
At a high level, launching a run must:
- Select a test case version, an agent harness, and a model, resolving the version from the backend.
- Seed a fresh git repository with the selected variant’s data.
- Start a container and invoke the agent harness against the seeded repository.
- Surface the harness’s activity as a live stream of harness events while the run is in progress.
- Record metrics as the run proceeds and collect the produced repository when it finishes.
- Run validation over the produced implementation.
- Write a run record; the driver reports it to the backend, which stores it privately, leaving the run ready to be reviewed and published.
Getting a run onto the gallery is two explicit steps: review lets people
(typically not the operator) submit assessments of the produced run (its build is
playable for review off the artifact service);
and publish releases the produced code to its own public repository, deploys
its build to Cloudflare Pages, and flips the reviewed run public, which refreshes
the snapshot the site is built from. The produced record is stored privately the
moment the run finishes, so neither step is a separate “push”. The CLI’s
tcab publish collapses both for the solo case. See
Results.
Live Streaming
Section titled “Live Streaming”Some progress happens inside the run container — most visibly an asset-generation run drawing through its in-container binary — and a watched run shows that progress to the viewer in real time. Because the container’s filesystem is not host-visible mid-run and a subprocess’s stdout is mediated by the harness, the host opens a small per-run network listener that the in-container process connects back to, and relays each update to the viewer over the run’s existing live channel. This is a reusable pattern; see Live Streaming.
A Note on “Harness”
Section titled “A Note on “Harness””The word harness is used two ways throughout these docs:
- The testing harness is The Test Cabinet’s own application that runs benchmarks.
- An agent harness is a third-party coding tool (for example Claude Code or Codex) that drives a model through a test case. See Agent Harnesses.