Execution

Every run executes inside an isolated, containerized environment seeded with a fresh git repository. Isolation protects the host, keeps runs from discovering each other’s work, and prevents models from finding solutions in places they should not be looking.

Containerization

Runs must occur in a container so that a model cannot access the host system. Without this, a model could discover other runs’ outputs or damage the host, for example by deleting files.

The testing harness must drive the run container through a runtime abstraction rather than hard-coding a single runtime. Two runtimes implement it: a Docker/Podman runtime that shells out to a host container engine, and the Kubernetes runtime that creates a pod per run through the Kubernetes API. Every test-case run is now driven by the driver under the Kubernetes runtime — the CLI and desktop app enqueue runs at the backend rather than executing them on the host (see Server-side Run Topology), so the host Docker/Podman path is no longer how a test case runs. The behavior below is identical across both runtimes — only the mechanism that starts the container, copies the working tree in and out, and runs commands in it differs.
A container must not have access to the host filesystem beyond the seeded repository and the inputs the run explicitly provides.
A container does require outbound network access so the agent harness can reach model APIs and install packages. Isolation is about protecting the host filesystem and other runs’ outputs, not about disabling the network.
When an asset-generation run is being watched (the driver supplies a preview sink), the container is additionally given a route back to the run host as host.docker.internal — --add-host …:host-gateway under the Docker/Podman runtime, a pod hostAlias to the driver pod’s own IP under the Kubernetes runtime — so the in-container drawing or voxel binary can stream its live preview back to a listener on the run host. No host mapping is added for an unwatched run.
A run executes in one of five run-container images, selected by the test case’s test type and — for asset-generation — its asset_kind: an end-to-end run uses the base-wasm image (the base plus the shared Rust → WebAssembly toolchain); a single-sprite asset-generation run (asset_kind = "sprite") uses the sprite image (the base image plus the baked-in draw tool); a sprite-sheet run (asset_kind = "sprite-sheet") uses the sprite-sheet image (the base image plus the baked-in draw-sheet tool); a static-voxel run (asset_kind = "voxel-model") uses the voxel image (the base image plus the baked-in voxel tool); and an animated-voxel run (asset_kind = "voxel-animation") uses the voxel-animation image (the base image plus the baked-in voxel-anim tool). None is a per-harness image: the selected harness’s CLI is installed into the container at run time (see Harness install below), not baked into the image. All five are registry images, and a runner resolves the one for the run from its own registry configuration — TCAB_CONTAINER_REGISTRY (default ghcr.io/theclockwyrks) and TCAB_CONTAINER_TAG (default latest) select the image named for the run (test-cabinet-base-wasm, test-cabinet-sprite, test-cabinet-sprite-sheet, test-cabinet-voxel, or test-cabinet-voxel-animation), and a per-image override pins a verbatim reference for one image without touching the others — one TCAB_CONTAINER_IMAGE_* per run image (_BASE_WASM for end-to-end, _SPRITE for single-sprite, _SPRITE_SHEET for sprite-sheet, and so on through every asset-generation kind — including _BLENDER for blender-character — plus _ADVERSARIAL and _PERFORMANCE; there is no override that spans every image, since they differ) — and pulls it at run start (--pull missing). No backend is consulted, so a runner resolves the image the same way against any backend or none. Whatever image actually runs is resolved to its registry digest where it has one and recorded in the run record, so a run still pins the exact image bytes it used even when launched by a mutable tag.

Seeding

Each run must be seeded into its own newly created git repository that contains the data a model needs to build the game: the selected variant’s workspace starter files, the specs of the selected variant, the test case’s assets, and the rendered reference screenshots that serve as visual targets. A run selects exactly one variant, and the variant’s specs are seeded at their declared dest paths — the common specs plus that variant’s own — rather than as a single specification at the repository root.

A new repository must be created per run so that no prior history exists. Models have been observed solving tasks by reading git history to recover a deleted reference implementation; starting from an empty history removes that possibility.
The seeded repository must begin from a clean initial commit with no upstream remote and no history beyond that commit.
The selected variant’s workspace starter files are seeded into the repository root first, before the specs, so the specification and reference screenshots land on top of a baseline project. They are copied verbatim. Resolution rejects any collision between a workspace file and a spec, asset, or reference destination, so seeding never silently clobbers one with another.
A spec whose source is a Handlebars template (a .hbs extension) is rendered with the selected variant and version while seeding, and the result lands at the spec’s dest; every other spec is copied verbatim. This lets a spec state per-variant facts directly instead of hedging about what a run might contain. See Spec templates.
A test case’s reference screenshots are seeded as visual targets so the model can see what each screen should look like. The reference source mockups are not seeded: handing over the mockup HTML/CSS would let a model copy the intended UI instead of building it from the specification, the same kind of shortcut the fresh repository is meant to prevent. A screenshot conveys the target without giving away the implementation.
The seeded specs must be self-contained, with no links or references to these harness docs or to any file outside the seeded repository, because none of them exist inside the container. They may, however, point at the seeded reference screenshots. See Test Cases.
The prompt is not seeded to disk. It is rendered from the version’s prompt.hbs template — with the run’s in-container workspace path and the selected variant’s seeded spec paths — and handed directly to the harness as its instruction. See Prompt template.

The seeded repository is normally created on the host, copied into the run container, and torn down as part of a run, so its contents are never visible on their own. The tcab seed command runs this same seeding step for a chosen variant (--variant) and leaves the result on disk (under tmp/ by default) so the exact inputs a harness receives — the variant’s seeded specs, the seeded assets, and the fresh git history — can be inspected without launching a container. Because the prompt is not seeded, tcab prompt renders and prints the instruction a run would hand the harness for a given variant, without seeding or launching anything.

Harness install

The base image ships no agent harness. Once the container starts, the run installs the selected harness’s CLI into it by running that harness’s install command — typically a single-line npm install -g … or a curl-piped installer. Installing at run time, rather than baking the CLI into an image, is what lets a run always pick up the harness’s most recently published version.

The install step runs after the container starts and before the test case’s init command and the harness session, so the CLI is in place for both.
It runs through a non-login sh -c as the container’s unprivileged run user, with the container’s own environment, so it installs into the user-writable locations the base image puts on PATH without needing root.
It is bounded by the run’s maximum runtime, the same cap that bounds the harness session, so a hung install can never run unbounded.
A non-zero exit, a timeout, or a missing binary afterward (the run probes <binary> --version to confirm the install worked and to record the version) aborts the run before a harness session is spent and tears the container down, with the captured output surfaced for diagnosis.

Init

A test case may declare an init command that runs inside the run container once the seeded repository is mounted and the harness CLI is installed, and before the harness session begins. It is where a case prepares the workspace it shipped — typically installing its dependencies — so the harness starts against a ready project. It runs as the container’s unprivileged run user with the seeded repository as its working directory.

The init step runs after the container starts, the workspace is mounted, and the harness is installed, and before the harness is invoked, so anything it installs is in place for the model.
It is bounded by the run’s maximum runtime, the same cap that bounds the harness session, so a hung setup can never run unbounded.
A non-zero exit or a timeout aborts the run before the harness starts and tears the container down — a broken setup would only waste a harness session — with the captured output surfaced for diagnosis.
Because init needs a running container, it is not performed by tcab seed, which only materializes the seeded files on disk.

Runtime limits

Two independent bounds end a run that will not end itself, and between them they guarantee that a run’s fate is always decided by the Test Cabinet, never by the platform underneath it.

The maximum runtime is the wall-clock cap on the run as a whole: the test case’s max_runtime_hours, overridable per invocation (tcab run --max-runtime). A run stopped by it is timed_out — the model was working and simply never converged, so its source and build are kept.
The idle watchdog bounds silence rather than duration. A harness that produces no output at all for 30 minutes is killed and the run is recorded as hung. This catches the failure the runtime cap cannot: a harness that has stopped doing anything — a stalled provider request, a subagent that never returns — is not “still working”, and waiting out the remainder of an eight-hour cap to discover that wastes the slot for hours.

The watchdog’s window is deliberately far below the platform’s own limits. A Kubernetes kubelet closes an exec stream that has been idle for streamingConnectionIdleTimeout — 4 hours by default, and 4 hours on our clusters. Before the watchdog existed, that limit is what actually ended a hung run: the stream closed without a terminating status frame, the exec reported exit code -1, and the run was misrecorded as a harness error four hours after it had in fact stopped. It also silently capped every case, because a run could never survive long enough to reach a max_runtime_hours above 4.

Because the watchdog always fires first, both problems go away together: a hang is attributed accurately and promptly, and a case’s maximum runtime is reachable however long it is set — a run that keeps producing output is never interrupted by anything but its own cap.

Model Authored Tests

The goal of a test case is to measure how well a model writes code in a large project, so the testing harness must not get in the way of the model testing its own work.

Any tests a test case provides must be visible to the model.
The model must not be blocked from writing its own tests.

Artifact Collection

When a run finishes, the testing harness must collect the run’s working tree as the run’s primary artifact. This produced repository is what gets validated and, if published, released. See Results.