v0.3.0 (2026-06-23)

v0.3.0 broadens The Test Cabinet in two big directions at once. Where v0.2.0 put the major components in place — a centralized backend, remote execution, and the runner/reporter consoles — v0.3.0 adds three entirely new test types beyond end-to-end (adversarial, asset generation, and performance), and it re-architects server-side run execution from a hand-scaled worker pool into a per-run-Job pipeline. Alongside those, it restructures the run lifecycle into explicit push → review → publish steps, adds real user accounts, and makes scoring numeric.

This is still pre-1.0 software intended primarily for project developers. The goal for v0.3.0 is breadth — new kinds of tests and a run pipeline that scales with the cluster — not polish.

Features

Three new test types

v0.2.0 had a single kind of test: end-to-end, where a model builds a whole playable game from a spec. v0.3.0 adds three more, each isolating a different capability. They share the run pipeline, the review lifecycle, and the gallery, but differ in what the model produces and how it is judged.

Adversarial — write a controller, then fight the field

An adversarial case asks a model to write the control logic for an actor in a game, then pits that implementation head-to-head against other models’. The model bakes intelligence into a “classical” AI controller; once written, the model itself is out of the loop and the controller is executed repeatedly against the field, with the model’s score being its controller’s record. The game’s rules, world, and authoritative state are owned by the case, never the model.

The engine and host ship as the Foray crates: foray-core (the authoritative rules engine — compiles natively and to wasm32-unknown-unknown for browser replay playback), foray-host (the reusable wasmtime host — the per-tick controller loop plus the fuel/memory sandbox, shared by the CLI and core’s AdversarialValidator), foray-cli (the foray binary), foray-controller-sdk (the controller-authoring SDK), and the baseline reference controllers. The first case is adversarial-pacman, with finished sprite art and an interpolated browser replay renderer.

A pushed run’s controller.wasm is stored on the backend, and every pushed run is automatically replayed against a fixed set of opponents (including a hidden fuel-probe) — those replays replace the old proof-of-implementation artifact. A single versioned v1 replay format (with kills) is used everywhere.

The arena service (crates/arena, the tcab-arena binary, default port 8791) is the dedicated execution host for adversarial play: quick matches (pit two controllers, get a replay back) and tournaments (run every pair in a field, streaming live per-match progress). Running matches is CPU-bound in-process wasm, so the arena is a stateless data-plane peer of the backend — it holds no database or disk, fetches every controller input from the backend over HTTP, and persists finished tournaments and replays back to it. The backend reports the arena’s public URL via GET /config; the console POSTs matches/tournaments to the arena and streams a tournament’s progress from it. Matches that end in a level-score draw are broken by total fuel (lower wins), and tournaments rank by wins.

Asset generation — drive a drawing tool toward a brief

An asset-generation case evaluates how well a model can produce a graphical asset rather than write a program. The model is given an isolated environment with a drawing binary (the draw / draw-sheet tools) that is the only way to make a mark; it produces the asset by calling the binary repeatedly, one recorded operation at a time, reading back the rendered image to see its progress, until it decides the asset is finished. The binary’s --help is the contract.

The result is deliberately subjective: the model is given a precise written brief and the freedom to draw something that matches it — there is no target image, no fidelity score, and no automated similarity check. A published run carries a human review of how convincingly the asset realizes the brief (cheat-divergence detection and the regenerate action are kept). Two asset_kinds are supported — a single sprite and a multi-frame sprite-sheet — each with its own authoring/variant guides. Asset media is served on the static gallery, and a watched run streams its live drawing previews to the run monitor through the run’s live channel.

Performance — correct first, then cheapest

A performance case evaluates not just whether a model writes working code, but how little work that code does. It targets the aspect of software development most benchmarks ignore: given two correct implementations, the one that does less work is the better engineering. Fairness comes from wasmtime’s deterministic fuel — the solution is compiled to wasm and metered, so the fuel it consumes is a reproducible function of the code and its input, identical on every host. Lower fuel is better.

A case is scored in two steps, in order: correctness first, then fuel. A fast wrong answer is no answer at all, so a solution earns a fuel result only once it is known correct. The manifest gains a [contract] (entry plus input/output schemas), a per-scenario [sandbox] (fuel_limit + max_memory_bytes), and a [[case]] set pairing each held-out input with the answer a correct solution must produce (see Manifests). core gains a PerformanceValidator that reuses the shared wasm host to run a submission once per scored case, gate on correctness, and record the consumed fuel.

The first case is Lattice (on-disk slug performance-factorio): a model writes a deterministic Factorio-style factory-simulation engine — two-lane belts with item compaction, splitters, inserters, assemblers, and source/sink fixtures — and is scored on how little fuel it spends reproducing a reference engine’s exact, bit-for-bit output. Everything is integer / fixed-point, so the state after N ticks is a single value every correct engine must agree on; correctness is checked by checksum at each snapshot — Factorio’s own desync-detection model. The case ships its authoritative engine and tooling as new crates: lattice-core (the oracle — rules, fixed-point world, canonical state + checksum; native and wasm32-unknown-unknown), lattice-host (the shared wasmtime host), lattice-cli (the lattice binary), lattice-sdk (the guest ABI glue), and two reference engines — lattice-ref-naive (the move-every-item floor) and lattice-ref-transport (the efficient baseline). Both post identical checksums; the transport engine does so for a fraction of the fuel, which is the whole point of the case.

The browser-replay renderer that visualizes a factory run is a planned enhancement, not part of v1 — a performance run’s decisive signal is correctness plus the fuel number.

Numeric scoring with rating domains

Reviews are no longer a single verdict. A run is now scored numerically from weighted review items, and its qualitative health is captured by per-domain ratings (the overall rating is the worst across domains). This gives the gallery a comparable per-run score and a separate at-a-glance health signal, and is the foundation the multi-review aggregation below builds on.

Push → review → publish

Publishing a run is no longer a single combined operation. Getting a run onto the gallery is now three explicit, separately-owned steps (see Results: Lifecycle):

• Push releases a finished run’s source (its own public GitHub repo) and playable build (Cloudflare Pages) and stores its run record on the backend without a review. A pushed run is private — it is not in the public gallery — but its build is playable, so it can be reviewed. CLI: tcab push.
• Review lets anyone — typically a different person than the operator — submit an assessment of a pushed run. CLI: tcab review <run-record> [--writeup writeup.md].
• Publish is the explicit gate that flips a run public. It is refused unless the run has at least one review, and the public snapshot (and therefore the gallery) now contains only published runs. CLI: tcab publish is the solo convenience that does push + self-review + publish in one step.

Multiple reviews per run

A run can now carry more than one review — one per account. The numbers shown for a run are aggregated across its reviews: the score is the average of its reviews’ scores, and the overall rating is the worst (lowest) across all of them (still worst-across-domains within each review, then worst-across-reviews). A single reviewer marking a domain broken pulls the run’s overall rating to broken, however generous the others were. Each writeup and its per-domain ratings are shown on the gallery attributed to the reviewer who wrote them.

User accounts and a standalone auth service

A new standalone auth service (crates/auth-service, the tcab-auth-service binary) holds real user accounts. It supports open self-registration and password login (Argon2id hashing), mints opaque bearer tokens, and exposes POST /auth/register, POST /auth/login, POST /auth/verify, and POST /auth/logout. Every review is attributed to the authenticated account that wrote it.

The backend verifies each mutating request’s bearer token against the auth service (TCAB_BACKEND_AUTH_URL, default http://127.0.0.1:8789): the mutating run endpoints (push, review, publish) now require a bearer token (401 without), while reads stay open. This is an added identity layer on top of the private network, not a replacement for it — the auth service is itself private, so open self-registration means anyone already on the network can register. The auth service binds to its own default port (8789, distinct from the backend’s 8787) and keeps its own database (TCAB_AUTH_DATABASE_URL), separate from the backend’s.

New CLI commands manage accounts: tcab register --username --display-name [--password], tcab login --username [--password] (password also via TCAB_PASSWORD), and tcab logout. The token is stored at ~/.config/tcab/credentials.json (overridable with $TCAB_CONFIG_DIR). The CLI, desktop app, and consoles reach the auth service through TCAB_AUTH_URL. See Register and Log In.

Orchestrators — pluggable execution strategies

A run’s harness sessions are now conducted by a chosen orchestrator, the data-driven strategy that decides how a run is conducted: single-session one-shot or multi-session ralph (the Ralph loop). Orchestrators are data, not code — each is a manifest plus a runner script under orchestrators/ — and the loader honors an external --orchestrator-dir, so the set is externally extensible without recompiling. See Orchestrators.

Subscription harness auth

Harnesses can now authenticate with a subscription (Claude Code, Codex, Antigravity) instead of only an API key. For backend-driven runs the operator supplies the credentials as a Kubernetes Secret; the dispatcher mounts it and the driver materializes the credential files into the run container’s filesystem (no host-filesystem read), so subscription-authenticated harnesses work in the server-side topology. See the auth quickstart.

Failures as publishable results

A run that fails no longer disappears. Every produced run record is retained, and a run’s state is classified into completed, catastrophic, timed_out, or infrastructure: completed runs are scored as before; catastrophic and timed-out runs are publishable manually (human-reviewed, unscored) so a model’s failures are visible on the gallery and counted toward leaderboard reliability; infrastructure failures are never publishable and are excluded entirely. The consoles gain a failures worklist and a publish-failures surface.

Re-architected server-side execution: dispatcher, drivers, and artifacts

The v0.2.0 long-lived worker pool is gone. A run launched from a web console is now enqueued at the backend’s run queue; a thin, stateless dispatcher claims the queued run and creates exactly one Kubernetes Job running a driver; the driver executes that one run (creating an untrusted sandbox pod via the Kubernetes API), streams its live progress back to the backend (which relays it to the console), uploads the produced tree to the new artifact service, and pushes the produced record — then exits. Each run is one schedulable Job, so concurrency scales with the cluster rather than a hand-sized pool, with no per-pod registration and no long-lived worker (see Server-side Run Topology).

The new pieces:

• The backend owns the run queue and a live relay: it admits jobs, hands each to exactly one dispatcher via an atomic POST /jobs/next, and relays driver progress to the watching console.
• The dispatcher (crates/dispatcher) is the bridge between the queue and the cluster — it claims jobs, creates driver Jobs, bounds in-flight work, and reports any driver-pod death the driver itself could not.
• The driver (crates/driver) is the per-run executor that resolves the definition, runs the case, streams progress, and uploads artifacts. Its upload authenticates with a per-job token verified against the job id (sent via an x-tcab-job-id header — a run id is not its job id).
• The artifact service (crates/artifacts) serves produced run trees — playable builds and proof/asset media — off a persistent volume. Its build/media reads are unauthenticated so a browser’s <img>/iframe (which cannot carry a bearer token) can load them.

Local development runs the same manifests on k3d, so a run is a Job everywhere; a new k3d overlay and Makefile give a deployment-parity dev stack, and the backend’s ingest now streams progress as NDJSON. The backend, auth, dispatcher, driver, and arena images are published to GHCR.

Generated contracts

The cross-language data contracts (TypeScript bindings + JSON Schemas) are now generated from the Rust types rather than hand-maintained, via a new crates/contract-codegen (ts-rs + schemars behind a contract feature) with a CI drift check. The generated set covers the run-record, arena, snapshot, auth, review, error, and catalog/versions contracts, plus the two formerly hand-written backend schemas (publish-run-request and resolved-test-case-version). As part of this, schemars was migrated 0.8 → 1.x workspace-wide and all baked schemas re-baselined.

New end-to-end cases

Three new end-to-end cases ship alongside the new test types: Fathom (pacman v1.0.0), Spectra (galaga v1.0.0), and Meltdown (desktop-td v1.0.0).

Backend and snapshot contract changes

The HTTP and snapshot contracts move with the lifecycle:

• The backend’s POST /runs is now push (record + links + events, no review, Bearer required). POST /runs/{id}/reviews submits a review, POST /runs/{id}/publish is the publish gate (422 with no reviews), and GET /runs takes a state of published (default) or review (the reviewer worklist, including pending runs). GET /runs/{id} now returns { record, reviews, published, links }, with reviews as an array.
• The backend also owns the run queue endpoints (POST /jobs/next, POST /jobs/{id}/status, and the live relay) that the dispatcher and driver use; the old worker’s HTTP surface is gone.
• The public snapshot contains only published runs; each per-run file now carries a reviews array (each { reviewerId, reviewer, ratings, writeup, checklist, reviewedAt }) plus published, and the runs index summary carries the aggregate rating (worst across reviews) and a reviewCount. The site computes the aggregate score (the average) client-side.

Known Issues

The split lifecycle adds a second human to the loop (a reviewer distinct from the operator); the consoles’ review and worklist surfaces for that hand-off are still rough and tuned for the desktop. The desktop app still executes runs through its own in-process runner rather than the new dispatcher/driver path, so desktop-produced failure publishing lags the server-side flow.