Agent systems
AutoBuilder
9 of 13 first deliveries worked without a re-prompt
- AI
- Data viz
Evidence
- Data
- AutoBuilder's own run log (the codex index), curated per run · 14 logged builds, 13 assessed
- Stage
- Prototype
- Stack
- Claude Desktop Cowork agent sessions, Role charters in Markdown + JSON state files, Node aggregator and HTML dashboard (codex), GitHub Pages
The question
Can one plain-language prompt (“build me an app to play blackjack”) turn into a working product with no follow-up, if the work is split across specialised agents that check each other? AutoBuilder is the architecture I built to find out. Each build is a measurement of that architecture, not a product to polish.
From sketch to roles
It started as a hand drawing: discovery, technical discovery, then sections with an overseer and builders each. Nothing in that drawing doubts the work, so before the first run the design grew roles whose only job is to check the others: a long-running Critic, an Arbiter that routes problems by severity, a Historian that keeps the decisions, and a Convergence Verifier that tests the finished product against the original spec.
How a build runs
Every role writes to its own files, and only the owner of a file may write it, so each decision leaves a record. After delivery, each build gets a second, user-side label: did the person who typed the prompt get something that worked on first contact? That outcome is the number this page leads with. It is kept apart from the pipeline’s own verdict, because the two disagreed more than once. A dashboard (the codex) reads the run files and draws each build’s topology.
What changed, and what it cost
Each failure became an amendment, and the failed prompt was re-run word for word afterwards. I didn’t patch failed builds: a patched build hides the failure the architecture needs to learn from.
v1.3 · Gave up trusting each section’s own tests → gained acceptance assertions per section and a required run of the integrated product in a browser-like sandbox. Trigger: a blackjack button that never re-enabled.
v1.5 · Gave up testing against a simulated browser with dependencies swapped in → gained a production-fidelity check: the verifier opens the deliverable the way the user will. Cost: an earlier pass was reclassified as a failure.
v1.7 · Gave up a growing list of per-artifact test rules → gained one property to check: every claim in a spec has at least one test pointing back at it.
v1.9 · Gave up trusting the model’s familiarity with outside products → gained four principles: a product named in the prompt is a specific external system, every load-bearing claim about it needs a citation, success is verified at the level it is claimed, and the verifier may not share the builder’s source of truth. Trigger: a plugin built for macOS, delivered to a Windows user.
Throughout · Gave up repairing builds that failed → gained clean before-and-after comparisons on the same prompt.
Evaluation
What was measured, against what, and how.
| Result | 9 of 13 assessed builds |
|---|---|
| 95% interval | 42% to 87% (Wilson score interval on 9 of 13) |
| Baseline | None yet. The same prompts without AutoBuilder (one plain agent session) were never run: needs measuring |
| Sample size | 13 assessed builds of 14 logged (1 unverified: gto-poker-trainer, no user signal) |
| Timeframe | May 2026, architecture v1.3 to v1.10.1 (4 runs record no version) |
| Method | Each run's first-delivery outcome: did the user get something that worked from the first prompt? Set from, in order: a curation file, the run report's status block, a root-cause analysis, or a non-empty uncertainty manifest (= worked with concerns). Counted: succeeded (3) + succeeded with concerns (6) |
| Source | codex/data/index.json (first_delivery_outcome_distribution) in the auto-builder repo |
| Confidence | Verified |
Failure modes
- MLB dashboard (v1.10.1): the install steps didn't work as written and the dashboard itself didn't satisfy the request; fixed by iterating in the same chat
- Blackjack (version not recorded): the Deal button stayed disabled after a hand ended, past every gate; Jett found it
- LaTeX renderer (v1.4): passed AutoBuilder's verifier, but the math library never loaded in a real browser
- Stream Dock plugin (v1.8): passed every gate, but was built for macOS; Jett is on Windows
Limits
- No baseline: nobody ran the same prompts in a single agent session, so this shows what AutoBuilder delivered, not that it beat anything
- Small sample: 13 builds. The Wilson 95% interval is wide (42% to 87%)
- Mixed versions: the 13 builds span architecture v1.3 to v1.10.1 (4 record none), and the architecture changed after almost every failure, so this is not one system measured 13 times and not a trend
- Partly self-labelled: 3 of the 9 successes are 'with concerns' from the uncertainty-manifest rule alone, without Jett's ratification; 5 labels were set by the Codex or Maintenance instance in curation files
- Narrow prompt mix: 12 of the 14 runs are browser apps (tic-tac-toe, kanban, blackjack and the like); the one prompt outside the browser, a hardware plugin, failed on its first attempt
Outcome: Partly. Goal: one prompt in, a working product out, with no follow-up. 9 of 13 assessed first deliveries met it, 4 needed a re-prompt, and there is no baseline yet to say whether that beats a plain agent session. Paused since May 2026, not retired: I may return to it.
Run telemetry
Counts from the run log, each with the rule that produced it.
4 of 13
Builds that needed a user re-prompt or fix
Rule Runs whose first-delivery outcome is 'user re-prompted', out of the 13 runs with an assessed outcome. None was unrecoverable.
The four cases are listed under failure modes above.
3 of 4
Re-prompted builds that had passed AutoBuilder's own checks
Rule Of the 4 re-prompted builds, those whose verdict at delivery (before any later reclassification) was pass or pass with concerns. The MLB dashboard's verdict was not recorded, so it is not counted as a pass.
Blackjack (pass with concerns), LaTeX renderer (pass), Stream Dock v1.8 (pass). This gap between the internal verdict and the user's first contact is what drove the v1.3, v1.5 and v1.9 amendments.
2 of 2
A/B re-runs that turned a re-prompt into a working first delivery
Rule Pairs where the same prompt was re-run after an architecture amendment, as linked in the run log. Counted when the re-run's first delivery succeeded, with or without concerns.
Blackjack to blackjack trainer (succeeded, ratified). Stream Dock v1.8 to v1.9 (succeeded with concerns, from its uncertainty manifest; not ratified, and its uncertainty manifest lists three unverified bets, starting with whether its Touch Bar declaration is valid at all). Two pairs is anecdote, not a trend.
| Category | Count |
|---|---|
| Succeeded | 3 |
| Succeeded with concerns · worked, with documented caveats | 6 |
| User re-prompted | 4 |
| Unrecoverable | 0 |
| Unverified · no user signal | 1 |
| Category | Count |
|---|---|
| Pass | 6 |
| Pass with concerns | 5 |
| Failed, then recovered · blackjack and the LaTeX renderer, reclassified by re-audit | 2 |
| Unknown · not recorded | 1 |
| Category | Count |
|---|---|
| Curation file | 5 |
| Uncertainty manifest | 5 |
| Run-report status block | 3 |
| Default (no signal) | 1 |
| Three of the six 'succeeded with concerns' come from the uncertainty-manifest rule alone, without Jett's ratification. They are the weakest part of the 9. | |
| Category | Count |
|---|---|
| Re-Verification · in 6 runs | 71 |
| Convergence Verifier · in 8 runs | 28 |
| Discovery · in 1 run | 7 |
| Technical Discovery · in 2 runs | 4 |
| Coordinator · in 1 run | 2 |
| Researcher · in 1 run | 2 |
| Builder, Critic, Historian, Integrator, Overseer · each, in 1 run | 1 |
| Re-Verification events are the gates of retroactive re-audits under v1.6. | |
Needs measuring
- Cost per build: not recorded for any run
- Time per build: wall-clock minutes are missing or zero for 11 of 14 runs
- Baseline: the same prompts in a single agent session without AutoBuilder
What was built
The deliverables this project covers.
- AutoBuilder framework Prototype
- Codex: AutoBuilder's run-log dashboard dashboard Shipped
- MLB daily dashboard dashboard Shipped · Live
- Apple Music now-playing Stream Dock plugin plugin Prototype
How I used AI
What the tools did, and what was checked.
- Tools
- Claude Desktop Cowork agent sessions: one chat per build, running AutoBuilder's role charters as sub-agents. Two long-lived Cowork instances: Codex (run-log dashboard and curation) and Maintenance (architecture amendments, per-build git).
- Models
- Not recorded per run.
- Extent
- Agents wrote every build, its run report and its verification, and the codex dashboard and aggregator. The architecture amendments were drafted by agents in conversation with Jett. Jett sketched the first architecture by hand and set the direction of each change.
- My role
- Spec and evaluation design: chose the prompts, defined the user-side first-delivery outcome as the measure, tested delivered builds himself, ratified finished builds and had the final say on amendments, including re-running failed prompts verbatim instead of patching builds.
- What checking caught
- The LaTeX renderer passed AutoBuilder's own verifier, then showed 'library not loaded' for every input the first time Jett opened it. The verifier had tested in a simulated browser with the library installed locally. Fix: v1.5 requires testing the deliverable the way the user opens it.
- The Stream Dock plugin passed every internal gate but was built for macOS; Jett is on Windows, and the build had Windows paths in front of it. Fix: v1.9 adds principles that treat a named product as a specific outside system and require a citation for every load-bearing claim about it.
- A re-audit reclassified the first blackjack build from 'pass with concerns' to failed, so the run log doesn't keep a misleading green flag.
- While building this page: the codex index labels the Stream Dock A/B pair the wrong way round (the v1.9 run is tagged as the first attempt). The page orders the pair by architecture version and date; the run log is left as it is.
- Evidence
- Run reports and the run log live in a public repo that hasn't been cleared for linking yet (it contains local paths and contact details). A cleaned excerpt will be self-hosted here.