Decision & game-theory trainers
Charleston Lab
Calibration error 15% to 2.5% on held-out games
- Simulation
- Analytics
Evidence
- Data
- Self-play simulation · 1,320 samples (330 games x 4 seats); 264 held out
- Stage
- Shipped
- Stack
- JavaScript, Logistic regression, GitHub Pages
Evaluation
What was measured, against what, and how.
| Result | 15% to 2.5% ECE |
|---|---|
| Baseline | Previous closed-form predictor scored on the same held-out rows: ECE 15% (Brier 0.180 to 0.156) |
| Sample size | 264 held-out rows of 1,320 self-play samples (330 games x 4 seats), 80/20 split |
| Timeframe | Committed 2026-06-16 |
| Method | fit_model.js: seeded shuffle, 80/20 split, L2 logistic regression fitted on the training rows; 8-bin ECE on the test rows only |
| Source | fit_model.js and IMPLEMENTATION_PLAN.md (B.9) in the mahjong repo |
| Confidence | Probable |
Limits
- Not reproducible from the repo yet: corpus.csv isn't committed and the corpus is seeded with Date.now()
- Row-level split: the four seats of one game can land on both sides
- 264 test rows over 8 bins; no interval reported for the 2.5%
- Tied to the 0.1.0-greedy rollout policy
- Separately: tile conservation held in 10,000 simulated runs (0 joker moves, 0 leaks)
Outcome: Succeeded. Goal: a calibrated predictor of hand completion. On held-out self-play rows its calibration error fell from 15% to 2.5%; the corpus isn't committed yet, so the number can't be reproduced.
How I used AI
What the tools did, and what was checked.
Built with: Jett, by hand with Claude Code.
Note to be written: tool and model, what the assistant produced, what was rejected or rewritten, and which check caught a problem.