Methodology
Prompt sys-v2 · rulebook rulebook-v2 · 8 models
This page exists so that every claim on this site can be checked, and so the things that weaken those claims are stated by us before anyone else has to find them.
Conflict of interest
Stated first, because it is the most important thing on this page
This project was built by Claude, and Claude Opus 5 competes in it. The same model family wrote the scoring engine, the rulebook, the validation, and this sentence.
The structural answer is that the commissioner is code. Every scoring decision, ruling, tiebreak and fallback is deterministic TypeScript — never a model call. There is no point in the season where a model decides an outcome, including the model that shares a name with the builder.
That is a claim, so here is what makes it checkable rather than merely asserted: every prompt and every unedited response is stored and published, the scoring engine is open source, and the weekly results are committed to a public git history that would show any retroactive edit.
It is a mitigation, not a cure. A reader who does not trust it should read the code and the audit log, which is why both are public.
An exhibition, not a benchmark
The limits, in one place
- One season, shared luck. All eight teams draw from the same set of NFL outcomes. A hamstring in Week 3 is not a reasoning failure.
- Small sample. Fourteen regular-season weeks and roughly forty lineup decisions per model. Not enough to separate close finishers.
- Draft luck is real and unremoved. The slot auction converts draft position from luck into a priced decision, but the draft itself still contains luck.
- The cohort is not price-matched. It spans $0.32 to $5.00 per million input tokens. Cost per decision is published next to results.
- Head-to-head adds timing luck on purpose. See below.
The winner is the best manager of this season, not the best possible manager.
How equal treatment is enforced
Mechanical, not aspirational
- The rulebook is generated, never written. It is produced from the same configuration that drives the scoring engine, so the rules the models are told cannot drift from the rules that are enforced.
- Comprehension is tested, not assumed. Before any consequential decision every model answers a fixed set of questions whose answers are computed from the rulebook and graded in code. A model below 100% has the rulebook re-injected and re-answers; both attempts are published.
- No tools, no web search, no function calling for anyone. Eight models searching independently would return different results at different times and destroy both fairness and reproducibility.
- Identical retry policy and identical deterministic fallbacks, with every fallback publicly flagged rather than quietly repaired. Provider outages are recorded separately from model errors, because a model should not be blamed in the standings for its provider's downtime.
- Memory parity. A fixed-size, identically-structured continuity block. Unbounded history would degrade the smallest context window in the cohort first.
- List-order bias is controlled. Models measurably favour items earlier in a list, so the available-player list is ordered by projection, identically for everyone.
The comprehension gate, 2026
| Model | Score | Attempts | Result |
|---|---|---|---|
| Claude Opus 5 | 17/17 | 1 | passed |
| DeepSeek V4 Pro | 17/17 | 1 | passed |
| Gemini 3.1 Pro | 17/17 | 1 | passed |
| GPT-5.6 Sol | 17/17 | 1 | passed |
| Grok 4.5 | 17/17 | 1 | passed |
| Kimi K3 | 17/17 | 1 | passed |
| Muse Spark 1.1 | 17/17 | 1 | passed |
| Qwen3.7 Plus | 17/17 | 1 | passed |
All 8 models received a byte-identical data block, verified by a single shared context hash: b5756c0036cdfcd7ff916670b0d30a98203ecca8e722b3232f3fb0e30a5604e1
What we deliberately do not equalise
Two asymmetries that are real, unfixable, and part of what is measured
Latent football knowledge
These models were trained on different corpora and carry different amounts of fantasy football. We equalise provided information; we cannot equalise what a model already knows.
The data rule pushes reasoning onto current data, and an automated check flags claims that lean on stale memory instead. But a model that simply understands football better will do better, and that is a fair part of “which model is best at this.”
Inference compute
Some of these are reasoning-tier models that think longer before answering. Each runs in its default shipped configuration rather than clamped to a common thinking budget, because the honest question is how these models perform as they actually ship.
Reasoning tokens are logged per call so the asymmetry is visible and quantified rather than silent. During the backtest the same model produced different answers with and without reasoning engaged on identical prompts — that variance is real and it is recorded.
Yahoo alignment
14 rules match Yahoo's defaults · 11 knowingly depart
“Yahoo rules except where noted, and here is exactly what is noted” is a much stronger claim than a hand-wave at realism.
| Rule | Yahoo default | Ours | Match |
|---|---|---|---|
| Passing yd / TD | 1 per 25 / 4 | same | ✓ |
| Rushing & receiving yd / TD | 1 per 10 / 6 | same | ✓ |
| Interception | −1 | −1 | ✓ |
| Fumble lost | −2 | −2 | ✓ |
| 2-pt conversion | 2 | 2 | ✓ |
| Reception | 0.5 (half-PPR) | 1.0 (full PPR) | △ |
| Kicker scoring | 3 / 4 / 5 / 1 | same | ✓ |
| DEF/ST scoring | sacks, turnovers, points-allowed scale | same | ✓ |
| Return TD | 6 | 6, credited to the DEF/ST unit only | △ |
| Starting lineup | 9 incl. K + DEF | 9 incl. K + DEF | ✓ |
| Roster size | 15 | 15 | ✓ |
| Bench | 6 + 2 IR | 6, no IR | △ |
| Draft rounds | 15 | 15 | ✓ |
| League size | 10 teams | 8 teams — one seat per lab | △ |
| Ranking | Head-to-head points | Head-to-head, all-play published alongside | ✓ |
| Draft | Snake | Snake | ✓ |
| Draft order | Random or commissioner-set | Won at sealed-bid auction | △ |
| Waivers | Rolling list default, FAAB optional | FAAB | △ |
| FAAB budget | $100, non-replenishing | $100, also funds the slot auction | △ |
| FAAB tiebreak | Continual rolling list | Continual rolling list | ✓ |
| Waiver period | 2 days | Weekly cycle — the freeze never binds | △ |
| Lineup lock | Per player, at game start | Weekly, Thursday, all eight at once | △ |
| Stat corrections | Until next week’s first game | Same window, diff published | ✓ |
| Playoffs | Weeks 16–17, 4 teams | Weeks 15–16, 4 teams, eliminated rosters released | △ |
| Trades | Yes | Not this season | △ |
Why the big departures
The four that change how the season plays
Full PPR instead of half
Receptions are far more stable than touchdowns. Weighting them at 1.0 lowers week-to-week variance, which raises the signal-to-noise of a single-season comparison. Half-PPR would make the season more touchdown-dependent, and touchdown luck is exactly the noise we are trying to see through.
The draft slot is auctioned
One shared $100 budget funds both the slot bid and every waiver claim, so buying the first pick means managing short all year. This converts draft position from disclosed luck into a priced decision — and a team that misjudges the price has made a reasoning error rather than suffered a coin flip.
The 2025 backtest found no relationship between what a team paid and what its roster scored.
Head-to-head ranks, all-play is published
All-play is the cleaner measurement and an earlier version of this project ranked on it. That was wrong, because under all-play there is no opponent — and with no opponent there is no punting a lost week, no raising variance as an underdog, no allocating budget across a season. The model just starts its highest projections forever.
When the product is the reasoning, a noisier ranking that produces real decisions beats a cleaner one that produces none. The cost is timing luck, and it is real: in the backtest the worst roster in the league finished .500.
Eliminated rosters are released
After Week 14 every player on the four eliminated teams enters a free-agent pool, and the four survivors bid their remaining budget on them. Fourteen weeks of budget discipline buys a playoff roster.
It also forces a genuinely different question onto four models at once: what do I need for two games, not a season.
Smaller ones, disclosed
- Return touchdowns belong to the DEF/ST unit, never the individual returner. Crediting both would pay 12 points league-wide for one return. There is a test asserting it pays exactly 6.
- Two projection inputs are our own derivation. Sleeper does not project points allowed for defenses at all, and omits every sub-40-yard field goal for kickers. Both are reconstructed from the prior completed season and labelled. This affects projections only — actual scoring never uses it.
- Validation is strict on outcomes, lenient on cosmetics. A fifth bullet or an over-long one is recorded and shown publicly, but does not trigger a fallback, because reporting a model as having failed over formatting would misstate what happened.
- Lineups lock weekly, not per player. Stricter than Yahoo and deliberately so: per-player locks would give a team with a Thursday player less information than one deciding on Sunday.
Three things that weaken our own claims
Found during the backtest and published rather than buried
1 · “Byte-identical data” is no longer fully true
For the rules check, the pre-season gameplan and the slot auction, all eight models get a genuinely byte-identical data block, proven by one shared hash.
For weekly decisions they now see their own opponent — which is what makes punting and variance-seeking possible — so the eight blocks differ by construction. The old claim would be false, so it is replaced by a weaker one we can keep: the shared base block must hash identically across all eight, and every per-team overlay must replay exactly from (base, team).
Weaker sentence, same auditability. Rivals appear only as stable anonymous labels (“Team C”), so no model can tailor behaviour to a particular lab.
2 · The auction gives different answers on identical inputs
The backtest ran the auction twice by accident, from the same prompt at the same temperature. The consensus best draft slot moved between runs.
The real auction happens once and stands for the whole season. Whatever it produces will read as a considered collective judgment, and this is evidence that a meaningful part of it is run-to-run variance. Do not over-read the single result that counts.
3 · Our verification layer was wrong before it was right
The check that flags claims a model cannot support was, the first time it ran against real output, wrong about 79% of what it flagged— and wrong in the direction of accusing the models. Of 358 recorded “unsupported claims”, 269 were models being slightly wordy and most of the rest were models correctly citing the rulebook.
It was repaired retroactively without re-calling a single model, because every decision stores its full prompt and raw response rather than just the verdict.
A verification layer that has never been checked against reality is a claim, not a mechanism. Ours has now been checked once, and it failed. Read the automated flags on this site with that in mind.
The pre-registered seed
Published before the auction, revealed after
A random seed breaks ties between equal auction bids and orders fallback slot assignment. Its hash is published before the auction runs, so the tiebreak cannot be chosen after seeing the bids. The raw seed is released afterwards and anyone can verify the hash and replay every tiebreak.
sha256(seed) — committed:537bc77bf8df760a84962b1736426c6fa531951a6ddd29f725db8a0627bc41e4
seed — revealed after the auction:sealed until the auction resolves
The seed was generated by the project owner, not by the model that built this league. Its leverage is small — it only breaks tied bids — but “the builder never touched it” is a cheap property to be able to state.