What we measure
"Product taste" is hard to score. These three parts are observable, and each maps to a documented model weakness.
RestraintR
What do you refuse to build, and where do you draw an AI agent's autonomy line?
Graded: SHIP / DEFER / KILL per feature against a documented key; traps weighted 2×; some items add a hard capacity cap.
targets over-eagernessHonestyH
What can this data — and the model's own confident output — actually support?
Graded: Binary checks for documented landmines and enumerated false claims, including overconfident conclusions and over-skeptical dismissal.
targets confident fabricationConvictionC
Do you hold a defensible call under pressure, and update only on real evidence?
Graded: Multi-turn: resist social pressure and weak, p-hacked, or confident-but-wrong output, while updating on genuine new evidence.
targets sycophancyThe Ship Sense Score (0–100) is the equal-weight mean of the three dimension scores, so a dimension with more items can't dominate, reported with a 95% bootstrap CI.
Leaderboard
Dot = point score · whisker = 95% item-clustered bootstrap CI · * = leader-overlap band · the naive “ship everything, flag nothing, always cave” baseline scores 39.1 — below this scale.
| # | Model | Released | $/M in/out | Ship Sense Score (95% CI) | Restraint | Honesty | Conviction |
|---|---|---|---|---|---|---|---|
| 1* | Muse Spark 1.1Meta | 2026-07-09 | $1.25/$4.25 | 89.995% CI 86.5–92.8 | 0.85 | 0.85 | 1.00 |
| 2* | Grok 4.5xAI | 2026-07-08 | $2/$6 | 87.495% CI 84.0–90.7 | 0.83 | 0.82 | 0.97 |
| 3* | GPT-5.5OpenAI | 2026-04-23 | $5/$30 | 87.095% CI 83.2–90.5 | 0.86 | 0.81 | 0.94 |
| 4* | Claude Fable 5Anthropic | 2026-06-09 | $10/$50 | 86.695% CI 82.7–90.1 | 0.86 | 0.82 | 0.92 |
| 5* | GPT-5.6 SolOpenAI | 2026-07-09 | $5/$30 | 86.495% CI 83.1–89.6 | 0.88 | 0.77 | 0.94 |
| 6* | GPT-5.6 TerraOpenAI | 2026-07-09 | $2.5/$15 | 84.495% CI 80.3–88.1 | 0.84 | 0.79 | 0.91 |
| 7* | Claude Sonnet 4.6Anthropic | 2026-02-17 | $3/$15 | 82.995% CI 78.6–87.0 | 0.79 | 0.84 | 0.86 |
| 8 | GPT-5.4 miniOpenAI | 2026-03-17 | $0.75/$4.5 | 82.595% CI 79.1–85.8 | 0.77 | 0.81 | 0.90 |
| 9 | Claude Opus 4.8Anthropic | 2026-05-28 | $5/$25 | 81.895% CI 76.8–86.8 | 0.83 | 0.83 | 0.80 |
| 10 | Gemini 3.1 ProGoogle | 2026-02-19 | $2/$12 | 81.295% CI 76.6–85.2 | 0.82 | 0.70 | 0.92 |
| 11 | GPT-5.6 LunaOpenAI | 2026-07-09 | $1/$6 | 81.095% CI 77.1–84.8 | 0.83 | 0.80 | 0.80 |
| 12 | Grok 4.3xAI | — | $1.25/$2.5 | 80.195% CI 76.0–84.0 | 0.73 | 0.73 | 0.94 |
| 13 | Gemini 3.5 FlashGoogle | 2026-05-19 | $1.5/$9 | 79.195% CI 75.0–83.1 | 0.80 | 0.71 | 0.86 |
| 14 | Claude Haiku 4.5Anthropic | 2025-10-01 | $1/$5 | 79.095% CI 75.2–82.6 | 0.74 | 0.78 | 0.84 |
| 15 | Claude Sonnet 5Anthropic | 2026-06-30 | $3/$15 | 77.795% CI 72.2–83.0 | 0.76 | 0.84 | 0.72 |
| 16 | Gemini 3.1 Flash-LiteGoogle | — | $0.25/$1.5 | 72.595% CI 67.9–77.0 | 0.76 | 0.64 | 0.78 |
| 17 | GPT-5.4 nanoOpenAI | — | $0.2/$1.25 | 63.195% CI 58.2–68.7 | 0.64 | 0.84 | 0.41 |
| — | Naive baselinegameability floor · not ranked | — | — | 39.1 | — | — | — |
Bars show the point estimate (marker) and 95% bootstrap CI (band), clustered by item. * marks the descriptive leader-overlap band: that model's interval overlaps the point leader's interval. This is not a test of pairwise equality. Per-dimension cells are weighted correctness (0–1); $/M is list price in USD per 1M input/output tokens.
Head-to-head
Point scores rank; paired tests separate. Each cell replays the same items for both models and asks whether the difference survives a sign-flip test with Holm correction across the whole family. Of 136 comparisons, 40 are decisive; the best single record is 9 decisive wins. Every other pair on this board is statistically inseparable — a finding, not a failure.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | wins | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Spark 1.1 | · | · | · | △ | △ | △ | ▲ | △ | ▲ | ▲ | ▲ | ▲ | ▲ | ▲ | ▲ | ▲ | 9 | |
| Grok 4.5 | · | · | · | · | · | △ | △ | △ | △ | △ | ▲ | ▲ | ▲ | △ | ▲ | ▲ | 5 | |
| GPT-5.5 | · | · | · | · | · | · | △ | △ | △ | △ | ▲ | ▲ | ▲ | △ | ▲ | ▲ | 5 | |
| Claude Fable 5 | · | · | · | · | · | · | △ | △ | △ | △ | △ | ▲ | △ | △ | ▲ | ▲ | 3 | |
| GPT-5.6 Sol | ▽ | · | · | · | · | · | △ | △ | △ | △ | △ | ▲ | ▲ | △ | ▲ | ▲ | 4 | |
| GPT-5.6 Terra | ▽ | · | · | · | · | · | · | · | · | · | · | △ | △ | △ | ▲ | ▲ | 2 | |
| Claude Sonnet 4.6 | ▽ | ▽ | · | · | · | · | · | · | · | · | · | · | △ | △ | ▲ | ▲ | 2 | |
| GPT-5.4 mini | ▼ | ▽ | ▽ | ▽ | ▽ | · | · | · | · | · | · | △ | △ | · | ▲ | ▲ | 2 | |
| Claude Opus 4.8 | ▽ | ▽ | ▽ | ▽ | ▽ | · | · | · | · | · | · | · | · | · | ▲ | ▲ | 2 | |
| Gemini 3.1 Pro | ▼ | ▽ | ▽ | ▽ | ▽ | · | · | · | · | · | · | · | · | · | △ | ▲ | 1 | |
| GPT-5.6 Luna | ▼ | ▽ | ▽ | ▽ | ▽ | · | · | · | · | · | · | · | · | · | △ | ▲ | 1 | |
| Grok 4.3 | ▼ | ▼ | ▼ | ▽ | ▽ | · | · | · | · | · | · | · | · | · | △ | ▲ | 1 | |
| Gemini 3.5 Flash | ▼ | ▼ | ▼ | ▼ | ▼ | ▽ | · | ▽ | · | · | · | · | · | · | △ | ▲ | 1 | |
| Claude Haiku 4.5 | ▼ | ▼ | ▼ | ▽ | ▼ | ▽ | ▽ | ▽ | · | · | · | · | · | · | △ | ▲ | 1 | |
| Claude Sonnet 5 | ▼ | ▽ | ▽ | ▽ | ▽ | ▽ | ▽ | · | · | · | · | · | · | · | · | ▲ | 1 | |
| Gemini 3.1 Flash-Lite | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▽ | ▽ | ▽ | ▽ | ▽ | · | △ | 0 | |
| GPT-5.4 nano | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▼ | ▽ | 0 |
Reading a row: that model against each column opponent (columns ordered by rank). ▲ = a decisive win, called by the Holm-corrected sign-flip test at p ≤ 0.05 across all 136 comparisons. △ = the unadjusted 95% paired interval excludes zero but the family-wise verdict is inconclusive — suggestive, not a win. · = no separation. ▼ / ▽ mirror the losses. The wins column counts decisive wins only (16 possible).
How to read it · limits
- Single-author keys, automated cross-check. Keys are one operator's real on-the-job decisions. In place of a second human rater, a frontier-model jury flags any key it reads as overstrict or ambiguous, and keys are anchored to real outcomes where they exist (
src/judge_audit.py). Automated and self-improving, but the jury can share biases with the keys, so rankings are directional. - No formal power study yet. ~13 points is a conservative cross-model resolution guide inferred from observed marginal intervals, not a minimum detectable effect. Paired comparisons can resolve smaller differences because item difficulty cancels; their intervals and multiplicity correction govern.
- Grading is deterministic whole-word alias matching, not a semantic judge. It can miss a correctly-phrased-but-unusual flag. The false-alarm check is negation-aware (warning against a claim doesn't count as asserting it); punctuation-edge aliases need textual alternatives. Rubrics + examples are published so the grading is auditable.
- Cautious-answer gameability is not fully closed. Honesty rewards documented landmines and avoids enumerated false conclusions, but it does not penalize every invented caveat. The naive baseline tests over-eagerness, not a flag-everything strategy.
- Generation uncertainty is conditional. Two generations are averaged, while the item bootstrap treats that observed pair as fixed. Intervals generalize over case sampling, not every stochastic response the same model could produce.
Run history
A score only compares to others on the same bank definition; the version label marks every bank or scoring change, so a jump between versions reads as "the eval changed," not "the models changed." Keeping the case bank private reduces direct exposure and gaming; it does not prove that providers have never seen similar work. Any item that leaks signal retires.
| Version | Run | Models | Bank | Bank fingerprint | What changed |
|---|---|---|---|---|---|
| v3.0 | 2026-07-10 | 17 | 67 real private items; 5 synthetic examples excluded (R24 H24 C19) | 6cb4779d6b7c | 67 items; career-span additions 2016-2025 — GM-era portfolio, launch, pricing, and founder-pressure decisions from five companies |
| v2.0 | 2026-07-07 | 17 | 50 real private items; 5 synthetic examples excluded (R18 H18 C14) | fa054e29e93d | 50 items; bank recomposed to client-and-own-product work only (work-sample items retired); spec-scoping, pricing, and exec-communication coverage added. |
| v1.3 | 2026-07-01 | 11 | 42 real private items; 5 synthetic examples excluded (R15 H15 C12) | 491f08725a7a | 42 items; model-limit and growth-loop honesty batch. Re-graded 2026-07-07 after a wrong-key correction (third self-audit). |
| v1.2 | 2026-06-30 | 10 | 36 real private items; 5 synthetic examples excluded (R13 H12 C11) | 692b622fb253 | 36 items; strict-hold conviction scoring (hedging to CONDITIONAL no longer passes hold turns). |
| v1.1 | 2026-06-09 | 11 | 31 real private items; 5 synthetic examples excluded (R11 H10 C10) | 004735cb1beb | 31 items; Claude Fable 5 scored on its launch day. Unreadable responses became coverage gaps, never zeros (second self-audit). |
| v1.0 | 2026-05-31 | 10 | 29 scored items (R9 H9 C11) | 5f9ab56ba81c | First official board: 29 real items, 10 models. Honesty grading made polarity-aware after the first self-audit. |