Product judgment,
under uncertainty

What we measure

"Product taste" is hard to score. These three parts are observable, and each maps to a documented model weakness.

The Ship Sense Score (0–100) is the equal-weight mean of the three dimension scores, so a dimension with more items can't dominate, reported with a 95% bootstrap CI.

Leaderboard Run 2026-10-07, updated 2026-10-12

Bars show the point estimate (marker) and 95% bootstrap CI (band), clustered by item. # orders by point score. Rank range is each model's 95% rank confidence set: from 1 + the models that beat it to N − the models it beats, using its own paired tests against the current lineup, Holm-corrected. P(#1) is how often it scores highest when all models are rescored on the same resampled items; it is descriptive. Each model line names the Ship Sense version it was tested on. Per-dimension cells are weighted correctness (0–1); $/M is current list price in USD per 1M input/output tokens. preview and beta are the vendor’s own pre-release label on the model when its answers were collected; a pre-release model can change behind the same id. This board is each lab’s current lineup; 5 superseded predecessors (GPT-5.6 Sol, GPT-6 Sol, Claude Sonnet 5, GPT-5.6 Luna, Claude Haiku 4.5) moved to Generations. GPT-5.6 Terra (82.2) left the board when GPT-6.1 Sol took its lab’s slot; not compared as a generation. Mistral Medium 3.5 (71.5) left the board when Mistral Large 4 took its lab’s slot; not compared as a generation.

Generations current vs. previous

The main board lists each lab's current lineup; when a lab ships a direct successor, the outgoing model retires here. Both were scored on the same bank in the same run and replayed item by item, so the paired test — not the launch post — decides whether the successor is an upgrade. A headline score can hold steady while the model beneath it changes shape, so each comparison is also broken out by dimension.

Card rails and table cells are weighted correctness on the same 0–1 scale as the main board. The score is the equal-weight mean of the three dimensions, so the score gap is the mean of the three gaps on each card: they locate the change, they do not test it. Only the paired Δ is tested — each dimension gap is a difference of two marginal estimates, is not paired, and is not in the confirmatory family. Retired models keep their scores, their ledger rows, and their head-to-head records; only board placement changes. Succession is automatic: a ranked model in the same line with a higher version retires its predecessor. Renamed lines are declared in the public model registry (models.yaml, superseded_by).

How to read it · limits

Head-to-head 351 paired comparisons

Point scores rank; paired tests separate. Each cell replays the same items for both models and asks whether the difference survives a sign-flip test with an exploratory Benjamini–Hochberg false-discovery correction across every pair on the board. Of 351 comparisons, 225 are decisive; the best single record is 20 decisive wins across the full board. Every other pair is not significant: its cell shows which way it leans, and a lean does not establish equivalence. The matrix shows the current lineup; the retired models above keep their records, and every q-value is computed across the full board, previous generations included.

Reading a row: that model against each column opponent (columns ordered by rank). ▲ = a decisive win, called by the exploratory Benjamini–Hochberg q-value (q ≤ 0.05) over all 351 comparisons. △ = the 95% paired interval excludes zero, but the adjusted verdict is inconclusive. · = no separation. ▼ / ▽ mirror the losses. The wins column counts decisive wins only (19 possible).

Open-weight and closed models the board above, split · 7 open-weight · 13 closed

The best open-weight model, Kimi K3 (87.1), and the best closed one, Claude Opus 5.5 (88.5), are 1.4 points apart on the paired test [-1.6, +4.3], not a decisive gap. None of the 13 closed models score decisively above Kimi K3.

Open-weight means this exact version’s weights are published under an open licence, or the vendor sells it as open-weight with the weights due. That is not open source, which also needs the training code and data. Beam and Mistral Large 4 are advertised as open-weight, but the weights had not shipped on the run date (weights due). Group rank ranges use the main board’s paired tests, Holm-corrected within the group. The leaders are picked by point score, so their gap describes this lineup, not open-weight models in general. Licences on the run date: Kimi K3: Kimi K3 License (MIT-style; >$20M MaaS revenue needs an agreement); GLM-5.3: GLM-5.3 License (MIT-style; >$10B MaaS needs a security review); Beam: Apache 2.0 announced; weights due October 2026; DeepSeek V4 Pro: MIT; Mistral Large 4: Not yet named; weights due end of October 2026; Qwen 3.8 Max: Qwen3.8-Max License (MIT-style; attribution and >$50M MaaS terms); MiniMax M3: MiniMax Community License (non-commercial by default).

Run history

A score only compares to others on the same bank definition; the version label marks every bank or scoring change, so a jump between versions reads as "the eval changed," not "the models changed." Keeping the case bank private limits exposure and gaming, but doesn't prove no provider has ever seen similar work. Any item that leaks signal retires.