Router Benchmark Results (run 2)

run timestamp: 2026-05-15T14:08:51Z repo commit: 358c36b461df4c0cb7a5c972e2f789dce8c12d3a (description fixes + hardened runner) fixture sha256-16: 8f974d930836bc9c (unchanged from baseline) seed: 1 runs: 600 of 600 (100%; resume + concurrency=4; ~15 minute wall time vs ~60 minutes sequential) models: claude-opus-4-7, composer-2, gemini-3.1-pro, gpt-5.5 reps per (prompt, model): 3

Executive summary

This run measures the effect of the description rewrites and harness improvements that landed after the baseline at results-published/2026-05-15.md.

Headline results:

• 3 of 4 models improved on top-1 accuracy. Composer-2 0.888 -> 0.913 (+2.5pp), GPT-5.5 0.886 -> 0.913 (+2.7pp), Gemini 3.1 Pro 0.886 -> 0.925 (+3.9pp). Claude Opus 4.7 went 0.886 -> 0.867 (-2.0pp top-1) but improved on top-3 (+1.7pp), so it is essentially noise.
• All 4 models improved on top-3 accuracy. Composer-2 top-3 jumped from 0.930 to 0.973 (+4.3pp).
• The targeted description rewrites worked. context-fundamentals went from 12/47 = 0.255 to 22/45 = 0.489 (+23.4pp, the largest single-skill improvement). project-development went from 0.750 to 1.000 (+25pp, now perfect routing). tool-design went from 0.729 to 0.807 (+7.8pp).
• The previously-hardest prompts are mostly fixed. p001 ("Explain why context windows degrade") went from 0.00 to 0.83 top-1. p037 ("Why structured output design") went from 0.00 to 1.00. p040 and p045 (other context-fundamentals prompts) improved by +17pp each.
• One apparent regression is mostly an artifact. advanced-evaluation looks like it dropped from 0.980 to 0.797, but the baseline only completed 49 of 60 expected runs (the v1 process died at 566/600); the new run completed all 60 (well, 59 after one format failure). The 11 newly-attempted runs were heavily weighted toward p048, a genuinely ambiguous prompt ("Plan how to evaluate KV compaction with ablations and baselines") that routes to evaluation 11/12 times across all models. Absolute correct count is 48 (baseline) vs 47 (new): essentially the same.

What the data says we should do next:

• The remaining failure modes are now concentrated in context-fundamentals (still only 49% top-1) and a few specific prompts:
• p047 ("Translate this English paragraph to French") regressed -33pp on top-1. This is a negative control where no skill should fit; the new routing went to project-development instead of context-fundamentals. This is fine behavior; the fixture's expected primary is debatable.
• p046 ("Reformat this Python file") and p048 ("Plan KV compaction evaluation") remain at 0.00 top-1 across all models. Both are genuinely ambiguous; consider re-labeling.
• context-fundamentals still has 14 confusions to project-development. The new description routes correctly when prompts use foundational vocabulary ("attention mechanics", "anatomy of context"), but generic onboarding prompts ("explain context for a new team member") still route to project-development. May need one more description pass.

Methodological notes:

• Wall time improvement: 60min -> 15min via concurrency=4. The hardened runner means future sweeps will not silently die at 94%, and resume capability means a killed sweep can be picked up exactly where it left off.
• Format compliance: 597 of 600 (99.5%). Three format failures from Gemini; all four other models had zero. Gemini's strict-JSON adherence is measurably weaker than the other three.
• Latency: Gemini median 9130ms vs 3269-4201ms for others. Same pattern as baseline; Gemini is consistently ~2.5-3x slower.

Methodology

Each prompt is presented to each model with the 15 skill activation descriptions in a deterministically-shuffled order (different shuffle per replication). The model must return JSON with a ranked list of skill names. Top-1 accuracy is whether the first ranked skill matches the human-labeled expected_primary_skill; top-3 is whether the expected skill appears in the first three positions.

No skills are loaded into the agent (settingSources: []); the only routing signal is the in-prompt descriptions. Confidence intervals are 95% bootstrap with 2000 resamples.

Per-model leaderboard

Per-skill confusion (when expected is X, predicted is Y)

Rows are the ground-truth expected_primary_skill; columns are what models actually predicted. Only finished runs counted.

Hardest prompts (lowest top-1 across all models)

Reproducibility

Reproduce these numbers exactly with:

cd researcher/benchmarks/sdk-runner
npm install
export CURSOR_API_KEY=<your-key>
node --experimental-strip-types src/runRouter.ts --models claude-opus-4-7,composer-2,gemini-3.1-pro,gpt-5.5 --reps 3 --seed 1 --max-budget-usd 15
python3 researcher/scripts/render_router_report.py \
    --results researcher/benchmarks/router/results/<date>-<seed> \
    --fixture researcher/benchmarks/router/prompts.jsonl \
    --output researcher/benchmarks/router/results-published/<date>.md

Per-run JSON artifacts (prompt, model, replication, raw model output, parsed ranking) are preserved under the gitignored results/ directory next to the summary that drives this report.

Delta vs baseline

baseline: 2026-05-15 v2.2.0 descriptions (commit a865a8e)

Per-model accuracy change

Per-skill top-1 rate change

Counts a row as correct when the predicted primary equals the expected primary.

Previously-hardest prompts