ngpt · experiment 3 · Tue 20 Oct 2026

Small models showdown

21 configurations of 19 small open language models, from 2.5B dense to 36B mixture-of-experts, run through one vLLM harness on ten benchmarks with the same prompts, budgets and graders. At the top, level with each other: Qwen3.6-35B-A3B (72.3) and Gemma 4 26B-A4B (72.1); behind them, five configurations within error of each other, from Nemotron 3.5 Lightning 30B-A3B (69.7) to Qwen3.5-9B (67.6). Among models of 10B parameters or fewer, Dory at 4 loops (69.1) and Qwen3.5-9B (67.6) lead, level within error. NVIDIA's looped Dory is first here on competition math, IFBench (tied) and AA-LCR at 4 loops, and in the lower half on SimpleQA.

configurations

19 models; Dory at 1, 2 and 4 loops

benchmarks

knowledge, math, code, instructions, long context

234K

graded answers

one harness: same prompts, budgets and graders

$812

Modal bill, of a $2,000 cap

$682 of it GPUs: 134 GPU-hours, 10 at a time

What stands out

Mixture-of-experts on top

Qwen3.6-35B-A3B (72.3) and Gemma 4 26B-A4B (72.1) lead the index, level with each other; both are mixture-of-experts models with 3–4B active parameters. Behind them, Nemotron 3.5 Lightning 30B-A3B (69.7), Dory at 4 loops (69.1), Dory at 2 loops (68.7), LFM2.5-24B-A2B (67.9) and Qwen3.5-9B (67.6) are level within error, dense and mixture-of-experts alike.

Best at 10B or under

Dory at 4 loops (69.1); the best other model is Qwen3.5-9B (67.6), 1.5 ± 1.1 behind, level within error.

Dory: math, instructions, long documents

At 4 loops Dory is first here on competition math, IFBench (tied) and AA-LCR. On the index it is fourth of 21 at 4 loops, fifth at 2 and ninth at 1. Its weak spots are knowledge (SimpleQA 3.4) and verbatim retrieval (MRCR 39.0).

Budgets shape some scores

Ornith 1.0 9B (68% of IFNovel answers cut), MiMo-V2.6 Distill 9B (56% of competition math answers cut), Qwen3.5-4B (43% of IFNovel answers cut) and Qwen3.5-9B (32% of IFNovel answers cut). Every model gets the same output budget: 64K tokens for math, code and GPQA, 64K or what the context window leaves for AA-LCR, and 32K for the rest. A cut answer scores 0.

Small models know few facts

SimpleQA: the best is Qwen3.6-35B-A3B at 18.1%; 20 of 21 configurations answer fewer than 10% correctly.

The cheapest card per hour is the dearest per token

An L4 costs $0.80 an hour and yields 1.8 million output tokens per dollar; a B200 costs $6.25 an hour and yields 9.7 million. So every job ran on B200, H200 or H100.

The leaderboard

The index is the mean of seven benchmark scores, each 0–100. Its ± is a standard error over questions, so two indexes closer than about twice their combined error are not separated by this run.

#	Model	Index	MMLU-Pro	GPQA	SimpleQA	Math	LCB	IFEval	IFBench	AA-LCR	MRCR	IFNovel	$
1	Qwen3.6-35B-A3B 36B / 3.3B active	72.3 ±0.8	85.9	84.7	18.1	86.6†	74.7†	90.0	66.2†	71.0	98.6	45.5†	22
2	Gemma 4 26B-A4B 26B / 3.8B active	72.1 ±0.8	85.8	79.7	7.5†	83.1†	74.7	94.6	79.1	63.0	52.0†	85.0†	20
3	Nemotron 3.5 Lightning 30B-A3B 33B / 3.0B active	69.7 ±0.8	80.5	76.1	9.6	81.3†	73.6†	93.3	73.2	62.0	41.8	85.0	15
4	Dory 4B, 4 loops 4.0B	69.1 ±0.8	77.7	73.4	3.4	88.8	68.4	93.3	79.1	74.0	39.0	93.3	53
5	Dory 4B, 2 loops 4.0B	68.7 ±0.8	77.0	70.6	3.2	87.6	70.3	93.9	78.3	66.0	32.4	92.7	30
6	LFM2.5-24B-A2B 24B / 2.0B active	67.9 ±0.8	77.8	78.9	5.0	77.4†	64.3	95.4	76.6	28.0	19.8†	82.5	13
7	Qwen3.5-9B 9.7B	67.6 ±0.8	83.1	82.2	8.1	79.7†	63.2†	90.6	66.7†	63.0	96.3	59.3†	25
8	Granite 4.2 8B 8.8B	66.6 ±0.8	71.8†	63.4	1.7	83.9†	75.0	93.5	77.1	48.0	26.6	94.3	75
9	Dory 4B, 1 loop 4.0B	65.1 ±0.8	75.1	70.1	2.8	79.1	60.2	93.5	74.9	65.0	25.3	84.3	22
10	gpt-oss-20b (high) 21B / 3.6B active	63.6 ±0.9	75.5	64.1†	5.0	73.4†	69.0†	88.7	69.4	34.0	17.5†	77.0†	31
11	Qwen3.5-4B 4.7B	63.2 ±0.9	80.0	78.4	5.6†	74.2†	53.3†	90.6	60.3†	59.0	93.7	47.8†	23
12	MiniCPM5-2B 2.5B	62.7 ±0.9	71.9	69.4	1.4	77.6†	67.0	87.4	64.2†	58.0	25.0†	37.8†	13
13	GLM-4.7-Flash 31B / 3.0B active	62.6 ±0.9	78.9	76.0	7.5	79.7†	51.1†	83.5	61.3	48.0	29.8†	32.0†	54
14	Granite 4.2 3B 3.7B	60.7 ±0.9	67.0†	54.2	0.7	72.8†	60.7	92.8	76.8	21.0	14.7	84.0	28
15	Ornith 1.0 9B 9.4B	60.6 ±0.9	82.6	79.7	6.7†	73.6†	47.8†	83.5†	50.1†	52.0†	83.4	21.7†	40
16	Nemotron 3 Nano 4B 4.0B	54.1 ±0.9	68.1	53.5	2.8	57.7	49.5	86.1	60.7	14.0	11.2	49.7	8
17	LFM2.5-2.6B 2.7B	52.4 ±0.9	68.5	57.2	2.3	45.9†	40.9	92.4	59.5	8.0†	15.4	60.7	9
18	Olmo 3 7B Think 7.3B	51.0 ±0.9	66.6	51.1	3.4	57.7	48.6	84.8	44.4	–	16.2‡	37.3†	64
19	Gemma 4 E4B 8B / 4.5B active	49.4 ±0.9	71.2	56.7	2.1	38.4	53.0	86.0	38.5	38.0	31.2	26.8	4
20	MiMo-V2.6 Distill 9B 9.4B	41.5 ±0.9	73.2†	49.6†	7.0†	39.2†	29.4†	66.5†	25.5†	49.0	64.8	7.8†	36
21	Jamba2-3B 3.2B	32.2 ±0.7	48.6	29.0	3.9	5.3†	17.0	83.5	37.9†	12.0	22.5	6.0†	5

Index = the mean of the seven core columns (MMLU-Pro through IFBench), each 0–100, ± its standard error. AA-LCR, MRCR and IFNovel are shown beside it, not inside it. LCB = LiveCodeBench. Bold blue = the column's best. † more than 5% of answers cut at the output budget; ‡ more than 20% of prompts too long for the model's context window. $ = the GPU time of the configuration's generation jobs at Modal's prices, engine start-up included. Hover a cell for its standard error.

Benchmark by benchmark

One panel per benchmark, best first. Whiskers are 95% intervals over questions. A note under a panel names the configurations whose score there is shaped by the output budget or the context window.

MMLU-Pro

MMLU-Pro, 3,008 of 12,032 questions (every 4th), one answer each

Qwen3.6-35B-A3B

85.9

Gemma 4 26B-A4B

85.8

Qwen3.5-9B

83.1

Ornith 1.0 9B

82.6

Nemotron 3.5 Lightning 30B-A3B

80.5

Qwen3.5-4B

80.0

GLM-4.7-Flash

78.9

LFM2.5-24B-A2B

77.8

Dory 4B, 4 loops

77.7

Dory 4B, 2 loops

77.0

gpt-oss-20b (high)

75.5

Dory 4B, 1 loop

75.1

MiMo-V2.6 Distill 9B

73.2

MiniCPM5-2B

71.9

Granite 4.2 8B

71.8

Gemma 4 E4B

71.2

LFM2.5-2.6B

68.5

Nemotron 3 Nano 4B

68.1

Granite 4.2 3B

67.0

Olmo 3 7B Think

66.6

Jamba2-3B

48.6

Shaped by limits: Granite 4.2 3B: 6% of answers cut at the budget; Granite 4.2 8B: 7% of answers cut at the budget; MiMo-V2.6 Distill 9B: 13% of answers cut at the budget.

GPQA

GPQA Diamond, 198 questions × 4 answer orders

Qwen3.6-35B-A3B

84.7

Qwen3.5-9B

82.2

Ornith 1.0 9B

79.7

Gemma 4 26B-A4B

79.7

LFM2.5-24B-A2B

78.9

Qwen3.5-4B

78.4

Nemotron 3.5 Lightning 30B-A3B

76.1

GLM-4.7-Flash

76.0

Dory 4B, 4 loops

73.4

Dory 4B, 2 loops

70.6

Dory 4B, 1 loop

70.1

MiniCPM5-2B

69.4

gpt-oss-20b (high)

64.1

Granite 4.2 8B

63.4

LFM2.5-2.6B

57.2

Gemma 4 E4B

56.7

Granite 4.2 3B

54.2

Nemotron 3 Nano 4B

53.5

Olmo 3 7B Think

51.1

MiMo-V2.6 Distill 9B

49.6

Jamba2-3B

29.0

Shaped by limits: MiMo-V2.6 Distill 9B: 41% of answers cut at the budget; gpt-oss-20b (high): 12% of answers cut at the budget.

SimpleQA

SimpleQA, 4,326 short factual questions, graded by gpt-oss-120b

Qwen3.6-35B-A3B

18.1

Nemotron 3.5 Lightning 30B-A3B

9.6

Qwen3.5-9B

8.1

Gemma 4 26B-A4B

7.5

GLM-4.7-Flash

7.5

MiMo-V2.6 Distill 9B

7.0

Ornith 1.0 9B

6.7

Qwen3.5-4B

5.6

gpt-oss-20b (high)

5.0

LFM2.5-24B-A2B

5.0

Jamba2-3B

3.9

Olmo 3 7B Think

3.4

Dory 4B, 4 loops

3.4

Dory 4B, 2 loops

3.2

Dory 4B, 1 loop

2.8

Nemotron 3 Nano 4B

2.8

LFM2.5-2.6B

2.3

Gemma 4 E4B

2.1

Granite 4.2 8B

1.7

MiniCPM5-2B

1.4

Granite 4.2 3B

0.7

Shaped by limits: Qwen3.5-4B: 5% of answers cut at the budget; MiMo-V2.6 Distill 9B: 17% of answers cut at the budget; Ornith 1.0 9B: 48% of answers cut at the budget; Gemma 4 26B-A4B: 5% of answers cut at the budget.

Math

AIME 2025 + AIME 2026 + HMMT Feb 2026 + HMMT Nov 2025, 123 problems × 4

Dory 4B, 4 loops

88.8

Dory 4B, 2 loops

87.6

Qwen3.6-35B-A3B

86.6

Granite 4.2 8B

83.9

Gemma 4 26B-A4B

83.1

Nemotron 3.5 Lightning 30B-A3B

81.3

Qwen3.5-9B

79.7

GLM-4.7-Flash

79.7

Dory 4B, 1 loop

79.1

MiniCPM5-2B

77.6

LFM2.5-24B-A2B

77.4

Qwen3.5-4B

74.2

Ornith 1.0 9B

73.6

gpt-oss-20b (high)

73.4

Granite 4.2 3B

72.8

Nemotron 3 Nano 4B

57.7

Olmo 3 7B Think

57.7

LFM2.5-2.6B

45.9

MiMo-V2.6 Distill 9B

39.2

Gemma 4 E4B

38.4

Jamba2-3B

5.3

Shaped by limits: Qwen3.5-4B: 19% of answers cut at the budget; Granite 4.2 3B: 7% of answers cut at the budget; Granite 4.2 8B: 6% of answers cut at the budget; MiniCPM5-2B: 9% of answers cut at the budget; LFM2.5-2.6B: 28% of answers cut at the budget; Jamba2-3B: 18% of answers cut at the budget; Qwen3.5-9B: 13% of answers cut at the budget; MiMo-V2.6 Distill 9B: 56% of answers cut at the budget; Ornith 1.0 9B: 23% of answers cut at the budget; gpt-oss-20b (high): 23% of answers cut at the budget; Qwen3.6-35B-A3B: 10% of answers cut at the budget; Gemma 4 26B-A4B: 9% of answers cut at the budget; Nemotron 3.5 Lightning 30B-A3B: 8% of answers cut at the budget; LFM2.5-24B-A2B: 15% of answers cut at the budget; GLM-4.7-Flash: 15% of answers cut at the budget.

LCB

LiveCodeBench, the 182 problems from Jan–Apr 2025 × 2, every hidden test run

Granite 4.2 8B

75.0

Qwen3.6-35B-A3B

74.7

Gemma 4 26B-A4B

74.7

Nemotron 3.5 Lightning 30B-A3B

73.6

Dory 4B, 2 loops

70.3

gpt-oss-20b (high)

69.0

Dory 4B, 4 loops

68.4

MiniCPM5-2B

67.0

LFM2.5-24B-A2B

64.3

Qwen3.5-9B

63.2

Granite 4.2 3B

60.7

Dory 4B, 1 loop

60.2

Qwen3.5-4B

53.3

Gemma 4 E4B

53.0

GLM-4.7-Flash

51.1

Nemotron 3 Nano 4B

49.5

Olmo 3 7B Think

48.6

Ornith 1.0 9B

47.8

LFM2.5-2.6B

40.9

MiMo-V2.6 Distill 9B

29.4

Jamba2-3B

17.0

Shaped by limits: Qwen3.5-4B: 24% of answers cut at the budget; Qwen3.5-9B: 20% of answers cut at the budget; MiMo-V2.6 Distill 9B: 46% of answers cut at the budget; Ornith 1.0 9B: 38% of answers cut at the budget; gpt-oss-20b (high): 23% of answers cut at the budget; Qwen3.6-35B-A3B: 12% of answers cut at the budget; Nemotron 3.5 Lightning 30B-A3B: 10% of answers cut at the budget; GLM-4.7-Flash: 21% of answers cut at the budget.

IFEval

IFEval, 541 prompts, prompt-level strict

LFM2.5-24B-A2B

95.4

Gemma 4 26B-A4B

94.6

Dory 4B, 2 loops

93.9

Dory 4B, 1 loop

93.5

Granite 4.2 8B

93.5

Dory 4B, 4 loops

93.3

Nemotron 3.5 Lightning 30B-A3B

93.3

Granite 4.2 3B

92.8

LFM2.5-2.6B

92.4

Qwen3.5-4B

90.6

Qwen3.5-9B

90.6

Qwen3.6-35B-A3B

90.0

gpt-oss-20b (high)

88.7

MiniCPM5-2B

87.4

Nemotron 3 Nano 4B

86.1

Gemma 4 E4B

86.0

Olmo 3 7B Think

84.8

Jamba2-3B

83.5

Ornith 1.0 9B

83.5

GLM-4.7-Flash

83.5

MiMo-V2.6 Distill 9B

66.5

Shaped by limits: MiMo-V2.6 Distill 9B: 6% of answers cut at the budget; Ornith 1.0 9B: 11% of answers cut at the budget.

IFBench

IFBench, 300 prompts × 2, mean strict fraction of instructions followed

Dory 4B, 4 loops

79.1

Gemma 4 26B-A4B

79.1

Dory 4B, 2 loops

78.3

Granite 4.2 8B

77.1

Granite 4.2 3B

76.8

LFM2.5-24B-A2B

76.6

Dory 4B, 1 loop

74.9

Nemotron 3.5 Lightning 30B-A3B

73.2

gpt-oss-20b (high)

69.4

Qwen3.5-9B

66.7

Qwen3.6-35B-A3B

66.2

MiniCPM5-2B

64.2

GLM-4.7-Flash

61.3

Nemotron 3 Nano 4B

60.7

Qwen3.5-4B

60.3

LFM2.5-2.6B

59.5

Ornith 1.0 9B

50.1

Olmo 3 7B Think

44.4

Gemma 4 E4B

38.5

Jamba2-3B

37.9

MiMo-V2.6 Distill 9B

25.5

Shaped by limits: Qwen3.5-4B: 14% of answers cut at the budget; MiniCPM5-2B: 6% of answers cut at the budget; Jamba2-3B: 8% of answers cut at the budget; Qwen3.5-9B: 11% of answers cut at the budget; MiMo-V2.6 Distill 9B: 23% of answers cut at the budget; Ornith 1.0 9B: 35% of answers cut at the budget; Qwen3.6-35B-A3B: 7% of answers cut at the budget.

AA-LCR

AA-LCR, 100 questions over ~100K-token documents, graded by gpt-oss-120b

Dory 4B, 4 loops

74.0

Qwen3.6-35B-A3B

71.0

Dory 4B, 2 loops

66.0

Dory 4B, 1 loop

65.0

Qwen3.5-9B

63.0

Gemma 4 26B-A4B

63.0

Nemotron 3.5 Lightning 30B-A3B

62.0

Qwen3.5-4B

59.0

MiniCPM5-2B

58.0

Ornith 1.0 9B

52.0

MiMo-V2.6 Distill 9B

49.0

Granite 4.2 8B

48.0

GLM-4.7-Flash

48.0

Gemma 4 E4B

38.0

gpt-oss-20b (high)

34.0

LFM2.5-24B-A2B

28.0

Granite 4.2 3B

21.0

Nemotron 3 Nano 4B

14.0

Jamba2-3B

12.0

LFM2.5-2.6B

8.0

Shaped by limits: LFM2.5-2.6B: 9% of answers cut at the budget; Ornith 1.0 9B: 12% of answers cut at the budget.

MRCR

OpenAI MRCR, 200 conversations, 2 and 8 needles, 4K–128K tokens

Qwen3.6-35B-A3B

98.6

Qwen3.5-9B

96.3

Qwen3.5-4B

93.7

Ornith 1.0 9B

83.4

MiMo-V2.6 Distill 9B

64.8

Gemma 4 26B-A4B

52.0

Nemotron 3.5 Lightning 30B-A3B

41.8

Dory 4B, 4 loops

39.0

Dory 4B, 2 loops

32.4

Gemma 4 E4B

31.2

GLM-4.7-Flash

29.8

Granite 4.2 8B

26.6

Dory 4B, 1 loop

25.3

MiniCPM5-2B

25.0

Jamba2-3B

22.5

LFM2.5-24B-A2B

19.8

gpt-oss-20b (high)

17.5

Olmo 3 7B Think

16.2

LFM2.5-2.6B

15.4

Granite 4.2 3B

14.7

Nemotron 3 Nano 4B

11.2

Shaped by limits: MiniCPM5-2B: 6% of answers cut at the budget; Olmo 3 7B Think: 22% of prompts longer than its context window; gpt-oss-20b (high): 8% of answers cut at the budget; Gemma 4 26B-A4B: 24% of answers cut at the budget; LFM2.5-24B-A2B: 7% of answers cut at the budget; GLM-4.7-Flash: 8% of answers cut at the budget.

IFNovel

IFNovel, 300 prompts with 22 constraint types written for this project

Granite 4.2 8B

94.3

Dory 4B, 4 loops

93.3

Dory 4B, 2 loops

92.7

Gemma 4 26B-A4B

85.0

Nemotron 3.5 Lightning 30B-A3B

85.0

Dory 4B, 1 loop

84.3

Granite 4.2 3B

84.0

LFM2.5-24B-A2B

82.5

gpt-oss-20b (high)

77.0

LFM2.5-2.6B

60.7

Qwen3.5-9B

59.3

Nemotron 3 Nano 4B

49.7

Qwen3.5-4B

47.8

Qwen3.6-35B-A3B

45.5

MiniCPM5-2B

37.8

Olmo 3 7B Think

37.3

GLM-4.7-Flash

32.0

Gemma 4 E4B

26.8

Ornith 1.0 9B

21.7

MiMo-V2.6 Distill 9B

7.8

Jamba2-3B

6.0

Shaped by limits: Qwen3.5-4B: 43% of answers cut at the budget; MiniCPM5-2B: 7% of answers cut at the budget; Jamba2-3B: 9% of answers cut at the budget; Olmo 3 7B Think: 12% of answers cut at the budget; Qwen3.5-9B: 32% of answers cut at the budget; MiMo-V2.6 Distill 9B: 36% of answers cut at the budget; Ornith 1.0 9B: 68% of answers cut at the budget; gpt-oss-20b (high): 8% of answers cut at the budget; Qwen3.6-35B-A3B: 29% of answers cut at the budget; Gemma 4 26B-A4B: 6% of answers cut at the budget; GLM-4.7-Flash: 24% of answers cut at the budget.

Two compute knobs: loops and reasoning effort

Two ways to spend more compute at inference time. Dory repeats its 20-layer middle block 1, 2 or 4 times per token, so each token costs more while answers get a little shorter. gpt-oss-20b's reasoning effort (low, medium, high) keeps the cost per token and writes longer reasoning. Both are drawn at x = 1, 2, 4, Dory's loop counts; for gpt-oss-20b the positions only order low, medium and high.

GPQA

Dory: 1, 2, 4 loopsgpt-oss-20b: low, medium, high effort

Math

Dory: 1, 2, 4 loopsgpt-oss-20b: low, medium, high effort

LCB

Dory: 1, 2, 4 loopsgpt-oss-20b: low, medium, high effort

IFBench

Dory: 1, 2, 4 loopsgpt-oss-20b: low, medium, high effort

Setting	GPQA	Math	LCB	IFBench	Mean output tokens, GPQA and IFBench
Dory 4B, 1 loop	70.1	79.1	60.2	74.9	13045
Dory 4B, 2 loops	70.6	87.6	70.3	78.3	12027
Dory 4B, 4 loops	73.4	88.8	68.4	79.1	11934
gpt-oss-20b (low)	58.1	28.5	45.6	39.2	572
gpt-oss-20b (medium)	64.3	68.7	62.6	61.2	3447
gpt-oss-20b (high)	64.1	73.4	69.0	69.4	16779

GPQA: Dory +3.3 from 1 to 4 loops, gpt-oss-20b +6.1 from low to high effort; IFBench: Dory +4.2 from 1 to 4 loops, gpt-oss-20b +30.3 from low to high effort. Mean output tokens on those two: Dory 13,045 → 11,934; gpt-oss-20b 572 → 16,779. At high effort gpt-oss-20b runs past the 64K budget on 12% of GPQA answers, which caps its GPQA score. The low and medium settings ran without MMLU-Pro and SimpleQA.

Size and price

The index against total parameters, and against what each configuration cost to run here (its generation jobs at Modal's prices; the judge and graders are shared and left out). A configuration is on the frontier when nothing smaller, or cheaper, scores higher.

Index against total parameters

Index against what it cost to run

Frontier by size: MiniCPM5-2B, Dory at 4 loops, Gemma 4 26B-A4B and Qwen3.6-35B-A3B. Frontier by cost: Gemma 4 E4B, Nemotron 3 Nano 4B, MiniCPM5-2B, LFM2.5-24B-A2B, Nemotron 3.5 Lightning 30B-A3B, Gemma 4 26B-A4B and Qwen3.6-35B-A3B. A configuration's cost here mixes model speed with how long it thinks, so a verbose model costs more than its size suggests; it also includes loading the engine, which is up to a third of the smallest models' bills.

Long context

Two different skills. AA-LCR asks questions that need reasoning across a set of documents of about 100K tokens; a judge grades the answer. OpenAI MRCR hides 2 or 8 near-identical requests (the needles) in a long conversation and asks for the answer to one of them, word for word. The charts show MRCR by conversation length for the five best configurations and Dory.

MRCR, 2 needles

Qwen3.6-35B-A3BQwen3.5-9BQwen3.5-4BOrnith 1.0 9BMiMo-V2.6 Distill 9BDory 4B, 4 loops

MRCR, 8 needles

Qwen3.6-35B-A3BQwen3.5-9BQwen3.5-4BOrnith 1.0 9BMiMo-V2.6 Distill 9BDory 4B, 4 loops

AA-LCR: Dory at 4 loops 74.0, Qwen3.6-35B-A3B 71.0 and Dory at 2 loops 66.0 lead. MRCR: Qwen3.6-35B-A3B 98.6, Qwen3.5-9B 96.3 and Qwen3.5-4B 93.7. Dory at 4 loops: AA-LCR 74.0, MRCR 39.0. A model with a 131K window has less room to think about a ~100K-token AA-LCR prompt than one with 262K; Olmo 3 (65K) cannot take the prompts at all.

What the makers report, and what this harness measures

Where a model's maker reports a benchmark this harness runs, the maker's number beside ours. A difference can come from the settings (output budget, sampling, prompt, benchmark version, how the answer is extracted) as much as from the model; the maker's setting is quoted where the card states it.

Model	Benchmark	Maker	Here	Δ	Maker's setting
Dory 4B, 1 loop	AA-LCR	63.0	65.0 ±4.8	+2.0	n_recurrent_loops=1; AA-LCR v1.1 corrected answer keys, older judge Qwen3-235B-A22B (not G
Dory 4B, 1 loop	GPQA Diamond	69.2	70.1 ±2.7	+0.9	n_recurrent_loops=1 ('1 loop' column), no tools
Dory 4B, 1 loop	IFBench	74.3	74.9 ±2.1	+0.6	n_recurrent_loops=1; bare 'IFBench' - no loose/strict, no prompt/instruction level
Dory 4B, 2 loops	AA-LCR	71.0	66.0 ±4.8	-5.0	n_recurrent_loops=2; AA-LCR v1.1 keys, older judge Qwen3-235B-A22B; 16 repeats
Dory 4B, 2 loops	GPQA Diamond	72.4	70.6 ±2.9	-1.8	n_recurrent_loops=2 ('2 loops' column), no tools
Dory 4B, 2 loops	IFBench	77.9	78.3 ±2.0	+0.4	n_recurrent_loops=2; bare 'IFBench' - no loose/strict, no prompt/instruction level
Dory 4B, 4 loops	AA-LCR	73.0	74.0 ±4.4	+1.0	n_recurrent_loops=4; AA-LCR v1.1 keys, older judge Qwen3-235B-A22B; 16 repeats
Dory 4B, 4 loops	GPQA Diamond	72.2	73.4 ±2.8	+1.2	n_recurrent_loops=4 ('4 loops' column), no tools
Dory 4B, 4 loops	IFBench	79.0	79.1 ±2.0	+0.1	n_recurrent_loops=4; bare 'IFBench' - no loose/strict, no prompt/instruction level
GLM-4.7-Flash	AIME 2025	91.6	85.0† ±5.4	-6.6	row label "AIME 25". Card's Evaluation Parameters, default settings (most tasks): temperat
GLM-4.7-Flash	GPQA Diamond	75.2	76.0 ±2.4	+0.8	row is labelled only "GPQA" — the card does NOT say Diamond. Default settings: temperature
GLM-4.7-Flash	LiveCodeBench	64.0	51.1† ±3.4	-12.9	v6 — row label "LCB v6"; no window dates. Default settings: temperature 1.0, top-p 0.95, m
Gemma 4 26B-A4B	AIME 2026	88.3	88.3 ±5.2	+0.0	"AIME 2026 no tools", %; 26B A4B is the 2nd value column; thinking mode
Gemma 4 26B-A4B	GPQA Diamond	82.3	79.7 ±2.4	-2.6	%, no tools/search stated; 26B A4B is the 2nd value column; thinking mode
Gemma 4 26B-A4B	IFBench	72.0	79.1 ±2.1	+7.1	NO metric variant stated; thinking mode; 26B-A4B is the 2nd value column. Tech report only
Gemma 4 26B-A4B	IFEval	98.5	94.6 ±1.0	-3.9	NO metric variant stated; thinking mode; 26B-A4B is the 2nd value column. Tech report only
Gemma 4 26B-A4B	LiveCodeBench	77.1	74.7 ±3.0	-2.4	"LiveCodeBench v6", % — v6 stated, NO date window stated; 26B A4B is the 2nd value column;
Gemma 4 26B-A4B	MMLU-Pro	82.6	85.8 ±0.6	+3.2	"MMLU Pro", % — 26B A4B is the 2nd value column; thinking mode
Gemma 4 E4B	AIME 2026	42.5	46.7 ±8.2	+4.2	"AIME 2026 no tools", %; E4B is the 4th value column; thinking mode
Gemma 4 E4B	GPQA Diamond	58.6	56.7 ±2.8	-1.9	%, no tools/search stated; E4B is the 4th value column; thinking mode
Gemma 4 E4B	IFBench	44.0	38.5 ±2.5	-5.5	NO metric variant stated (no prompt/instruction level); thinking mode; E4B is the 4th valu
Gemma 4 E4B	IFEval	96.7	86.0 ±1.5	-10.7	NO metric variant stated (no strict/loose, no prompt/instruction level); thinking mode; E4
Gemma 4 E4B	LiveCodeBench	52.0	53.0 ±3.5	+1.0	"LiveCodeBench v6", % — v6 stated, NO date window stated; E4B is the 4th value column; thi
Gemma 4 E4B	MMLU-Pro	69.4	71.2 ±0.8	+1.8	"MMLU Pro", % — E4B is the 4th value column; thinking mode per tech report Table 5 caption
Granite 4.2 3B	AIME 2025	78.3	77.5 ±6.9	-0.8	row label "AIME25". IBM blog describes the reasoning chart results as pass@1; no budget st
Granite 4.2 3B	GPQA Diamond	54.8	54.2 ±3.0	-0.6	row is labelled only "GPQA" — the card does NOT say Diamond; subset unstated. No reasoning
Granite 4.2 3B	IFBench	74.3	76.8 ±2.1	+2.5	prompt level — row label "IFBench (prompt)"
Granite 4.2 3B	LiveCodeBench	69.7	60.7 ±3.3	-9.0	v6 (row label "LiveCodeBench v6"); no window dates given
Granite 4.2 3B	MMLU-Pro	67.8	67.0† ±0.9	-0.9	no reasoning on/off stated for the eval; model's default is thinking mode (<think>). Card'
Granite 4.2 8B	AIME 2025	86.7	82.5† ±6.0	-4.2	row label "AIME25"; IBM blog calls the reasoning results pass@1
Granite 4.2 8B	GPQA Diamond	64.1	63.4 ±2.9	-0.8	row is labelled only "GPQA" — Diamond not stated by the card
Granite 4.2 8B	IFBench	79.3	77.1 ±2.1	-2.2	prompt level — row label "IFBench (prompt)"
Granite 4.2 8B	LiveCodeBench	73.2	75.0 ±2.9	+1.8	v6 (row label "LiveCodeBench v6")
Granite 4.2 8B	MMLU-Pro	74.0	71.8† ±0.8	-2.3	8B Dense column of the same shared table; no reasoning setting stated
Jamba2-3B	IFBench	36.0	37.9† ±2.6	+1.9	0-1 scale as printed on the vendor's bar chart (= 36%); no metric variant and no eval sett
Jamba2-3B	IFEval	93.0	83.5 ±1.6	-9.5	0-1 scale as printed on the vendor's bar chart (= 93%); no strict/loose or prompt/instruct
LFM2.5-2.6B	AIME 2025	51.9	45.0† ±7.3	-6.9	row label "AIME25". Card states no eval settings; the vendor blog's footnote says all mode
LFM2.5-2.6B	IFBench	59.2	59.5 ±2.5	+0.3	no metric variant stated — row is plain "IFBench". Blog footnote: vLLM, "Other evals" temp
LFM2.5-2.6B	LiveCodeBench	59.4	40.9 ±3.5	-18.5	v6 (row label "LiveCodeBenchv6"); no window dates. Blog footnote: vLLM, "Other evals" temp
LFM2.5-24B-A2B	GPQA Diamond	74.4	78.9 ±2.4	+4.5	row label "GPQA Diamond". Card's "Evaluation settings": all models run with vLLM; this one
LFM2.5-24B-A2B	IFBench	74.7	76.6 ±2.0	+1.9	no metric variant stated. Settings: vLLM, "Other evals" temperature = 0.6, max output toke
LFM2.5-24B-A2B	IFEval	96.8	95.4 ±0.9	-1.4	no strict/loose, no prompt/instruction level stated — row is plain "IFEval". Settings: vLL
MiniCPM5-2B	AA-LCR	59.0	58.0 ±5.0	-1.0	carries the dagger — from the official Artificial Analysis release, not reproduced interna
MiniCPM5-2B	AIME 2025	86.5	85.8† ±5.8	-0.7	reproduced internally; no reasoning budget stated
MiniCPM5-2B	AIME 2026	86.5	91.7 ±4.7	+5.2	reproduced internally; the only AIME 2026 number any of the eight cards reports
MiniCPM5-2B	GPQA Diamond	70.2	69.4 ±2.7	-0.8	row label "GPQA-Diamond"; carries the dagger — "Scores marked † come from the official Art
MiniCPM5-2B	HMMT Feb 2026	63.8	59.8† ±7.3	-4.0	reproduced internally; the only HMMT Feb 2026 number any of the eight cards reports
MiniCPM5-2B	IFBench	66.3	64.2† ±2.4	-2.1	no metric variant stated — row is plain "IFBench" (contrast: IBM reports IFBench (prompt))
MiniCPM5-2B	IFEval	86.7	87.4 ±1.4	+0.7	no strict/loose and no prompt/instruction level stated — row is plain "IFEval"
MiniCPM5-2B	LiveCodeBench	69.1	67.0 ±3.2	-2.1	v6 (row label "LiveCodeBench v6"); no window dates. Card also reports LCB-Pro 25Q2 Easy 68
MiniCPM5-2B	MMLU-Pro	70.8	71.9 ±0.8	+1.1	reproduced internally by OpenBMB (no dagger). No reasoning on/off or budget stated; recomm
Nemotron 3 Nano 4B	AIME 2025	78.5	55.8 ±7.0	-22.7	Reasoning-On mode; card label 'AIME25'; no pass@k or repeat count stated
Nemotron 3 Nano 4B	IFEval	87.9	86.1 ±1.5	-1.8	Reasoning-On mode; prompt level; strict/loose NOT stated
Nemotron 3 Nano 4B	LiveCodeBench	51.8	49.5 ±3.5	-2.3	Reasoning-On mode; card label 'LCB' - version and date window NOT stated
Nemotron 3.5 Lightning 30B-A3B	AA-LCR	52.0	62.0 ±4.9	+10.0	no AA-LCR version and no judge stated on this card
Nemotron 3.5 Lightning 30B-A3B	GPQA Diamond	75.4	76.1 ±2.6	+0.7	no tools
Nemotron 3.5 Lightning 30B-A3B	MMLU-Pro	81.9	80.5 ±0.7	-1.4	card label 'MMLU Pro'; no CoT / subset setting stated
Olmo 3 7B Think	AIME 2025	64.6	67.5 ±6.7	+2.9	3rd numeric column = "Olmo 3 7B Think" (the final RLVR model; 57.6 is SFT, 62.7 is DPO). E
Olmo 3 7B Think	GPQA Diamond	46.2	51.1 ±2.7	+4.9	row is labelled only "GPQA" — the card does NOT say Diamond. 3rd numeric column = final Th
Olmo 3 7B Think	IFBench	41.6	44.4 ±2.5	+2.8	no metric variant stated. 3rd numeric column = final Think model. temperature 0.6, top_p 0
Olmo 3 7B Think	IFEval	88.2	84.8 ±1.5	-3.4	no strict/loose or prompt/instruction level stated. 3rd numeric column = final Think model
Qwen3.5-4B	AA-LCR	57.0	59.0 ±4.9	+2.0	row label "AA-LCR"; no AA-LCR version, judge or repeat count stated
Qwen3.5-4B	GPQA Diamond	76.2	78.4 ±2.5	+2.2	row label "GPQA Diamond"; no setting on the table; thinking mode default
Qwen3.5-4B	HMMT Nov 2025	76.8	72.5† ±6.6	-4.3	row label "HMMT Nov 25"
Qwen3.5-4B	IFBench	59.2	60.3† ±2.5	+1.1	row label "IFBench" only — no metric variant stated
Qwen3.5-4B	IFEval	89.8	90.6 ±1.3	+0.8	row label "IFEval" only — no strict/loose, no prompt/instruction level
Qwen3.5-4B	LiveCodeBench	55.8	53.3† ±3.4	-2.5	row label "LiveCodeBench v6" — version v6, no date window anywhere on the card
Qwen3.5-4B	MMLU-Pro	79.1	80.0 ±0.7	+0.9	row label "MMLU-Pro"; no reasoning setting in the table; card line 712: "Qwen3.5 models op
Qwen3.5-9B	AA-LCR	63.0	63.0 ±4.9	+0.0	row label "AA-LCR"; no version, judge or repeat count stated
Qwen3.5-9B	GPQA Diamond	81.7	82.2 ±2.3	+0.5	row label "GPQA Diamond"; thinking mode default
Qwen3.5-9B	HMMT Nov 2025	82.9	81.7† ±6.1	-1.2	row label "HMMT Nov 25"
Qwen3.5-9B	IFBench	64.5	66.7† ±2.4	+2.2	row label "IFBench" only — no metric variant stated
Qwen3.5-9B	IFEval	91.5	90.6 ±1.3	-0.9	row label "IFEval" only — no strict/loose or level stated
Qwen3.5-9B	LiveCodeBench	65.6	63.2† ±3.3	-2.4	row label "LiveCodeBench v6" — no date window stated
Qwen3.5-9B	MMLU-Pro	82.5	83.1 ±0.7	+0.6	row label "MMLU-Pro"; thinking mode is the default (card line 710)
Qwen3.6-35B-A3B	AIME 2026	92.7	88.3† ±5.0	-4.4	row label "AIME26 " (trailing space in the cell)
Qwen3.6-35B-A3B	GPQA Diamond	86.0	84.7 ±2.1	-1.3	row label is the bare "GPQA" on this card (card prints 86.0); thinking mode default
Qwen3.6-35B-A3B	HMMT Feb 2026	83.6	79.5† ±5.8	-4.1	row label "HMMT Feb 26"
Qwen3.6-35B-A3B	HMMT Nov 2025	89.1	88.3† ±5.6	-0.8	row label "HMMT Nov 25"
Qwen3.6-35B-A3B	LiveCodeBench	80.4	74.7† ±3.0	-5.7	row label "LiveCodeBench v6" — no date window stated
Qwen3.6-35B-A3B	MMLU-Pro	85.2	85.9 ±0.6	+0.7	row label "MMLU-Pro"; card line 670: "Qwen3.6 models operate in thinking mode by default"
gpt-oss-20b (high)	AIME 2025	91.7	79.2† ±6.0	-12.5	"AIME 2025 (no tools)", accuracy %, reasoning effort HIGH; columns are gpt-oss-120b low/me
gpt-oss-20b (high)	GPQA Diamond	71.5	64.1† ±2.9	-7.4	"GPQA Diamond (no tools)", accuracy %, reasoning effort HIGH
gpt-oss-20b (high)	SimpleQA	6.7	5.0 ±0.3	-1.7	accuracy, reported as a FRACTION not a % (= 6.7%); run without browsing; reasoning effort

† more than 5% of this harness's answers were cut at its output budget (and score 0), which pulls the number below what the model would reach with more room. Left out: 10 numbers the makers measured another way (reasoning off, instruction-level or loose instruction-following metrics, tools on, a hallucination rate, or an older LiveCodeBench release). A maker's 0–1 fraction is shown as a percentage. LiveCodeBench here is the 182 problems of Jan–Apr 2025; the makers' v6 numbers cover a wider window.

84 comparisons; the median gap is 2.0 points and 57 are within 3. Furthest below the maker: Nemotron 3 Nano 4B AIME 2025 (-22.7), LFM2.5-2.6B LiveCodeBench (-18.5) and GLM-4.7-Flash LiveCodeBench (-12.9). Furthest above: Nemotron 3.5 Lightning 30B-A3B AA-LCR (+10.0), Gemma 4 26B-A4B IFBench (+7.1) and MiniCPM5-2B AIME 2026 (+5.2).

Which card is cheapest

Before the run, the same Dory job (198 GPQA prompts × 4,096 tokens, batch 256) ran on eight Modal GPU types to decide where to run. Evaluation is mostly generating tokens one at a time, which pays for memory bandwidth: the dearest card per hour is the cheapest per token. Every showdown job ran on B200, H200 or H100. The mixture-of-experts models ran on H200, where their kernels started (Gemma 4 26B-A4B on B200 as well).

GPU	$ per hour	Output tokens/s	Million tokens per $
B200	6.25	16782	9.67
H200	4.54	11053	8.76
H100	3.95	9498	8.66
A100-80GB	2.50	2470	3.56
A100-40GB	2.10	1932	3.31
L40S	1.95	1250	2.31
A10G	1.10	658	2.15
L4	0.80	393	1.77

How it was run

One harness for every model: vLLM with the Dory pull request, each model with reasoning on at its maker's recommended sampling, the same prompts, output budgets and graders. MiMo-V2.6 Distill 9B, Ornith 1.0 9B and Jamba2-3B publish no recommended sampling and ran at temperature 0.6, top_p 0.95, the field's most common reasoning setting. Each configuration's weights, sampling and the GPUs it ran on:

Configuration	Weights	Sampling	Chat settings	Ran on
Dory 4B, 1 loop	nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16	temperature 1.0, top_p 0.95	1 loop	H200, B200
Dory 4B, 2 loops	nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16	temperature 1.0, top_p 0.95	2 loops	B200, H200
Dory 4B, 4 loops	nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16	temperature 1.0, top_p 0.95	4 loops	H200, B200
Qwen3.5-4B	Qwen/Qwen3.5-4B	temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5	–	B200, H200
Nemotron 3 Nano 4B	nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16	temperature 1.0, top_p 0.95	–	H200, B200
Gemma 4 E4B	google/gemma-4-E4B-it	temperature 1.0, top_p 0.95, top_k 64	enable_thinking=True	H200, B200
Granite 4.2 3B	ibm-granite/granite-4.2-3b	temperature 1.0, top_p 0.95	enable_thinking=True	B200, H200
Granite 4.2 8B	ibm-granite/granite-4.2-8b	temperature 1.0, top_p 0.95	enable_thinking=True	H200, B200
MiniCPM5-2B	openbmb/MiniCPM5-2B	temperature 1.0, top_p 0.95, min_p 0.0	enable_thinking=True	B200, H200
LFM2.5-2.6B	LiquidAI/LFM2.5-2.6B	temperature 0.1, top_k 50, repetition_penalty 1.1	–	H200, B200, H100
Jamba2-3B	ai21labs/AI21-Jamba2-3B	temperature 0.6, top_p 0.95	–	B200, H200
Olmo 3 7B Think	allenai/Olmo-3-7B-Think	temperature 0.6, top_p 0.95	–	B200, H200, H100
Qwen3.5-9B	Qwen/Qwen3.5-9B	temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5	–	H200, B200
MiMo-V2.6 Distill 9B	XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B	temperature 0.6, top_p 0.95, top_k 20	enable_thinking=True	B200, H200
Ornith 1.0 9B	ornith-ai/Ornith-1.0-9B	temperature 0.6, top_p 0.95, top_k 20	–	B200, H200
gpt-oss-20b (high)	openai/gpt-oss-20b	temperature 1.0, top_p 1.0	reasoning_effort=high	H200
Qwen3.6-35B-A3B	Qwen/Qwen3.6-35B-A3B	temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5	–	H200
Gemma 4 26B-A4B	google/gemma-4-26B-A4B-it	temperature 1.0, top_p 0.95, top_k 64	enable_thinking=True	H200, B200
Nemotron 3.5 Lightning 30B-A3B	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	temperature 1.0, top_p 0.95	–	H200
LFM2.5-24B-A2B	LiquidAI/LFM2.5-24B-A2B	temperature 0.1, top_k 50, repetition_penalty 1.05	–	H200
GLM-4.7-Flash	zai-org/GLM-4.7-Flash	temperature 1.0, top_p 0.95	–	H200
gpt-oss-20b (medium) (side control)	openai/gpt-oss-20b	temperature 1.0, top_p 1.0	reasoning_effort=medium	H200
gpt-oss-20b (low) (side control)	openai/gpt-oss-20b	temperature 1.0, top_p 1.0	reasoning_effort=low	H200
LFM2.5-2.6B at temperature 1.0 (side control)	LiquidAI/LFM2.5-2.6B	temperature 1.0, top_p 0.95, top_k -1, repetition_penalty 1.0	–	B200

Environment and versions

vLLM: the Dory pull request, vllm-project/vllm#60149, at 55076950, installed over the precompiled cu130 wheel of main 153a1716 (0.32.0rc1.dev319), torch 2.14.0.
Image: experiment 2 ran on Modal's debian_slim Python 3.12 image; the showdown on nvidia/cuda:13.0.1-devel-ubuntu24.04 with Python 3.12, because nvcc is needed on B200, where vLLM JIT-compiles FlashInfer's attention. Both set VLLM_USE_FLASHINFER_SAMPLER=0; the vLLM build is the same.
Hardware: one GPU per job on Modal. Dense models ran on B200, falling back to H200 or H100. The MoE models (gpt-oss-20b, Qwen3.6-35B-A3B, Nemotron 3.5 Lightning, LFM2.5-24B-A2B, GLM-4.7-Flash) ran on H200 with moe_backend="triton", because their B200 start-ups failed in this build. Gemma 4 26B-A4B ran on B200 and H200 with Triton MoE. Each configuration's GPUs are in the settings table.
Weights: Dory nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 at b366b87f; every other model pinned to its Hugging Face main commit of Tue 20 Oct 2026 (table below).

git clone https://github.com/vllm-project/vllm && cd vllm
git fetch origin pull/60149/head && git checkout 55076950b6c155f384d6766d63563c3bc20a0e3f
VLLM_USE_PRECOMPILED=1 \
VLLM_PRECOMPILED_WHEEL_COMMIT=153a17166e5d07bc38184dedeea542d2e8885e55 \
VLLM_PRECOMPILED_WHEEL_VARIANT=cu130 pip install .

Serving Dory the call every Dory number came from

from vllm import LLM, SamplingParams
llm = LLM(model=DORY_PATH, max_model_len=73728,   # GPQA here; 69,632 in experiment 2, up to 196,608 for AA-LCR
          enable_prefix_caching=True, max_num_seqs=256,      # down to 4 for AA-LCR at 4 loops
          seed=zlib.crc32(run_tag.encode()) % 2**31,          # a fresh seed per launch = an independent draw
          hf_overrides={"n_recurrent_loops": k,              # 1, 2 or 4
                        "recurrent_kv_cache_mode": "per_loop"})
sp = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=65536)   # the card's sampling
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text = llm.generate([prompt], sp)[0].outputs[0].text
answer = text.rsplit("</think>", 1)[1].strip() if "</think>" in text else ""   # unclosed = no answer = wrong

Output budget: 64K tokens for GPQA, competition math, LiveCodeBench and AA-LCR (AA-LCR up to what each model's context window leaves after the ~100K-token prompt); 32K for MMLU-Pro, SimpleQA, IFEval, IFBench, IFNovel and MRCR. A reply cut at the budget, or with its reasoning never closed, scores 0. Every model ran with reasoning on, at its maker's recommended sampling.

Benchmarks, prompts and graders

Benchmark	Source (pinned)	Questions × repeats	Prompt and grading
GPQA Diamond	simple-evals CSV (openaipublic)	198 × 4 answer orders	simple-evals template; letter from “Answer: X”, else “answer is X”, else \boxed{X}; last match
MMLU-Pro	TIGER-Lab/MMLU-Pro @ b189ec76	3,008 (every 4th of 12,032) × 1	same template, ten options
Competition math	MathArena aime_2025 @ c94da77e, aime_2026 @ d61f4868, hmmt_feb_2026 @ 589a53fe, hmmt_nov_2025 @ 118dbfb4	123 × 4	MathArena's instruction; last \boxed{} chosen as MathArena's parser does; integers exact, expressions by math-verify
LiveCodeBench	livecodebench/code_generation_lite @ 0fe84c39	182 (1 Jan – 30 Apr 2025) × 2	LCB's chat prompt; last fenced block; lcb_runner @ 28fef95e, all public + private tests, 6 s each
IFEval	google/IFEval @ 966cd895	541 × 1	google-research instruction_following_eval @ e6890f85; prompt-level strict
IFBench	allenai/IFBench @ 1c40f0c1	300 × 2	AllenAI's verifiers; mean strict fraction (NVIDIA's metric)
IFNovel	written for this project	300 × 1	22 constraint types not in IFEval or IFBench, with their own verifiers
SimpleQA	simple-evals CSV (openaipublic)	4,326 × 1	gpt-oss-120b (low effort, temperature 0) with simple-evals' grader template @ 652c89d0
AA-LCR	ArtificialAnalysis/AA-LCR @ 9a77ef56 (v1.1 keys)	100 × 1	gpt-oss-120b (medium effort, temperature 0) with AA-LCR v1.1's judge prompt
OpenAI MRCR	openai/mrcr @ f4c69fae	200 (2 and 8 needles, 20 per bin, 4K–128K tokens)	the dataset README's grade(): prefix check, then SequenceMatcher ratio

Checked against NeMo Gym NVIDIA's own prompt, metric and judge, for Dory

The Dory card names NeMo Gym as its harness. Gym has no Dory recipe, so experiment 2 ported its Super 3.5 recipes at the commits their published numbers were made with (Gym ae8d388d for GPQA and IFBench, 5a7dd6fe for AA-LCR) and ran them beside this harness:

GPQA with Gym's prompt (aai/mcq-4choices) and its lenient extractor: 66.4 / 73.2 / 72.5 at 1 / 2 / 4 loops (2 repeats each, one fixed answer order per question, so position bias is in it too), against the card's 69.19 / 72.41 / 72.16.
IFBench: the card's number is Gym's reward, the mean over prompts of the strict fraction of instructions followed, graded by AllenAI's verifiers on the final answer. This page reports the same metric on the same 300 prompts.
AA-LCR: Gym's judge (Qwen3-235B-A22B-Instruct-2507, FP8) and our gpt-oss-120b judge, re-grading the same 12 complete answer sets: Gym's judge reads between 1 point lower and 2 points higher than ours (mean +0.4).

Statistics

A score is the mean over questions of each question's mean over its repeats. Two error bars appear. Against the card, which measured the same questions, the yardstick is the sampling standard error: the run-to-run noise of repeats on those questions. Between loop counts or models, differences are paired by question: the spread of the per-question differences divided by √questions. Each launch is seeded from its run name, so repeats are independent draws; byte-identical texts are checked for before pooling. Long-context columns (AA-LCR, MRCR) and IFNovel are reported beside the index, not in it.

What we fixed along the way

IFBench metric: the card's number is NVIDIA's mean strict fraction, not IFBench's prompt-level loose; experiment 1 compared the wrong one.
Repeats that weren't: vLLM seeded once per engine made separately launched AA-LCR “repeats” byte-identical copies; every launch now gets its own seed.
Math answers: the boxed answer is chosen as MathArena's parser does (an exact box beats a later decimal approximation), and math-verify gets sums written out (an exact HMMT answer left as a double sum was graded wrong).
LiveCodeBench: four test workers per 8-CPU grader (oversubscription had pushed correct solutions past the 6 s limit), time-limited records re-run alone, and a finished reply without a reasoning marker graded on its last code block as lcb_runner does.
IFEval: Google's verifiers draw a random letter for two malformed rows and langdetect samples; both are now seeded, so re-grades agree.
Equal budgets: MRCR kept at 32K for every model (grouping it with AA-LCR had given five models 64K), and context limits taken from each repo's config.

Seven review passes checked each benchmark's prompt, extraction and grading against the reference code and real outputs, and a sceptic tried to refute each finding; 17 of 37 held, grouped into the six fixes above.

Limitations

No agentic tasks. The card's Terminal-Bench, τ³-Bench, GDPval and SWE-bench columns are not run here, so this does not re-measure its Artificial Analysis index estimate.
Humanity's Last Exam (a gated dataset), AA-Omniscience and SciCode are not run either.
One draw of AA-LCR and MRCR per configuration in the showdown; AA-LCR draws of the same model differ by up to 11 points (experiment 2).
Context windows decide long-context budgets. A 131K-context model has much less room to think about a ~100K-token AA-LCR prompt than a 262K one; Olmo 3 (65K) cannot run it at all.
Some models exceed the output budgets: Ornith 1.0 9B on up to 68% of a benchmark's prompts; MiMo-V2.6 Distill 9B on up to 56% of a benchmark's prompts; Qwen3.5-4B on up to 43% of a benchmark's prompts; Qwen3.5-9B on up to 32% of a benchmark's prompts; Qwen3.6-35B-A3B on up to 29% of a benchmark's prompts; LFM2.5-2.6B on up to 28% of a benchmark's prompts; GLM-4.7-Flash on up to 24% of a benchmark's prompts; Gemma 4 26B-A4B on up to 24% of a benchmark's prompts; gpt-oss-20b (high) on up to 23% of a benchmark's prompts; Jamba2-3B on up to 18% of a benchmark's prompts; LFM2.5-24B-A2B on up to 15% of a benchmark's prompts; Olmo 3 7B Think on up to 12% of a benchmark's prompts. A cut answer scores 0, so their scores there partly measure that.
Vendor sampling differs: Liquid recommends temperature 0.1 for its LFM2.5 models, which ran that way; a control of LFM2.5-2.6B at temperature 1.0 is in the side controls below.

Side controls

Liquid recommends temperature 0.1 for LFM2.5-2.6B, the setting it ran at above. The same model at temperature 1.0, the most common setting among the other models here:

Setting	GPQA	Math	IFEval	IFBench	IFNovel
LFM2.5-2.6B	57.2	45.9	92.4	59.5	60.7
LFM2.5-2.6B at temperature 1.0	57.3	46.7	92.4	60.1	61.2

gpt-oss-20b at low and medium reasoning effort, the other side controls, are in the section on compute knobs.