Pass@k alone can't tell a real gain from a model that quietly stopped trying. This adds two health probes — answer diversity and parse rate — across every General / LittleLearner / Hybrid checkpoint evaluated in the 300–1500 step budget sweep, so collapsed runs are flagged instead of just under-performing silently.
Eval: MathCAMPS no-ICL, N=128, T=0.8, top_k=50Filter: paper eval-filter applied to pass@k onlyWindow: 2026-05-05 → 2026-07-28
How “fried” is decided
Healthy modal-answer share < 20% on both bands — the model's answers track problem diversity, as expected.Degraded partial collapse — modal share > 30% on one band, parse rate visibly down, but still emitting varied, mostly-coherent text.Collapsed modal share > 50% on ≥1 band — the model gives (near-)the same answer regardless of the question, or degenerates into repeated-token garbage.
finish_eos_frac is not the signal here. It reads 0.0 for every General/LittleLearner/Hybrid row — healthy or fried alike — because that eval script's hardcoded stop strings don't match this family's chat template; every completion runs to the 512-token cap regardless. The real tell is modal-answer share: the fraction of problems where the model's first-sample answer equals its single most common answer, computed on the raw grade-restricted eval set (N=3700 ≤K-5 / N=1200 >K-5, pre eval-filter — pass@k columns use the filtered numbers).
General · unfiltered pretrain
≤K-5 climbs cleanly through ~900–1000 steps at lr=2e-5, then flattens. 5× the LR (aggressive) breaks it at the same step count.
≤K-5
>K-5
Stage
Verdict
p@1
p@10
p@128
modal share
p@1
p@10
p@128
modal share
baseno post-train
Healthy
4.8%
17.6%
37.2%
3.0%1550 distinct · parse 98%
0.2%
1.0%
4.5%
3.1%767 distinct · parse 97%
GRPO 300 stepslr=1e-5 (v4 baseline)
Healthy
2.9%
18.0%
45.4%
4.6%1126 distinct · parse 100%
0.3%
1.6%
5.8%
3.9%597 distinct · parse 100%
GRPO 600 stepslr=2e-5 (moderate)
Healthy
7.1%
26.6%
46.7%
5.3%676 distinct · parse 100%
0.5%
2.2%
6.8%
4.5%427 distinct · parse 99%
GRPO 600 stepslr=5e-5 (aggressive)5× the moderate LR at the same step count — partial collapse; >K-5 eval was never run for this one
Degraded
0.5%
2.7%
10.1%
35.4%97 distinct · parse 66%
not evaluated
GRPO 900 stepslr=2e-5
Healthy
8.4%
28.1%
46.1%
5.1%538 distinct · parse 99%
0.5%
2.5%
6.2%
5.5%330 distinct · parse 94%
GRPO 1200 stepslr=2e-5
Healthy
9.1%
27.0%
42.8%
12.1%406 distinct · parse 98%
0.4%
2.2%
5.5%
14.1%276 distinct · parse 92%
GRPO 1500 stepslr=2e-5
Healthy
9.1%
27.0%
44.2%
11.7%398 distinct · parse 98%
0.7%
2.8%
6.6%
10.3%273 distinct · parse 92%
LittleLearner · K-5-filtered pretrain
The same lr=2e-5/600-step recipe that helped General collapses this base by step 600, and continuing training doesn't recover it — it degrades further. Dropping back to the safe lr=1e-5 avoids collapse, but 600 steps still quietly erases the 300-step gain rather than building on it.
≤K-5
>K-5
Stage
Verdict
p@1
p@10
p@128
modal share
p@1
p@10
p@128
modal share
baseno post-train
Healthy
1.1%
7.6%
24.2%
13.3%511 distinct · parse 94%
0.2%
1.3%
3.7%
11.6%268 distinct · parse 93%
GRPO 300 stepsbaseline recipe
Healthy
1.5%
9.5%
27.7%
9.8%638 distinct · parse 90%
0.1%
0.8%
3.7%
19.8%392 distinct · parse 82%
GRPO 600 stepslr=1e-5 (fresh from sft_v3_filtered_base, isolation test)Doesn't mode-collapse — modal share stays near the healthy range, parse rate is actually higher than the 300-step checkpoint (99%). But every single pass@k number is now below the 300-step baseline: le_k5 p@128 is statistically indistinguishable from the untrained base (24.06% vs 24.18%), and gt_k5 p@128 drops below base (3.68% → 2.45%). 600 steps at the "safe" LR quietly erased the 300-step gain instead of building on it — a third failure mode, distinct from both General's clean improvement and the mode-collapse below.
Degraded
1.4%
8.8%
24.1%
18.6%610 distinct · parse 99%
0.1%
0.7%
2.5%
20.8%353 distinct · parse 99%
GRPO 600 stepslr=2e-5 (moderate)Mode-collapsed: answers ‘1’ on ~94% of ≤K-5 problems, regardless of the question.
Collapsed
1.9%
1.9%
2.0%
94.4%5 distinct · modal '1'
0.2%
0.2%
0.2%
85.3%3 distinct · modal '1'
GRPO 900 stepslr=2e-5, continued from the 600-step checkpointDegenerated further into literal token-repeat garbage — sample completion: “Step matched matched matched matched…” repeated for all 512 tokens. Parse rate craters.
Collapsed
0.7%
1.7%
2.3%
56.2%4 distinct · parse 36%
0.03%
0.1%
0.2%
74.3%3 distinct · parse 18%
Hybrid · K-5-filtered pretrain + unfiltered SFT
Same story as LittleLearner at 600 steps — a different, but equally total, collapse. Confirmed it isn't an LR artifact: a fresh run at the original lr=1e-5 collapses too, with a third distinct failure signature. A reviewer-suggested curriculum (introduce grades 6→7→8 one at a time instead of all at once) fixes the collapse entirely and beats the healthy baseline on ≤K-5 — but >K-5 still doesn't move.
≤K-5
>K-5
Stage
Verdict
p@1
p@10
p@128
modal share
p@1
p@10
p@128
modal share
base= LittleLearner base (shared checkpoint)
Healthy
1.1%
7.6%
24.2%
13.3%511 distinct · parse 94%
0.2%
1.3%
3.7%
11.6%268 distinct · parse 93%
GRPO 300 stepsbaseline recipe
Healthy
1.4%
10.2%
31.1%
6.6%780 distinct · parse 93%
0.2%
1.0%
3.9%
6.4%483 distinct · parse 92%
GRPO 600 stepslr=2e-5 (moderate)Mode-collapsed onto a repeated-digit string (‘555555…’, 200+ digits) — not word-repetition like LittleLearner's, but the same reward-hacked-format-without-solving-anything failure.
Collapsed
0.01%
0.04%
0.18%
37.8%125 distinct · parse 100%
0.00%
0.00%
0.00%
46.1%74 distinct · parse 100%
GRPO 600 stepslr=1e-5 (low-LR isolation test)Fresh from sft_v3_mixed (not warm-started from the lr=2e-5 collapse above) — isolates whether the elevated LR was the cause. It wasn't. Caught via a fast 12-problem/1-sample probe instead of the full N=128 sweep (the sweep's own throughput signature already matched the known-fried pattern): 12/12 wrong, looping a generic template — “Step 1: Find the sum… Step 9: Find the final answer…” — without computing anything. A third distinct failure signature (templated step-counter babble, vs. word-repeat and digit-repeat above).
Collapsed
full N=128 eval not run — stopped after the probe confirmed collapse, to avoid burning ~5h+ of GPU time on a checkpoint already known to be fried
not evaluated
GRPO 600 stepsCURRICULUM: K-6 → K-7 → K-8, 200 steps each, lr=1e-5Reviewer hypothesis test: stage the grade introduction instead of dumping the full K-8 pool at once. Fresh from sft_v3_mixed, K-5 synth present throughout, grades 6/7/8 added one at a time. Result: collapse is fully avoided (best modal-share/diversity numbers in the whole Hybrid track) and ≤K-5 beats even the healthy 300-step baseline (32.3% vs 31.1% p@128). But >K-5 p@128 lands at 3.90% — exactly tied with the 300-step baseline's 3.90%. Staging fixes the instability and gives a real ≤K-5 bonus; it does not move the >K-5 ceiling at all.