Post-training · MathCAMPS · nanochat-d34

Does Relaxing SFT Compute-Matching Close the Pretraining Gap?

The three-track experiments deliberately compute-matched SFT mixes at 7,878 rows so filtered/unfiltered comparisons were apples-to-apples — capping Bespoke-Stratos at 30/2,860 rows and Reasoning Gym at 565/5,712 to match LittleLearner's smaller K-5 pools. A reviewer asked the natural follow-up: is that cap hiding a real fix? This drops it entirely.

Eval: MathCAMPS no-ICL, N=128, T=0.8, top_k=50 Mix: sft_v2.jsonl, 15,599 rows (all available unfiltered SFT data, no cap) Date: 2026-07-27
Answer No. Dropping the compute-matching cap doesn't let LittleLearner catch up to General — on >K-5 the gap actively widens as more data is added, because General benefits far more from the extra data than LittleLearner does.

What changed

The compute-matched mixes (sft_v3_unfiltered.jsonl / sft_v3_filtered.jsonl, 7,878 rows each) capped every source to the smaller of the two tracks' natural pool sizes. sft_v2.jsonl (confirmed via assemble.py) is simply every available unfiltered-source row at its natural size — no cap, no oversampling:

Compute-matched (7,878 rows)
  • GSM8K richcot: 5,027
  • Bespoke-Stratos: 30 (of 2,860 available)
  • Reasoning Gym: 565 (of 5,712 available)
  • IFEval: 2,256
Unbounded (15,599 rows)
  • GSM8K richcot: 5,027
  • Bespoke-Stratos: 2,860 (full pool)
  • Reasoning Gym: 5,712 (full pool)
  • IFEval: 2,000

Same unbounded mix trained onto both bases with each base's own established recipe (General: lr=3e-5; LittleLearner: lr=1e-5; 3 epochs, batch 4 × grad_acc 4). The LittleLearner + unbounded-General-SFT run is the "hybrid, no GRPO" arm.

Pass@k
≤K-5 >K-5
Stage p@1p@10p@128 p@1p@10p@128
General base 4.80%17.63%37.20% 0.19%1.04%4.46%
General SFTdcompute-matched, sft_v4 (7,878 rows) 2.54%−2.26 16.74%−0.89 45.66%+8.46 0.24%+0.05 1.46%+0.42 5.69%+1.23
General SFTdUNBOUNDED, sft_v2 (15,599 rows) 2.84%−1.96 18.70%+1.07 48.51%+11.31 0.20%+0.01 1.43%+0.39 6.69%+2.23
LittleLearner base 1.15%7.57%24.18% 0.17%1.26%3.68%
LL SFTdcompute-matched, sft_v3 (7,878 rows) 1.05%−0.10 8.57%+1.00 34.48%+10.30 0.11%−0.06 0.88%−0.38 4.24%+0.56
LL + unbounded General SFThybrid, no GRPO — sft_v2 (15,599 rows) 1.21%+0.06 9.26%+1.69 35.31%+11.13 0.10%−0.07 0.74%−0.52 3.79%+0.11

Deltas are vs. that track's own base. Highlighted rows are the new unbounded runs.

The General − LittleLearner gap, stage by stage (p@128)
≤K-5 gap
13.0ppbase
→
11.2ppcompute-matched
→
13.2ppunbounded

Both tracks gain almost identically from unbounded data (General +11.3pp, LL +11.1pp) — the gap reverts to essentially its base-level width. The compute-matched cap had incidentally narrowed it slightly; removing the cap undoes that, it doesn't produce new narrowing.

>K-5 gap
0.8ppbase
→
1.5ppcompute-matched
→
2.9ppunbounded

Monotonically widening. General's >K-5 gain from unbounded data (+2.23pp) is 20× LittleLearner's (+0.11pp, within noise). More post-training data helps the better-pretrained base more, not less.

Health check: neither new checkpoint collapsed — modal-answer share (the same probe used in the GRPO budget-scaling audit) is 4.8% for General-unbounded and 3.2% for LL-unbounded, both comfortably under the ~20% healthy threshold. This is a clean comparison, not an artifact of one side degenerating.