What it is. The jump from multiple choice to open-ended QA: the answer space is unbounded, so the model can no longer enumerate it. It generates K candidate answers with weights plus an Other bucket for everything unlisted, summing to 1. Dataset: HotpotQA-Modified — 10 context paragraphs (2 gold + 8 distractors), with 0/1/2 gold paragraphs randomly dropped to induce genuine ambiguity. Model: Qwen3-4B, GRPO, 25k train / 500-sample calibration eval.

<hypotheses>
<h1>First candidate</h1><w1>0.40</w1>
<h2>Second candidate</h2><w2>0.30</w2>
<h3>Third candidate</h3><w3>0.20</w3>
<other>0.10</other>
</hypotheses>

Reward semantics. The backoff branch implements the induced distribution \(\hat p(\text{gold}) = w_0 \, p_{\text{LM}}(\text{gold}\mid x)\); the intractable denominator \(p_{\text{LM}}(\mathcal{Y}\setminus H \mid x)\) is dropped (weak function of H).

Design decisions

Feared failure mode — "w_other=1 as a safe hedge" — never happens. With one listed hypothesis at weight \(w\), expected reward is the binary log-score \(q\log w + (1-q)\log(1-w)\), maximized at \(w=q\): listing your best guess always beats pure Other. The pure-log-score local optimum is therefore "1 hypothesis + Other", which is exactly what collapse experiments show.
Diversity terms (the pre-framework hacks): hard min-K (penalty if fewer than 3 hypotheses) and a smooth soft-diversity variant \(\min(0, \log(n/K))\). Superseded in principle by the backoff, but still pragmatically useful (see Exp 9/10).
Equivalence-class matching: token-F1 ≥ 0.75 plus token-contiguous substring rule; all matching hypotheses' weights are summed. The earlier F1 ≥ 0.5 first-match rule produced ~9% spurious matches ("David Diamond" ↔ "David Weissman" via shared "David"); the fix brings this to ~3%, all legitimate variants.
Fail reward ordering is load-bearing: a valid-but-wrong output must never score below a parse failure, or the model learns to abandon the format (observed: format rate 94% → 47% in 21 steps when this was violated).

The backoff bug — and why "read the rollouts" is a project rule

Broken until 2026-04-25: a silent no-op chat-template patch. The precompute script "disabled thinking" via chat_template.replace("enable_thinking=True", "enable_thinking=False") — but the Qwen3 template never contains that literal string, so nothing changed. The base LM was then scored for \(p(\text{gold})\) at a position where it expects to open a <think> block: single-token answers like "no" or "Scotland" got log-probs of −30 to −42. Every backoff experiment (Exp 7/8) trained against this pathological prior.

The fix: append a literal "Final answer: " prefix after the chat template before scoring gold. Diagnostic over prompt variants (200 examples, per-token mean \(\log p(\text{gold})\)):

Prompt variant	per-token mean	median
V0 — broken baseline (assistant + think position)	−15.24	−13.46
V5 — thinking disabled only	−15.80	−12.55
V1 — "Final answer: " prefix (chosen)	−1.53	0.00
V2 — raw QA format	−1.18	0.00

On the full 5k-example precompute: total \(\log p\) mean −43.92 → −4.64; per-token −14.66 → −1.33. The "gold missed" reward is now comparable in magnitude to the "gold listed" reward, as the theory wants.

Experiment grid

Exp	Config	Acc	top1 ECE	Brier	# hyps	w_other	Takeaway
Base	Qwen3-4B untrained	0.499	0.361	0.347	1.9	—	overconfident
1	log + CW	—	—	—	1.0	0.43	collapses without diversity
2	log + CW + min-K (best ckpt s600)	0.547	0.046	0.169	3.0	0.30	best overall
3	pure log	—	0.099	—	1.1	0.39	self-calibrating but collapses
4	log + min-K	0.505	0.043	0.181	3.0	0.28	CW adds ~2% acc
5	Brier + CW + min-K	0.515	0.060	0.176	3.0	0.29	log > Brier
6	log + CW + soft-diversity	0.516	0.069	0.181	3.0	0.25	smoothest w_other plateau
7v2	backoff, no min-K (broken precompute)	0.506	0.101	0.189	2.0	0.30	partial collapse
8	backoff + min-K (broken precompute)	0.499	0.083	0.179	2.9	0.26	no gain over min-K
9	fixed backoff, no min-K (5k train)	0.476	0.063	0.175	2.9	0.25	diversity without min-K
10	fixed backoff + min-K (5k train)	0.520*	—	—	2.7	0.22	*training-eval; calib eval pending

Evals at step 300 (Exp 2: step 600) on 500 validation samples, temp 0.6. Exp 9/10 used 5k train samples vs 25k for Exp 2 — scale confound when comparing accuracy.

The headline of Exp 9 (2026-04-25). With the corrected backoff, the model maintains ≈2.9 hypotheses with no min-K hack at all (vs 2.0 under the broken precompute, and 1.0–1.1 with nothing). The principled formulation does the job the hack was doing. Caveat: Exp 10 suggests min-K may still buy accuracy (0.520 vs 0.470 training-eval), and the broken-era conclusion "backoff adds nothing over min-K" needs the scale-matched rerun before it's final.

Calibration must be viewed three ways

The historical metric, top1_ece, looks only at the heaviest hypothesis — it never touches w_other and hid systematic under-confidence. The eval now reports:

top1_ece — heaviest-hypothesis calibration (Exp 2 s600: 0.076)
p_gold_ece — uses w_other when gold is unlisted (same model: 0.424 — large hidden underconfidence)
p_gold_backoff_ece — full induced distribution \(w_0 \cdot e^{\text{base-logprob}}\)

Open items

Scale-matched Exp 2 vs Exp 9 comparison (5k/300 vs 25k/600 confound).
Exp 10 calibration eval (training done; eval crashed at GPU init).
BPE-boundary outliers: ~2% of answers tokenize as a single merged token at the prompt boundary, distorting the auto-computed fail reward (−42 instead of ≈−10); switch to a percentile.
Jacob's residual (include \(w_0\, p_{\text{LM}}\) even when gold is listed) implemented but untrained; semantically cleaner, numerically tiny (+0.036 at \(w_0\)=0.3).
Other-bucket mass currently double-counts listed answers in \(p_{\text{LM}}\) — documented approximation (clean fix needs per-rollout precompute).
Full 7,405-sample validation eval; LoRA run of the best config.

Status: analysis phase — no runs active. Last activity 2026-04-25 (audit + Exp 9/10). Per the 2026-05 meetings, exact-match was reinstated for HotpotQA (fuzzy matching removed → more stable), and the LM-prior backoff is the framework-level formulation going forward.