One sentence: we train language models, with reinforcement learning whose reward is a proper scoring rule, to stop emitting a single answer and instead emit a calibrated probability distribution over answers — and we show this works across multiple-choice QA, open-ended QA, medical diagnosis, forecasting, and continuous regression.

Tabs: each project line is self-contained. 1·MCQ replication is the foundation (replicating Bereket et al.); 2·HotpotQA extends it to open-ended answers with an "Other" bucket; 3·DDXPlus trains against full gold distributions (49 diseases); 4·Forecasting moves to continuous targets (Polymarket, Metaculus) with Beta-distribution outputs; 5·Regression-PPL generalizes to mixture-of-Gaussians and probabilistic programs (pluck); Live status shows what is running right now.

The framework (current, after the 2026-05 meetings)

The model reasons freely, then commits to a structured distribution: a short list of named hypotheses with probabilities, plus a residual Other bucket.

<think> ...reasoning... </think>
<hypotheses>
  <h p="0.40">Pneumonia</h>
  <h p="0.30">Bronchitis</h>
  <other p="0.30"/>
</hypotheses>

Listed hypotheses keep their stated weight; the Other bucket spreads its mass over all unlisted answers using the base LM's prior (the LM-prior backoff):

\[ \hat p(a \mid x) \;=\; \begin{cases} w_i & a = h_i \text{ (listed)}\\[2pt] w_{\text{other}}\cdot p_{\text{LM}}(a \mid x) & \text{otherwise} \end{cases} \]

Training reward is the cross-entropy log-score against the gold distribution \(p^*\) (a point mass for single-answer tasks, a full differential for DDXPlus):

\[ R \;=\; \sum_a p^*(a)\,\log\bigl(\hat p(a) + \epsilon\bigr) \]

The pipeline. The reward is a strictly proper scoring rule — its expectation is maximized exactly when the model reports its true beliefs.

Why this matters: no behavioral hacks. Earlier iterations needed hand-tuned penalties (hypothesis-count / min-K constraints, soft-diversity bonuses) to stop the model collapsing to one answer or hiding in the Other bucket. The LM-prior backoff gives Other real semantics instead of a black-hole sink, and empirically the cross-entropy objective alone is stable: the model spreads when gold is spread, and commits when gold is concentrated. Formalizing why this objective avoids collapse is the conceptual core of the paper (open theory item).

The cleanest empirical validation: ProtoQA vs TriviaQA

Same objective, opposite learned behaviors — exactly what a proper scoring rule should produce:

Dataset (gold structure)	# hypotheses	w_other	key metric
ProtoQA (multi-cluster gold → should spread)	stays ≈ 3.4	≈ 0.12	coverage 0.07 → 0.20
TriviaQA (single answer → should commit)	3.0 → 1.0	0.02 → 0.48	p_gold 0.21 → 0.55

Project map

Tab	Task / gold structure	Status	Headline
1 · MCQ replication	MedMCQA A–D, single gold	complete	Reproduced Shepard/Bereket: ECE 0.36→0.03 at equal accuracy
2 · HotpotQA	open-ended QA, single gold	Exp 1–10 done	ECE 0.36→0.046; backoff bug found & fixed; principled backoff ≈ min-K hack
3 · DDXPlus	distribution over 49 diseases	Exp 1–5 + TVD audit	Cross-entropy beats every alternative reward; TVD 0.69 is semantic, not formatting
4 · Forecasting	binary + continuous, market-resolved	Tracks 1–2 done	Beta-format 4B ≈ o1 on Brier; mixtures capture 82% of oracle gain on branching
5 · Regression-PPL	continuous targets, censoring, Fermi	Phase 3 training now	MoG-XML wins 3/5 baselines; pluck (exact-inference PPL) beats MoG on survival

Shared findings across projects

Log score > Brier as a reward, consistently (HotpotQA, forecasting): the log score's unbounded penalty disciplines overconfidence harder.
Pure log score is self-calibrating — even with no corrections, ECE drops ~4–8× from base model on every task tried.
Stability depends on scale and structure: Qwen3-4B needs the Other-bucket semantics (or a diversity term) on open-ended tasks; Qwen 34B avoids collapse naturally.
LoRA ≥ full fine-tuning for this objective: LoRA improves monotonically; full FT peaks (~step 100) then overspreads. Hypotheses: weight decay penalizes absolute norm rather than distance-from-base; harder optimization landscape; forgetting. Distance-to-base / KL-to-base regularization flagged as principled fixes.
Always read rollouts, not just metric tables — every major bug (chat-template no-op, off-list diagnoses, prose-instead-of-XML) was found by reading outputs.

Output-representation comparison (DDXPlus, team-level result)

Representation	KL ↓	Mode acc ↑	ECE ↓	Fmt/exec
CoT (verbalized list)	2.17	0.51	0.41	0.97
DSL (class-hierarchy spec)	1.74	0.43	0.16	0.99
Pyro (writes `def model()`)	2.05	—	—	0.98

DSL wins KL and ECE but trades mode accuracy; Pyro writes flat programs — it never reaches for hierarchical structure or noisy-OR (open diagnosis: few-shot demos vs reward design).

Measurement standards

Total variation distance is reported alongside KL on closed-set tasks because it has two readable interpretations:

\[ \mathrm{TV}(P,Q) = \tfrac12 \sum_x |p(x)-q(x)| \;=\; \sup_A |P(A)-Q(A)| \;=\; \inf_{\text{couplings}} \Pr[X \neq Y] \]

Set view (Scheffé): TV = 0.70 means there is a set of answers where gold places 70% more mass than the model; the maximizing set is \(\{x : p(x) > q(x)\}\).
Coupling view: \(1-\mathrm{TV} = \sum_x \min(p(x), q(x))\) is the "paint overlap" between the two distributions.

Reporting rules: always give validation reward next to accuracy and ECE (the reward jointly captures calibration + accuracy); report TV alongside KL on closed sets. Caveat: Tim Vieira flagged literature questioning ECE as the right calibration metric — under review.

Matched-subset reporting (project-wide standard, adopted 2026-06-18). When a model can abstain by failing to produce a parseable answer, parse failure is informative: models decline the examples they find hardest. A metric averaged only over parsed outputs is therefore selection-biased, and is not comparable across models or methods with different parse rates (a model that answers only the easy 60% looks better than one that answers all 100%). The standard is to compare on the matched subset — the examples every series parses — and to quote each series' parse rate beside the metric. This first bit us on Regression-PPL (a trained model looked worse than base on fermieval purely because base abstained on the hard questions); the same commit-vs-abstain effect underlies the forecasting Beta-format result. Tooling: matched_report.py.

The broader vision (Alex)

LM outputs as a spectrum of increasingly expressive uncertainty representations:

Aspirational endpoint: the model emits a latent-variable diagnosis simulator; conditioning it on symptoms yields the posterior. Pluck (Alex's exact-marginalization PPL, unlike approximate Pyro/NumPyro/Gen) is the candidate engine. The cross-entropy + Other framework is the ICLR-scoped contribution; this is the long arc.

Glossary

Other bucket	residual-mass token whose probability is distributed over unlisted answers via \(p_{\text{LM}}\)
LM-prior backoff	using the base model's logit probabilities to spread Other-bucket mass
Coverage	fraction of gold mass captured by listed hypotheses
Mode accuracy	argmax of predicted distribution matches argmax of gold
Proper scoring rule	loss minimized in expectation at the true distribution (here: log-score / cross-entropy)
RLVF	RL with verifiable feedback — the single-correct-answer baseline
GRPO / Dr. GRPO	group-relative policy optimization: sample G completions per prompt, advantage = reward − group mean (Dr. variant: no std normalization)
ECE	expected calibration error — gap between stated confidence and empirical accuracy
Pluck	Alex's probabilistic programming language doing exact marginalization (Rust, Python bindings)