What it is. The only task where gold is itself a distribution: DDXPlus (Fansi Tchango et al., NeurIPS 2022) is a synthetic medical-diagnosis benchmark with a closed set of 49 pathologies and a weighted differential diagnosis per patient (median ~8 plausible conditions). Prior work (Puri et al. 2025) collapses the differential to binary set membership; we train Qwen3-4B (GRPO, LoRA) against the actual probability weights — distribution-to-distribution. Patient evidence arrives as codes (E_135_@_V_3) and is mapped to readable symptom descriptions; the system prompt enumerates all 49 condition names (without this the model invents non-matching terms and the reward signal is too weak to learn from).

Design

Reward — cross-entropy against the gold differential (strictly proper; reduces to \(\log p(\text{correct})\) when gold is a point mass): \[ R = \sum_{j} p^*_j \,\log\bigl(\max(\hat p_j,\ \epsilon)\bigr) \]
\(\epsilon\) must be exactly 1/49 — i.e., a missing condition is scored as if under a uniform prediction (\(\log 1/49 = -3.89\) = fail reward). With smaller \(\epsilon\), wrong-but-formatted outputs score below a parse failure and the model learns to break the format.
No Other bucket — the set is closed; unlisted conditions get \(\epsilon\).
Variable-length output, same <hypotheses> XML as HotpotQA; the model typically lists ~4 conditions (gold has ~9).

Reward ablation: cross-entropy dominates everything

Metric (2,000-sample eval, best ckpt each)	Base	Exp 1 xent	Exp 2 xent+CW	Exp 3 Brier+CW	Exp 4 mode-only	Exp 5 set-coverage
Format rate	0.17	0.987	0.992	0.991	0.995	0.996
Mode accuracy	0.040	0.459	0.508	0.496	0.496	0.473
Top-3 coverage	0.030	0.439	0.456	0.414	0.436	0.298
KL ↓	2.436	1.042	1.012	1.081	1.056	1.217
TVD ↓	0.978	0.690	0.687	0.696	0.703	0.775
ECE ↓	0.190	—	0.114	—	0.133	0.434
Brier ↓	0.203	—	0.043	—	0.052	0.491
# conditions listed	—	4.0	4.0	4.0	4.1	1.3

Paper-ready claims. (1) Cross-entropy + correctness-weight wins on every metric. (2) Puri-style set-coverage reward causes mode collapse: 1.3 conditions at 0.89 max confidence, ECE 0.434 — set membership is fundamentally incompatible with calibration. (3) Mode-only RLVF underperforms distributional training even on mode accuracy itself (0.496 vs 0.508) — the secondary conditions act as a regularizer. This is the baseline that supports the headline claim that distributional training adds value. (4) A 4B model with distributional training matches Puri's 8B set-coverage results (50.8% mode accuracy).

TVD decomposition audit (2026-05-28): is 0.69 real disagreement or formatting noise?

KL ≈ 1.0 with TVD ≈ 0.69 sits essentially on the Pinsker bound \( \mathrm{TV} \le \sqrt{\mathrm{KL}/2} \) — which signals mass-in-wrong-places rather than mild miscalibration. The audit decomposed the per-sample mean TVD by Scheffé sets:

Scheffé / signed-gap aggregation over 2,000 validation patients, best model (Exp 2 step 250).

Reading the 30 highest-TVD patients (all TVD = 1.0) classifies the disagreement into three semantic modes:

1 — Base-rate / familiarity bias (most common)

The model defaults to prototypical internet-medicine respiratory diagnoses regardless of evidence. Over-predicted: Influenza (+4.4pp), Acute rhinosinusitis (+3.3), Viral pharyngitis (+3.2), URTI (+2.5), Pneumonia (+2.0). Under-predicted: Bronchitis (−3.5), Chagas (−3.1), Anemia (−3.1), Acute dystonic reactions (−2.3), Sarcoidosis (−2.0). Example: 6-year-old with sharp chest pain → gold: pulmonary embolism + pneumothorax + myocarditis; predicted: pneumonia + bronchitis + pharyngitis.

2 — Sibling-disease confusion (right organ system, wrong specific)

NSTEMI/STEMI ↔ unstable angina; laryngospasm ↔ croup/epiglottitis; acute otitis media ↔ rhinosinusitis variants. These look like token-overlap canonicalization problems but are clinically distinct diagnoses. This mode makes PE and NSTEMI appear on both sides of the macro ledger — correct rate, wrong patients.

3 — Off-list invention (closed-set violation)

2.7% of predicted mass per sample lands on names outside the 49 (12% of samples have any). Most common: "Heart failure" (2.4%), "Acute pharyngitis" (1.4%), "Psoriatic arthritis" (1.2%). Genuine canonicalization gaps ("Acute pharyngitis" → "Viral pharyngitis", "Myocardial infarction" → "Possible NSTEMI/STEMI") total under 1% of mass — fixing string matching would barely move TVD. Parse failures: 17/2000 (0.85%), all from <think> overrunning the 1,024-token budget.

Verdict for the paper. TVD ≈ 0.69 is not a measurement artifact: it reflects genuine diagnostic disagreement — prior over-shift toward common infections, under-discrimination within disease families, and a 2–3% closed-set violation rate. The levers are better intra-family discrimination and breaking the base-rate prior, not harder string canonicalization.

Open items

Test-split evaluation (134k samples) for final paper numbers; current finals are on 2,000 validation samples.
Explain the DSL representation's mode-accuracy drop (calibration-for-sharpness tradeoff or genuinely worse argmax?) and diagnose why Pyro writes flat programs — see the representation-comparison table in the Overview tab.
Data/model-scale levers against the base-rate prior (25k of 1M+ train examples used; all runs LoRA).
Optional: alias-canonicalization postprocessor (recovers <1pp TVD — probably not worth it).

Status: Exp 1–5 and the TVD audit complete. Artifacts: tvd_decomposition_exp2_s250.json, tvd_per_sample_inspection.md, analyze_tvd.py, inspect_per_sample_tvd.py.