What it is. The newest line (built 2026-05-28 → today): general distributional regression. The model answers numeric questions — house prices, Fermi estimates, patient survival times — with a full density over the target, scored by the log density \(\log p_{\text{model}}(y^{*})\). Two expressiveness levels are compared: an explicit mixture-of-Gaussians XML format, and pluck programs — small probabilistic programs whose exact marginal the pluck engine (Alex's Rust PPL, Python bindings) computes, inducing the density for scoring. Model: Qwen3-4B.

Datasets (6 built, 4 headline)

Methods — five formats, one comparable score

Every method induces a real density, so log scores are directly comparable across formats:

A pluck program for a mixture-cure survival model — the harness wraps the body in a (query …) scaffold, runs exact inference, and scores the resulting mixture:

(let ((cured (flip 0.3))
      (baseline (gaussian 4.0 1.5)))   ; std-dev, not variance
  (if cured 100.0 baseline))
; → exact marginal: 30% point mass + 70% · N(4.0, 1.5)

Phase 2 — base-model baselines (n=200/cell, temp 0.7, 2026-05-28)

Log-score mean (primary; higher is better)

Readings. MoG-XML wins the mean log score on 3/5 datasets. The mean-vs-median gap on calibrated_ci and mog_xml is itself informative: means are dominated by a small fraction of catastrophically overconfident outputs — the proper scoring rule punishing overconfidence. Calibrated-CI replicates FermiEval's "perception tunnel": asked for a 90% interval, the base LM emits one ~10× too narrow (9% coverage on openestimate/survival). And beta parses only ≈50% of the time (the model writes "I'd say Beta(3,5)" as prose) — so beta is a format-discipline test, not a scoring-rule test; it is excluded from Phase 3 training.

Example rollouts — base model (click to expand)

Phase 3 — GRPO training on MoG-XML completed 2026-06-11

Results — held-out 200 Phase 2 cells, matched-subset (best checkpoint)

All numbers below are computed on the pairwise matched subset — the examples both the base and trained model parse — via matched_report.py. Parse rate is shown separately because the base model abstains (fails to parse) on the hard examples, so a parse-conditioned mean would flatter it (see the methodological lesson below).

fermieval nuance: training improves the matched mean (−6.33 → −3.88, far fewer catastrophic tail misses, abs-err 6.96 → 3.70) but the median slips (−1.17 → −1.55) — mild over-sharpening on the typical question while the tails tighten. The other two datasets improve on every metric.

Example rollouts — trained MoG-XML (click to expand)

Design

• Reward: \( R = \mathrm{clip}(\log p_{\text{MoG}}(y^{*}),\, -50,\, +10) \) on parse success; parse failure = −55 (strictly below the floor, so abandoning the format is never optimal — the DDXPlus/HotpotQA lesson). Censored survival rows score \(\log S(y)\). The clip floor bounds catastrophic-outlier gradients (one −10⁵ log-density would dominate a batch) while preserving direction. The reward calls the same parse/density code as the Phase 2 eval, so reward and metric are identical by construction.
• Config: LoRA r=64, lr 1e-5 cosine, 300 steps/dataset, group 16, effective batch 512, vLLM colocate; checkpoints every 25 steps, resume-safe (already survived one crash and one power loss today).
• Split protocol: the first 200 examples of the fixed seed-0 shuffle — the exact Phase 2 eval cells — are never trained on; during-training eval uses the first 100 of them every 50 steps. Post-training: merge LoRA, run the full 200-cell Phase 2 eval, compare against the baseline tables above (e.g. ames mog_xml base = −0.707).

Phase 4 — pluck programs: GRPO trained in progress 2026-06-18

The model emits a Pluck program; exact inference induces its density, scored by the same clipped log-score reward as Phase 3 (reward_pluck.py caches each inference so the reward + 5 metric heads cost one program execution, ~1 ms/rollout). Training order survival → fermieval → ames, 300 steps each, LoRA. Two things resolved a prior open question:

• No format curriculum needed. GRPO lifts the pluck parse rate on its own — survival 0.70 → 1.00 by step ~26, exactly like MoG-XML. An improved-prompt variant (pluck_v2: explicit no-division / binary-+. rules + a log-space recipe) was tested zero-shot and hurt (survival parse 0.70 → 0.33; the denser prompt worsened <think> overrun), so it was dropped — the base model needs fewer rules, not more; GRPO is what fixes the language gaps.
• Trained pluck ≈ a MoG expressed as a program. Reading the rollouts (not just the metric): the model branches with flip/if, but between two gaussians, inducing a gaussian mixture (≈1.8 components). It does not reach for point-mass "cure" leaves — the one structure a plain MoG cannot express.

Trained pluck vs trained MoG, matched subset (best checkpoint each)

The fair comparison is trained-vs-trained on the full held-out 200 cells (both parse everything). Base baselines (parse-conditioned) shown for context. All three datasets done.

Follow-up: cure-exemplar probe — does showing structure unlock it?

A survival run (--method pluck_cure) adds one worked exemplar showing a point-mass cure branch ((if no-event 99.0 (gaussian 4.0 1.5))), everything else identical to v1, to separate an exploration/prompt limit from a reward limit. The result is nuanced — and surprising.

Variance control (2 runs each). v1 pluck across two runs lands at −2.10 and −2.17/−2.22; the cure model across two runs lands at −1.85 and −1.92. Both cure runs beat both v1 runs — the distributions don't overlap, so the ~0.27 nat cure gain is real and reproducible (cure ~−1.90 > trained MoG −1.99 > v1 pluck ~−2.16). But all runs share a median ~−1.70…−1.79, so the gain lives in the mean (tail robustness, coverage 0.86–0.91), not the typical case — a better-calibrated prior, not correct deployment of the cure structure.

Example rollouts — pluck programs (click to expand)

The failure traces below show the two distinct ways a pluck rollout fails: a program is emitted but the language rejects it (variadic +., division, scientific notation), or <think> overruns the budget so no program appears at all.

Model-size scaling — does a bigger model use pluck's structure?

The central finding (pluck never realizes structure) could be a capacity limit. To test it, the same survival pluck GRPO was run at 4B, 8B, and 14B (the largest that fits this 96 GB GPU under vLLM colocate). Held-out 200 cells, best checkpoint each:

Engineering notes

• Pluck runs sandboxed: a worker process survives Rust panics; a deliberate 2²²-path timeout bomb is killed and respawned in <5s; steady-state ~0.6 ms/program.
• Bindings built into the repo-root venv on Euler (rustup user-local + maturin develop --release) — generation, scoring, training, and eval all run from one node since today; the Mac round-trip is retired.
• The whole session is one idempotent script: bash training/gpu_session.sh 300 (skip-if-done pluck generation → in-place scoring → three sequential resumable trainings).

Open questions

• Pluck GRPO go/no-go: trained MoG-XML now beats the untrained pluck baseline even on survival (−1.99 vs −6.66), so the fair fight is trained-vs-trained — does pluck training close the gap, and is the format curriculum (few-shot scaffolding for the 0.47 fermieval parse rate) worth building first? Should the language accept scientific notation?
• Fresh post-cutoff Metaculus pull (needs API token) to replace the leaky split.
• what_are_the_odds label regeneration → would add a 5th headline dataset.