What it is. The foundation everything else builds on: replicate Bereket et al.'s result (via Shepard's LoRA branch) that GRPO with a log-score reward teaches Qwen3-4B to output a calibrated distribution over the four MedMCQA options instead of a single letter.

<hypotheses>{A: 0.40; B: 0.30; C: 0.20; D: 0.10}</hypotheses>

Reward is \( R = \log p(\text{gold}) \) — the categorical log-score over the four options. Training: GRPO, group size 16, effective batch 512, LoRA r=64 on Qwen3-4B with vLLM colocated for generation.

Results: replication achieved

Metric (4,183 val samples)	Base Qwen3-4B	Shepard LoRA 240s (reference)	Ours: LoRA 5e-6, 450s
Format rate	0.865	0.998	0.994
Accuracy	0.556	0.585	0.580
ECE	0.104	0.019	0.034
Mean log-score	−1.40	−1.04	−1.10

Our configs	Steps	ECE	Acc	Fmt	Log score
LoRA lr=5e-6 (reproduction)	450	0.034	0.580	0.994	−1.100
LoRA lr=1e-5 cosine (3.75× faster)	120	0.032	0.550	0.966	−1.272
Full FT lr=5e-6 (peaks then regresses)	100	0.054	0.553	0.977	−1.233

LoRA vs full fine-tuning (also verified on GSM8K) LoRA improves monotonically with no regression — its implicit regularization prevents overspreading. Full FT peaks around step 100, then accuracy drops and ECE regresses, even with KL regularization. Working hypotheses: weight decay penalizes absolute weight norm rather than distance-from-base; harder optimization landscape; forgetting. Principled alternatives flagged: distance-to-base and KL-to-base regularization.

Hard-won technical lessons (inherited by every later project)

vLLM sleep mode is broken in TRL 1.1.0.dev0 — corrupts weights on wake, producing garbage generations. Always vllm_enable_sleep_mode=False; weights sync in-place instead.
loss = 0 is normal for Dr. GRPO — with advantages centered per group and importance ratios ≈ 1, the loss value collapses to 0; learning happens through the (non-zero) gradients.
Qwen3 thinking mode must be disabled via chat-template patching, or long <think> chains eat the token budget before the distribution is emitted.
Learning-rate window is narrow: LoRA 1e-6 doesn't converge on this TRL version, 5e-6 converges in ~450 steps, 1e-5 cosine in ~90–120, 1.5e-5 goes overconfident.
bf16 off for LoRA (fp32 needed for small-lr adapter updates); bf16 on for full FT (memory).
Zero-advantage filtering: completions whose reward equals the group mean (e.g., when the whole group fails to parse) are excluded from the loss to remove gradient noise.

Code lives at the repo root (train.py, reward.py, patches.py, research_log.md). Status: complete; serves as the baseline-replication anchor for the paper.