Unified report (condenses the R1–R18 study and the R19–R23 overnight "handoff ladder" study;
details live in report.md, report_ladder.md, docs/results_log.md).
TL;DR. On a controlled visual state-tracking task across four open unified multimodal models (UMMs): (1) Mental simulation dies after ~1–2 steps in every architecture — so "thinking in images" targets a real bottleneck. (2) Models can use a visual state that is handed to them as a single image — they ground in it causally, and where they don't, privileged distillation fixes it (provided training covers all "rungs"; narrow training silently breaks neighboring skills). A deep 54k-item run drives handoff to a perfect, causally-grounded ceiling across task widths and extends the mental horizon to ~2–3 swaps — but no further, and never beyond the trained depth: in-context simulation saturates. (3) The same information presented as an interleaved image sequence breaks some architectures (Janus) and not others (Emu3, Show-o2-7B) — the "interpretation error" of MentisOculi is a sequence-binding property of specific architectures, not a universal law. (4) The unsolved wall is generation: self-generated states look clean but their content is random (≈ chance from the very first step), resisting 12 fix attempts. Fix generation and the rest of the pipeline is ready to convert it into accuracy.
UMMs generate interleaved text+images, so they could "think with images" — generate a picture of an intermediate state, then reason on top of it (visual chain-of-thought). The team's MentisOculi benchmark (arXiv 2602.02465) showed this doesn't beat text-only reasoning and split the failure in two: the generation error (the image produced is wrong) and the interpretation error (the model doesn't use even a correct image). This project reproduces, decomposes, and attacks both on a minimal controlled task, with causal dependence (not accuracy alone) as the success criterion, and privileged distillation (teacher sees oracle images, student doesn't) as the main training tool.
Task — SlotSwap. n_slots colored disks in a row; a sequence of swaps ("Step 1: swap
slot 2 and slot 3. …"); question: "what color disk ends in slot q?" (chance = 1/n_slots =
0.25 here). Procedural and exactly verifiable, and every intermediate state can be rendered —
a free oracle visual chain-of-thought. All evals: constrained first-token scoring over
the color words (no free-text parsing); train and eval seeds disjoint.
| initial state | final state (after 3 swaps) |
|---|---|
|
|
Models (three image-generation families):
| model | understanding path | generation | role here |
|---|---|---|---|
| Show-o2-1.5B / 7B | VAE latent → und+gen embedders → fusion | flow-matching head in Wan-VAE latent space | the trainable target |
| Janus-Pro-7B | SigLIP encoder (decoupled from gen) | discrete VQ tokens, AR | eval: decoupled-arch datapoint |
| Emu3-Chat-8B | one unified discrete-token stream (video-pretrained) | discrete VQ tokens, AR | eval: unified-discrete datapoint |
| Chameleon-7B / Anole-7B | early-fusion discrete | discrete VQ | excluded — safety tune refuses/can't read the task (P≈0.5 / 0.1) |
Everything below is one instance with n swaps and oracle renders s_0…s_n. Each condition controls which visual state the model gets and how:
| condition | model sees | must do | isolates |
|---|---|---|---|
| P | final frame + "what color in slot q?" | read | perception ceiling (=1.00 everywhere) |
| A / A-curve | initial frame + swap list (r swaps) | simulate r swaps in ONE forward pass | mental-simulation horizon acc_A(r) |
| S_j | ONE frame = state after step j, a sentence saying so, + the full swap list | pick up the state, simulate the remaining r = n−j swaps | single-image state handoff |
| O_j | initial frame + swap text with oracle frames interleaved after steps 1…j | same as S_j, but in-sequence | interleaved binding |
| F_j | S_j with a wrong (counterfactual) frame | — we watch whether the answer follows the image or the text | causal dependence |
| B (teacher-forced) | oracle prior frames; model generates the next state; we read the answer off its own image | conditional generation | generation accuracy |
| B_free | initial frame only; model generates the whole chain from its own outputs | full self-generated visual CoT | generation + error compounding |
| ~~C1~~ (retired) | final frame + redundant swap text | read despite distraction | only distraction susceptibility — superseded: it is the j=n rung of S |
The key metric — residual-load collapse. A model that perfectly uses a handed visual state satisfies acc(S_j at n swaps) = acc_A(n−j): plotted against residual load r = n−j, every ladder point falls on the A-curve. Flat-in-j instead = the images are ignored. The uptake index U_j = (S_j − A(n)) / (A(n−j) − A(n)) ∈ [0,1] quantifies partial use.
4.1 The mental-simulation horizon is ~1–2 swaps, in every architecture. A-curves are steep and near-identical across all four models: 1.00 → 0.52–0.60 (r=1) → ~0.5 (r=2) → ≈chance (r≥3). Scale only helps the shallow end (7B: A(1)=0.92 vs 0.60). One forward pass buys about one swap — this is the quantitative case for step-by-step visual CoT. A text-CoT control (think step-by-step before answering) rescues depth only for Show-o2's Qwen backbone (0.58 vs 0.20 at r=4), not Janus (0.32), and Emu3 can't CoT at all (0.00) — for some backbones the visual channel is the only scalable state store.
4.2 Single-image handoff works — and is causally grounded. S-ladders rise monotonically and roughly collapse onto the A-curve. At the top rung (j=n: the frame shows the answer, with provenance) Janus = 1.00 and Emu3 = 0.96, and on counterfactual frames both follow the image 100% / text 0% (F_n) — exactly the causal dependence the project set out to measure. Show-o2-1.5B is the partial case (top rung 0.68, the old "interpretation error", now visible as one anomalous rung; flips reach only 0.56/0.16). Mid-rung uptake is real everywhere (U ≈ 0.4–1.0): a correct intermediate frame does reduce the remaining mental burden.
4.3 Interleaved presentation is the real, architecture-dependent failure.
| model | S_4 (single image) | O_4 (interleaved) | O_4 with frame labels |
|---|---|---|---|
| Janus-Pro-7B (decoupled) | 1.00 | 0.44 | 0.68 |
| Emu3-Chat-8B (unified, video-pretrained) | 0.96 | 0.82 | 0.80 |
| Show-o2-1.5B (und never trained interleaved) | 0.68 | 0.76 | — |
| Show-o2-7B | 0.72 | 0.96 | — |
Janus, perfect on one image, collapses when the identical final state sits at the end of an interleaved sequence — it can't bind to the latest frame among near-identical ones. Burning "after step k" labels into the frames recovers half its gap (→0.68): partly a fixable frame-indexing problem. Emu3 (one token stream, video-pretrained) barely drops, and Show-o2-7B even prefers the interleaved format. The MentisOculi interpretation error is a property of architectures/training that never bound text to multi-image context — not a universal UMM deficit.
Recipe (all rows): teacher = the frozen base model on the easy view (final frame + perception question — where it scores 1.0); student = same model trained on a hard view; loss = KL(student‖stop-grad teacher) + CE(true answer); only the LLM trains, the visual readout stays frozen. ~800 steps, batch 4, single GPU. Held-out seeds; the n=4 eval ladder is beyond the trained lengths.
| run | student's hard view | outcome (held-out) |
|---|---|---|
| L_IC v1 (R12) | C1 (redundant text) | closes the distraction gap in <150 steps — trivially learnable; retired with C1 |
| T1 "sim" | S_0 only (initial frame + swaps) | learns to simulate: A(1) 0.60→1.00, A(2) 0.48→0.96, A(3) 0.24→0.52, generalizes +1 past trained length (A(4)→0.64). But destroys handoff: S-ladder →≈chance, flips invert to follow-text |
| T2 "uptake" | S_j, j∈{1…n−1} | mirror image: S_3 0.60→0.92 (flip 0.96/0.04), S_2→0.56; shallow sim improves free (A(1)→1.00); untrained top rung dips |
| T3 "mix" | S_0 + mid rungs | both skills; only the never-trained j=n rung is damaged (0.52) |
| T4 "all rungs" | S_j, j∈{0…n} | capstone: the whole ladder collapses onto an improved A-curve — S = 0.40/0.72/0.88/0.96, flips 0.92/0.04, P=1.00 preserved |
Two lessons. (a) The seesaw: "what does this image depict relative to my computation" is a learned, rung-specific binding — narrow training silently destroys neighboring rungs while P stays at 1.0, so endpoint-only evals would never notice. (b) With rung coverage, privileged distillation teaches both genuine multi-step simulation and causally-grounded state pickup, with no forgetting — far beyond the v1 "ignore the distracting text" result.
One 2.5-hour run (--mode deep: 9000 steps × batch 6; curriculum n_slots∈{3…6} ×
depth∈{1…6} × all rungs), with ladder evals at every snapshot:
Net: distillation makes the model a perfect consumer of visual state at any trained width/depth, and a moderately better mental simulator — but in-context simulation capacity saturates. Depth must come from externalized state (a generated image every step), which keeps the residual load at r=1 where the model is now flawless. That makes fixing generation (§6) the single unlock for the whole pipeline.
After L_SC (flow-matching the gen head to oracle next-state latents) Show-o2 draws clean,
in-domain SlotSwap frames. But the content is random: per-slot accuracy of the generated
state ≈ chance whether teacher-forced (B), free-running (B_free, drift ≈ chance from step 1 —
no compounding needed, step 1 is already at the floor), at 7B, or on a trivial 2-slot task.
Diagnosis. Flow-matching trains on interpolated latents z_t that already point at the target (the z_t target-leak), so the model learns the marginal layout of SlotSwap frames, never the conditional prior+action→state map. Aggravated by entanglement: the generation embedder is shared with the understanding readout, so it must stay frozen or understanding collapses to 0 (R4/R5).
12 failed fixes (one line each): 3× data; 2.5× steps; 7B scale; CFG; trainable embedder; 2-slot toggle; L_SC direct latent match; read-back through the frozen reader (raw and decoded); multi-step differentiable sampling K=4 and K=8; LLM-unfreeze (also forgets); LLM-unfreeze + frozen reader (understanding preserved — but generation still chance). K≥14-step read-back does not fit 96 GB without gradient checkpointing (engineering bound, R23b). Notably: the privileged teacher→student recipe that works on the interpretation side was never applied to generation — still untested.
Needs GPU (queued for when one returns): 1. Test the z_t-leak prediction on a discrete-AR generator — run condition B on Emu3-Gen or Anole generation: if discrete generation IS content-causal, the wall is flow-specific and the fix is an architecture choice, not a new objective. 2. Apply the privileged teacher→student recipe to generation (the one untested, well-motivated fix): teacher conditioned on the oracle next state, student must generate it; KL/feature-match on the gen-head trajectory. 3. Gradient-checkpointed ≥20-step read-back (the R23b engineering unblock), and/or separate gen/und image embedders to resolve the R4 entanglement. 4. Janus interleaved-binding fix: frame-index tokens / light finetune on interleaved traces (labels already recover half the gap for free). 5. Port the handoff ladder to MentisOculi's five tasks — does the SlotSwap picture (handoff OK, sequence-binding arch-dependent, generation broken) transfer to the team's benchmark? 6. BAGEL (strongest open interleaved arch; blocked only on a flash-attn build).
No GPU needed (can do now): report the transformers-5.9 Emu3 image regression upstream; write up the ladder + seesaw as a paper section (the collapse metric and the counterfactual-flip protocol are publishable methodology); design the MentisOculi port.
src/visual_reasoning/ladder.py (+ eval_core.py, tasks/slotswap.py);
runners scripts/eval_ladder_{showo2,janus,hf}.py (--backend emu3|chameleon|anole),
scripts/eval_bfree_showo2.py. Raw results: outputs/ladder/*.json.scripts/train_lic_v2.py (--mode sim|uptake|mix|mixall|deep|c1,
--max_seconds for unsupervised caps) → checkpoints outputs/licv2_*/final; the deep run:
scripts/run_deep_night.sh; generation attempts scripts/train_gen_cycle.py, train_lic.py (v1).report_assets/ladder/ (make_ladder_figures.py); this report:
scripts/build_report_unified.py.docs/results_log.md (R1–R23b), docs/FINDINGS.md, report.md (R1–R18
long form), report_ladder.md (R19–R23 long form), notes/overnight_2026-06-11.md (design)..venv-bagel); Show-o2 runs from its repo
dir in .venv-showo2; SLURM job caps host RSS at 48 GiB → mmap big torch.loads; GPU box
alternates between Blackwell-96GB and A100-40GB — run scripts/check_env.py first.