Set-wise SMC for k-DPP sampling — session findings

Definitions

\(k\)-DPP target

Fix a language model \(p_{\text{LM}}\) and a temperature \(T>0\). For a string \(y\), define quality

For a string \(y\), let \(\phi(y)\in\mathbb{R}^d\) be a unit-norm sentence embedding (\(\|\phi(y)\|=1\)). For a set \(S=\{y_1,\dots,y_K\}\) of \(K\) strings, stack the embeddings as columns of \(\Phi_S\in\mathbb{R}^{d\times K}\) and define the kernel

The \(k\)-DPP is the distribution over size-\(K\) sets

The determinant factors as

The first factor scores quality, the second scores diversity. Exact sampling of \(\pi\) requires summing \(\det(L_{S'})\) over all \(\binom{N}{K}\) subsets of an unbounded string space; the sections below build the set incrementally instead.

Definitions

Sequential Monte Carlo

Let \(q\) be a tractable proposal distribution over sequences (here, the LM itself), sampled token-by-token to build particles. For a particle with target-to-proposal ratio \(w=\pi/q\), define the effective sample size over \(M\) particles

• SIS (sequential importance sampling): extend every particle, accumulate \(w\), never intervene. Unbiased; weights can concentrate on few particles (weight degeneracy).
• Resampling: when \(\mathrm{ESS}<\tau M\) for a threshold \(\tau\in[0,1]\), sample particles with replacement proportional to \(w\), then reset \(w=1\). \(\tau=0\) never resamples (reduces to SIS); larger \(\tau\) resamples more often.
• SMC is SIS with resampling. The estimate \(\hat Z\), a running product of average weights, is the normalizer estimate; \(\mathbb{E}[\hat Z]=Z\) (unbiasedness) is the correctness criterion used throughout this report.

Definitions

Set-wise SMC

Ordinary SMC over individual strings is incompatible with diversity: resampling clones the highest-weight string until all particles coincide. Set-wise SMC lifts the particle to a set of \(K\) strings, so resampling clones or discards whole sets, preserving intra-set diversity by construction.

• Intermediate target \(\gamma_t(S)\defeq\det(L_S)\), evaluated on the length-\(tR\) partial strings at step \(t\); \(\gamma_T=\pi\) up to normalization is the final target.
• Parameters: \(M\) = set-particles, \(K\) = strings per set, \(T\) = temperature, \(R\) = tokens appended per step.
• Two algorithms: naive (ordinary set-level resampling) and mix-and-match (resampling re-forms sets by drawing a fresh DPP over the pooled strings; defined in Defaults).

Metric

Marginal L1 metric

For a string \(y\) in the universe of \(N\) strings, define its marginal inclusion probability under \(\pi\)

Stacking \(\mu(y)\) over all \(N\) strings gives a vector summing to \(K\).

Exact marginal

For a \(k\)-DPP, \(\mu\) has a closed form from the eigendecomposition of \(L\) (Kulesza & Taskar 2012, §5.2), avoiding enumeration of the \(\binom{N}{K}\) sets:

Empirical marginal

Given \(n\) sampler runs with output sets and weights, normalize the weights to sum to \(1\) and, for each string \(y\), sum the normalized weight of every run containing it. This defines \(\hat\mu(y)\); empirical_marginals in the code.

Marginal L1

Both \(\mu\) and \(\hat\mu\) sum to \(K\), so marginal-L1 is \(0\) for a perfect method and bounded by \(2\min(K,N-K)\). It is a per-item calibration error: it does not check pairwise repulsion (a separate pairwise-inclusion L1 metric in the same code covers that), but it is estimable even when the full set-level distribution is too large to enumerate, which is why it is the metric used throughout this report.

Reference

Notation

Workstream A

Default parameter changes

Three parameters, changed to the values the exact-\(k\)-DPP study identified as correct. Prior values remain reachable for ablations; unsafe pool kernels emit a warning.

Parameter

Gram regularizer \(\varepsilon\)

Two members of a set sharing a prefix have identical embeddings so far, so \(\det(G_S)=0\) exactly. The intermediate target then assigns the set zero mass, and resampling removes it — but the two members can diverge later, and the completed set can carry positive \(k\)-DPP mass. The zero is a support violation, not evidence of low diversity.

Fix: while a set is still growing, score it with \(\det(G_S+\varepsilon I)\) instead of \(\det(G_S)\). This is strictly positive for \(\varepsilon>0\), so no set is zeroed. Completed sets use the exact \(\det\), so the final target is unchanged. \(\varepsilon\) trades variance only. Proof in Theory.

Parameter

Pool kernel and proposal double-counting

At a mix-and-match trigger, all \(M\!\cdot\!K\) strings across particles are pooled, and \(M\) new sets are drawn by sampling a DPP over the pool (the pool kernel). Each pooled string is weighted by \(\sqrt{\kappa}\), corrected for its proposal probability \(q\) (the LM probability of the string).

The pool sampler de-duplicates identical strings and passes their counts to the kernel. For an i.i.d. pool, a count ratio \(c/N\) already approximates \(q\), so the count supplies one full factor of \(q\). An earlier implementation additionally multiplied in \(\sqrt{q}\) explicitly (in log-space, \(0.5\log q\)). This double-counts \(q\): the sampler targets \(\kappa\cdot q\cdot\det\) instead of \(\kappa\cdot\det\), with error growing in pool size.

Parameter

Atoms-eps kernel

Because the pool kernel merges duplicate strings, a re-formed set cannot contain two copies of the same current prefix. As with \(\varepsilon\) above, such sets can diverge downstream and carry final mass; excluding them biases the sampler. The atoms-eps kernel:

• keeps every pooled string as its own atom (no merging);
• weights atom \(i\) by \(\sqrt{\kappa/c_i}\), \(c_i\) = number of copies of that string in the pool;
• appends a per-atom \(\sqrt{\varepsilon'}\) coordinate to each embedding, so identical prefixes are no longer exactly collinear.

The resulting kernel determinant equals the \(\varepsilon\)-regularized target exactly (Theory), and duplicate prefixes remain selectable. It is the only pool kernel measured to be unbiased.

Workstream B

Rollout lookahead guides

While a set is still growing, member embeddings are of partial strings, a weak proxy for the final embedding, so the diversity signal driving resampling is noisy. An oracle guide embeds each prefix at \(\mathbb{E}[\phi(\text{final})\mid\text{prefix}]\), the expected final embedding over all completions. This expectation is not computable in general; it is estimated by rollout: sample \(L\) continuations from the LM and average their final embeddings.

The estimate is memoized per string, so it is a fixed function of the prefix; completed strings use the exact final embedding, so the final target is unchanged and only intermediate resampling decisions use the estimate. Implemented at three levels: toy, async llamppl, and a production vLLM RolloutGuide (tested on Qwen3-8B).

Dataset synthetic toy grammar, not real text. make_random_trie_lm: 40 random strings, alphabet {a,b,c}, length 30–50, Dirichlet-random string probabilities (seed 3). Constructed so the exact \(k\)-DPP is computable as ground truth.

The rollout guide (\(L=8\)) tracks the oracle closely, reducing naive-SMC's error 25–35% at both temperatures. SIS (dotted) is the reference floor. 1500 runs/cell; source: experiments/toy_validation/results/convergence/.

Result

Real-LM restricted-trie results

Real MiniLM embeddings over restricted GPT-2 / Qwen3-4B tries; mean weighted marginal-L1 over 4 environments. Real prefix embeddings are noisier proxies of the final embedding than the toy's one-hot prefixes, so guides improve these short horizons by approximately 8–12%; rollout \(\approx\) oracle here as well.

Dataset restricted real-LM trie, not free generation. GPT-2 / Qwen3-4B conditionals restricted to their top-3 tokens per step for \(H=4\) or \(6\) steps after 2 fixed prompts, giving an exactly-enumerable 81- or 729-leaf trie; MiniLM (all-MiniLM-L6-v2) embeddings of the decoded text.

New arms are \(Z\)-unbiased: mean \(|\mathbb{E}[\hat Z]/Z-1|=0.023\) at \(M=64\) (max 0.077).

Workstream C · statement.tex

Expected reachable mass

Define \(\nu_t(S)\), the total final-\(k\)-DPP mass of all completions of a partial set \(S\):

Each finished set descends from a unique partial ancestor, so \(\nu\) satisfies the partition identity \(\sum_{S_t}\nu_t(S_t)=Z\) at every step \(t\). The support-violation bias, the \(\varepsilon\)-fix, and the mix-match residual below all follow from this identity.

Propositions 1–2

Support violation and exact bias

If every intermediate target retains positive mass wherever the final target does (dominates the support), resampling is \(Z\)-unbiased. If dominance fails at one step, exactly the reachable mass of the zeroed sets is lost:

Two members sharing a prefix give \(\det(G_S)=0\) exactly, while \(\nu_t(S)>0\) in general: the naive intermediate target violates dominance and loses that mass.

Proposition

Mix-and-match reset bias

After a pool draw, mix-and-match resets weights to uniform, as ordinary resampling would. The draw is instead from the pool-restricted \(\gamma^\varepsilon\), not the weighted particle cloud. With the lookahead ratio \(\rho_t(S)=\nu_t(S)/\gamma^\varepsilon_t(S)\):

The gap is the difference between \(\rho\) averaged over recombined pool sets and \(\rho\) averaged over the weighted particle distribution. Two corollaries: a perfect guide (\(\gamma^\varepsilon\propto\nu\)) makes \(\rho\) constant, giving unbiasedness for any pool; merging duplicates additionally removes the duplicate-prefix mass.

The proposition specifies the correction — reset weights \(\propto\hat\rho_t\), not uniform (next tab).

Finding 1 · exact correction

Reachable-mass reweighting

The uniform reset is incorrect by the factor \(\rho_t=\nu_t/\gamma^\varepsilon_t\). Maintain one additional per-particle ledger and multiply each \(\hat Z\) event factor by \(\rho_t\); the sampler is otherwise unchanged. This makes \(\hat Z\) use the ideal intermediate target \(\nu_t\) at every event, and \(\mathbb{E}[\hat Z]=Z\) exactly when \(\rho\) is exact.

On enumerable problems, \(\nu\) has closed form for \(K\le3\) (reduces to traces of one global Gram matrix). Verified to machine precision: \(\nu(\text{roots})=K!\,Z\), \(\nu(\text{complete})=\det L_S\), and the tower property.

Dataset same synthetic toy grammar as the lookahead figure (40 random strings, alphabet {a,b,c}, length 30–50), not real text.

Top row: exact \(\nu\) (blue) reduces the bias to approximately \(0\) at every \(M\) and both temperatures; the clipped rollout estimate (dashed) is noisier but centered. Bottom row: marginal error is nearly identical across all three — the correction fixes \(\hat Z\), not the support restriction. 1500 runs/cell; source: experiments/toy_validation/results/convergence/.

Finding 2a · stability

Rollout estimator clipping

The correction divides by \(\hat\nu\). Where the true \(\nu\) is near zero (recombined near-duplicate sets), a sparse rollout underestimate makes \(1/\hat\nu\) diverge — unclipped, \(\hat Z\) reached \(\sim10^{14}\) in the easy regime. nu_rho_clip floors each reset offset at \((\text{batch max}-C)\) nats, bounding the overweighting at \(e^C\).

\(C=4.6\) nats is a \(100\times\) cap; the clip's bias affects only sets whose reachable mass is \(\sim e^C\) below the batch maximum.

Finding 2b · readout

Cross-run \(\hat Z\)-weighted readout

Each run gives an unnormalized-measure estimate \(\hat\mu(f)=e^{\log\hat Z}\sum_i\tilde w_i f(S_i)\). Pooling runs weighted by their own \(\hat Z\), rather than equally, is globally self-normalized: it replaces each run's self-normalization bias (\(\sim1/M\)) with a cross-run bias (\(\sim1/(nM)\)). Every cell below pools \(n=1500\) independent SMC runs (the toy study's default run count per regime/\(T\)/method/\(M\) cell); \(n\) is fixed across the \(M\) sweep.

Dataset synthetic toy grammar, three regimes (easy: 24 strings length 4–7; hard: 30 strings length 10–16; long: 40 strings length 30–50; alphabet {a,b,c}), not real text.

At every \(M\) in the grid, the cross-run readout is at or below the per-run readout, with the largest gap at small \(M\) (\(n=1500\) runs/cell in both curves). The two converge by \(M=64\).

Workstream D

Qwen3-8B validation

The frontier sweep selects its model by environment variable, defaulting to full-attention Qwen3-8B (its attention pattern keeps vLLM's prefix cache effective; gemma-2's sliding window disables it, and its V0 path is unstable on this GPU). The full validation grid (268 cells) reproduces the GPT-2 and Qwen3-4B results.

Dataset restricted real-LM trie, Qwen3-8B, identical construction to the GPT-2/Qwen3-4B table above (top-3-token restriction, \(H=4/6\), 2 prompts, MiniLM embeddings), not free generation.

• naive-SIS: \(0.246\to0.058\) as \(M\) grows, \(|\mathbb{E}[\hat Z]/Z-1|\le0.021\) at \(M=64\).
• atoms-eps is the only converging mix-match; lookahead reduces naive-SMC error \(\sim\)10–15%.
• Resampling outperforms SIS only at T2/M4, the small-\(M\), high-\(T\) case, matching the toy result.

Result

Quality-diversity frontier

Each method traced as \(M\) grows from 4 to 100 (\(K=10\), sampled sets, Qwen3-8B). The "mix-and-match" series in this section is the atoms-eps kernel, the only variant validated as distributionally consistent (Defaults); "naive" is naive set-SMC at the default trigger \(\tau=0.25\). Naive is approximately stationary in \(M\); mix-match moves toward quality as \(M\) grows, at the cost of diversity (the pool-restriction bias at scale). Error bars are the standard error across runs; mix-match's \(M=100\) cell has 20 runs (the GPU session was reclaimed mid-run), hence its wider bar.

Dataset Qwen3-8B generation (vLLM), 5 prompts from FineWeb, \(K=10\), up to 50 tokens/continuation, \(T=1\), Qwen3-Embedding-0.6B embeddings. Free generation, not a restricted trie.

Naive holds mean log-prob \(\approx-88\) and diversity \(\approx0.30\) from \(M=4\) to \(100\); mix-match's log-prob rises from \(-90\) to \(-33\) while diversity falls from \(0.30\) to \(0.20\). Source: experiments/results_v2_qwen3-8b/K10_M*.pkl.

Dataset same Qwen3-8B/FineWeb run as above, plus an ancestral-sampling baseline: \(K=10\) i.i.d. LM continuations with no DPP/SMC (5 prompts × 10 runs = 50 sets/temperature; experiments/frontier_ancestral_baseline.py, RTX 4090). Naive and mix-and-match carry a second, dashed line at \(k\)-DPP temperature \(T=2\) (experiments/frontier_temperature_sweep.py); ancestral's two \(T\) values form a dash-dot line, since it has no \(M\) to sweep. Definitions of \(T\) differ between the two families — see the note below.

Each connected line is an \(M\)-sweep from \(M=4\) to \(M=100\) at a fixed \(T\) (arrow indicates direction; dashed = \(T=2\)). Naive is approximately stationary; mix-match trades diversity for quality; PoolDPP loses quality as \(M\) grows without a diversity gain.

Dataset identical runs to the figure above (including the \(T=2\) lines); only the diversity statistic changes (\(\det(\Phi_S^\top\Phi_S)\) of the set's unit embeddings, log-scaled \(x\)-axis).

\(\det(G_S)\) is more sensitive to near-collinear sets than average pairwise cosine similarity: the mix-match collapse that appears as a 33% drop in \(1-\cos\) (0.30→0.20) is a \(>100\times\) collapse in \(\det(G_S)\) (\(\sim4\times10^{-4}\to\sim5\times10^{-6}\)) from \(M=4\) to \(M=100\). This is the statistic the \(k\)-DPP target is built from (Background).

Cross-repo comparison

Intrinsic frontier: PoolDPP reference and setty-smc

This is the predecessor project's reference plot (pool-dpp-release/analysis/plot_intrinsic.py): quality (mean log-probability) vs. diversity (log-determinant of the Gram matrix — not \(1-\cos\)-sim or raw \(\det\) as used elsewhere in this report), each curve sweeping a temperature parameter. Reproduced with identical metric definitions, colors, markers, and annotation style; setty-smc's methods are added as a second family of curves.

Dataset reference curves: PoolDPP's FineWeb cache (Qwen3-8B, Qwen3-Embedding-0.6B, 50 prompts, pool-dpp-release/cache-fineweb/est10qwen/). setty-smc curves: same 5-prompt Qwen3-8B/FineWeb runs as the rest of this tab, \(K=10\), \(T=1\), \(M\) swept 4–100, re-embedded with Qwen3-Embedding-0.6B and scored with PoolDPP's metric (experiments/intrinsic_comparison.py).

Temperature-swept comparison: same knob as PoolDPP-Samp

The \(M\)-sweep above holds \(T=1\), where the exact \(k\)-DPP target barely moves quality/diversity at \(K=10\)/50 tokens — our faithful samplers correctly sit on the ancestral point. PoolDPP-Samp's frontier movement instead comes from sweeping its target temperature \(tT\) down (\(\kappa = p^{1/tT}\)), which is exactly our \(T\) (identical \(L\)-ensemble). This figure gives our methods the same knob: \(T \in \{0.3, 0.5, 0.7, 1, 2\}\) at fixed \(M=100\), crossing a second axis — where resampling happens: set-level at the end (naive-SIS), set-level intermediate (naive, \(\tau=0.25\)), pool-level intermediate (mix-and-match atoms-eps), pool-level at the end (PoolDPP IS-kernel — algorithmically PoolDPP-Samp itself run on our prompts/pool, the anchor separating method effects from prompt-set effects — and pool-atoms-eps, the same end-only draw with our support-consistent kernel).

Dataset \(T \in \{0.3, 0.5, 0.7\}\) for all five methods, the paper's full \(tT\) grid \(\{0.3, 0.35, 0.4, 0.45, 0.5, 0.55\}\) for PoolDPP IS-kernel (replication campaign), and the pool-atoms-eps \(T=1\) anchor: experiments/frontier_lowT_sweep.py → K10_lowT_sweep.pkl, \(M=100\), \(K=10\), 10 runs × 5 prompts per cell, all cells complete; the 2026-07-22 extension adds naive at \(T \in \{0.2, 0.4, 0.85\}\), the mix-and-match trigger ablation \(\tau \in \{0.05, 0.10\}\) at \(T=0.3\), naive-SIS at \(M \in \{200, 400\}\) (\(T=0.5\), drawn as a dotted branch off its \(T=0.5\) point), the PoolDPP-MAP \(\beta\) grid on our prompts (K10_map_baseline.pkl, \(\beta = 1/(2tT)\), \(tT \in \{500, 100, 50, 25, 10\}\), deterministic greedy selection), and token-level greedy set decoding (K10_greedy_set.pkl, \(\beta \in \{0.005, 0.02, 0.05, 0.1, 0.2\}\), 5 runs per cell); joined with the \(T=1\) (K10_M100.pkl, K10_extra_variants.pkl) and \(T=2\) (K10_temperature_sweep.pkl) cells. Cells are excluded from the figure unless their runs cover all 5 prompts (prompt-composition bias guard).

The recommended sampler

Naive set-SMC: what it is, when it resamples, why it works

One sampler emerged from the validation and frontier work as the recommendation: naive set-wise SMC (Algorithm 4; set_smc/llamppl.py::smc_set_naive). It samples the global \(k\)-DPP \(\pi(S) \propto \det(L_S)\) with \(L = \mathrm{diag}(\sqrt\kappa)\,\Phi\Phi^\top \mathrm{diag}(\sqrt\kappa)\), quality \(\kappa(y) = p_{\text{LM}}(y)^{1/T}\), by evolving whole candidate sets as SMC particles. On the intrinsic frontier it traces the same curve as PoolDPP-Samp while remaining a sequential sampler with a normalizing-constant estimate, and among everything tested here it is beaten only by the two argmax methods (\(k\)-Means, PoolDPP-MAP).

The algorithm, and when it resamples

Run \(M\) sets of \(K\) particles each (\(M{\cdot}K\) LM streams total). Every particle extends one token per step from the untempered LM (proposal \(q = p_{\text{LM}}\); the temperature knob never touches generation). After each step, every set \(S\) gets an absolute weight recomputed from its full current strings:

\[ w(S) \;=\; \underbrace{\textstyle\sum_{y \in S} \tfrac{1}{T}\log p(y)}_{\text{quality}} \;+\; \underbrace{\log\det\!\big(G_S + \varepsilon I\big)}_{\text{repulsion}} \;-\; \underbrace{\textstyle\sum_{y \in S} \log q(y)}_{\text{proposal}}, \]

where \(G_S\) is the Gram matrix of the members' unit embeddings and \(\varepsilon\) applies only while a set is still growing (completed sets use the exact determinant; the ratios telescope, so the final target is unchanged). Resampling triggers when the effective sample size of the weights relative to the last resampling event drops below \(\tau M\): whole sets are multinomially cloned or killed, members are never mixed across sets, and each survivor's baseline resets to its current absolute weight. With \(\tau = 0\) nothing ever triggers and the method degenerates to naive set-SIS: \(M\) independent ancestral sets, importance-reweighted once at the end. The final readout samples one set proportionally to the final relative weights (at \(T < 1\) the weights concentrate so hard that this equals the argmax; measured: identical to 2 decimals at \(T \le 0.7\)).

Why it is winning now

Three ingredients, in the order they were fixed:

1. Correct resampling bookkeeping (correctness rule 1). Resampling on recomputed absolute weights compounds selection across events: after \(k\) events the sampler targets \((\det(L_S)/q)^k\) instead of \(\det(L_S)/q\). The per-set baseline ledger fixed this in April; every later result depends on it.

2. Support-dominating intermediate targets (rule 2). Shared prefixes make \(\det(G_S)\) exactly zero while descendants still carry final mass; a zero-weight set can never be resampled back. Scoring growing sets with \(\det(G_S + \varepsilon I)\), \(\varepsilon = 10^{-2}\), keeps every lineage selectable without changing the final target. Without it, intermediate resampling silently prunes support.

3. A target worth chasing. At \(T=1\) the exact \(k\)-DPP over \(K{=}10\) fifty-token continuations sits at the ancestral point: 1024-dimensional embeddings of independent continuations are already nearly orthogonal, so the determinant barely moves anything (naive's \(M\)-sweep stays at \((-4, -88)\) for every \(M\); that is the correct answer, verified against exact references). The frontier movement of PoolDPP-Samp comes from sweeping its target temperature down, and \(T\) is the same knob: at \(T = 0.3/0.5/0.7\) naive set-SMC lands at \((-7.6, -37.8)\), \((-6.9, -39.6)\), \((-6.3, -44.1)\), on the reference curve (their \(tT{=}0.45\) point: \((-7.9, -39.2)\)) and with no analog of the IS-kernel's \(tT \ge 0.5\) cliff. Intermediate resampling is what buys the quality: the end-only variant (set-SIS) can only reorder \(M\) pre-formed i.i.d. sets and saturates at \((-5.5, -67)\), while the \(\tau{=}0.25\) trigger concentrates the particle budget on high-weight set lineages step by step, 30 nats better at the same \(M\).

Current parameters

What failed along the way

• Absolute-weight resampling (rule 1) and zero-determinant intermediate targets (rule 2): both produced measured bias against exact \(k\)-DPP references before the fixes.
• Merged mix-and-match pool kernels (rules 3–4): duplicate merging prunes support; \(\mathbb{E}[\hat Z]/Z - 1 \to -0.7\), error growing with \(M\). Replaced by the atoms-eps kernel.
• Prompt-prefix contamination of Gram diversity metrics (rule 8): deflated every cross-repo comparison until continuations were re-embedded alone.
• The pool sampler's fixed per-event cost at \(M{=}100\), serialized through the event loop, read as a hang (rule 9). Fixed by the fast dual sampler plus process isolation; zero timeouts in roughly 2,500 runs since.
• Sweeping \(T\) upward only. \(T{=}2\) moves quality from \(-88\) to \(-240\) for 1 log-det unit of diversity; every interesting point sits at \(T < 1\).

Limitations

• Distributional guarantees are certified near \(T{=}1\), used at \(T<1\). \(\hat Z\) unbiasedness and marginal convergence were validated against exact references on toys and restricted-support tries. At \(T{=}0.3\) the importance weights degenerate (ESS \(\approx 1\)), so the readout is effectively the best surviving lineage, a sampler-shaped optimizer rather than a certified \(k\)-DPP draw. The frontier claims are about the quality/diversity of the output sets, not about distributional fidelity at low \(T\).
• Lineage coalescence at low \(T\): cloning shares prefixes, so diversity drifts down as \(T\) drops (\(-4.1\) at \(T{=}1\) to \(-7.6\) at \(T{=}0.3\)). Mild here; expect worse at longer horizons.
• The sampler's own embeddings include the prompt (rule 8): repulsion among same-prompt candidates is blunted by the shared component. Fixable in _embed_particles; untested.
• Scale of evidence: 5 FineWeb prompts (3 shared with the reference's 50), 50-token continuations, \(K{=}10\), one LM family at 8B. Cost: \(M{\cdot}K\) concurrent LM streams per run, roughly \(M{\cdot}K/K = M\times\) an ancestral baseline.
• Below the argmax ceiling: \(k\)-Means and PoolDPP-MAP dominate the top-right of the frontier. A sampler pays a stochasticity tax; see below.

Mix-and-match: why its points sit far up-left

Mix-and-match re-forms sets from the pooled atoms at every ESS trigger. The kernel is not the problem: the identical atoms-eps kernel applied once, to an i.i.d. pool at the end, yields \((-10.9, -30)\) at every \(T\) (the pool-atoms-eps rows). The problem is compounding: at \(T<1\) the trigger fires near-continuously, each event re-selects atoms proportional to \(\sqrt{\kappa/c}\) from a pool that the previous event already concentrated, and cloned high-\(\kappa\) atoms accumulate multiplicity until sets are near-duplicates: diversity \(-47\) to \(-57\) at quality \(-16\) to \(-20\). So: worse as configured, and the configuration, not the kernel, is the culprit — the 2026-07-22 ablation sharpened where exactly. Lowering \(\tau\) to 0.10/0.05 recovers diversity only mildly (\(-56.7 \to -43.2\)) because the ESS trigger saturates at \(T=0.3\): even \(\tau=0.05\) fires on 33 of 50 steps (\(\tau=0.25\): 49.7). No threshold reaches the few-events regime, so the ablation's knob is weaker than designed. The event-count diagnostic pins the mechanism to the event type: naive fires just as often (49.7/50) with no collapse, so a pool-level re-selection event is damaging per se and the damage compounds with event count (50 events \(\to -57\), 33 \(\to -43\), 1 \(\to -10.9\)). Mix-and-match at low \(T\) should run with its single end-of-run event — which is already the pool-atoms-eps method.

Toward a token-level argmax method

The two dominant reference curves are argmax methods over a completed pool. Two facts from this study point at what an autoregressive analog needs. First, argmax at the readout level is already saturated: the best-of-\(M\) readout equals the sampled readout at \(T \le 0.7\), so nothing is gained by argmaxing over SMC runs. Second, end-only selection at the set level is capped (set-SIS), while atom-level selection is not (PoolDPP): the argmax has to happen inside the decoding loop, at the token level, over set-aware scores.

Greedy set decoding: implemented (v1), and it reaches the MAP frontier at its high-diversity end. The design is now experiments/greedy_set_decoding.py: one set of \(K{=}10\) partial strings; at each step every slot scores its top-\(n{=}5\) candidate tokens by \(2\beta \log p(t \mid \text{prefix}) + \log(\text{Schur gain} + \varepsilon)\) — the repulsion term is the conditional log-determinant gain (the greedy_map arithmetic) of the candidate's rollout-lookahead embedding (lookahead.RolloutGuide, \(L{=}4\), memoized — the cache is also what keeps conditioning consistent within a run) against the other nine slots' current embeddings; commits are round-robin with rotating order, finished slots switch to their exact final embedding, and \(\beta = 1/(2tT)\) matches the MAP baseline's kernel exactly. Measured (intrinsic scale, 5 runs/cell, \(tT = 1/(2\beta)\)): \(\beta=0.2 \to (-4.87, -36.0)\), \(\beta=0.1 \to (-3.59, -38.0)\), \(\beta=0.05 \to (-2.63, -45.9)\), \(\beta=0.02 \to (-2.05, -62.1)\), \(\beta=0.005 \to (-1.74, -109.7)\). The \(\beta=0.1\) and \(\beta=0.05\) points lie on the interpolated PoolDPP-MAP curve (within 0.15 and 0.3 nats respectively), and they extend it to diversity levels no reference curve reaches (log-det \(-1.7\) to \(-2.6\); the reference tops out near \(-2.3\)). What v1 does not reach is the MAP curve's high-quality corner: quality saturates near \(-36\) as \(\beta\) grows (\(\beta: 0.1 \to 0.2\) buys 2 nats and costs 1.3 log-det), and MAP's \(tT{=}10\) point \((-6.70, -28.3)\) dominates greedy's \(\beta=0.2\) point on both axes — per-token greedy commits with \(L{=}4\) lookahead pay a quality floor that per-string selection over a completed pool does not. Cost: \(\approx\)478k rollout tokens per run at \(K{=}10, n{=}5, L{=}4\), about \(10\times\) the generation tokens of one \(M{=}100\) pool run. Open for v2: a small per-step beam over joint assignments against slot-interaction myopia, prompt-stripped guide embeddings (rule 8 applied to decoding-time repulsion), and larger \(n\)/\(L\).

Provenance all numbers on this tab: run-level means from the temperature-swept intrinsic study (Models & frontier tab), Qwen3-8B, 5 FineWeb prompts; sampler cells \(M{=}100\), 10 runs; PoolDPP-MAP and greedy set decoding from the 2026-07-22 extension (K10_map_baseline.pkl, 10 runs; K10_greedy_set.pkl, 5 runs).