The Tokenizer Audit

Definitions

1.0 Objects

A document \psi is a string of UTF-8 bytes; |\psi| is its length in bytes. A tokenizer \tau maps \psi to a string of tokens \tau(\psi) = t_1 \cdots t_T over a vocabulary \Delta of size V. Every tokenizer used here is invertible on the corpus, \tau^{-1}(\tau(\psi)) = \psi, verified byte-exactly on every document. A model q is a probability distribution over token strings.

Byte-level BPE, the algorithm under study, fits \tau as an ordered merge list. Start with the 256 byte types (a type is an entry of the vocabulary; a token is one occurrence of a type in a sequence). Repeatedly count adjacent type pairs in the fitting corpus, merge the most frequent pair into a new type, and append it to the list. After k merges the vocabulary has V = 256 + k types. Encoding replays the merges in learned order, so the merge list is both the model and the program.

Pretokenization is a regular expression applied before fitting and before encoding. Merges may not cross the fragment boundaries it creates. Three pretokenization variants are explored below: raw (none, so merges may swallow whitespace and cross word boundaries), GPT-2 (the GPT-2 regex, which cuts at whitespace and at letter / number / punctuation class changes), and attach (leading whitespace attached to the following word, no other cuts).

1.1 Coding metrics

Probabilities and code lengths are the same object. For any distribution P over a countable set there is a prefix code (a binary code where no codeword is a prefix of another) with lengths \lceil -\log_2 P(x)\rceil, and every prefix code defines such a P. Assigning probabilities and assigning code lengths are the same thing. Every quantity on this page is therefore a code length for the same text, divided by the same number of bytes:

Expectations are over documents. In practice each one is a sum over the documents of one evaluation partition, so a metric is total bits divided by total bytes on that partition. Only the numerator changes. The denominator is a property of the text, not of the tokenizer. That is what makes the comparison legitimate. Two tokenizers with different vocabularies cut the same text into different numbers of tokens, so bits per token compares nothing. Two reference points bound the metric. Transmitting raw UTF-8 costs 8 bits per byte, so any number above 8 is worse than sending the bytes unchanged. Byte perplexity is 2^{\text{BPB}}: 1.5 bits per byte is a perplexity of 2.83 per byte.

1.2 Tokens per byte

The standard intrinsic metric counts symbols rather than bits:

Its reciprocal is mean bytes per token. It uses no probability model, so it is not a code length. It measures the length of tokenized \psi (compression) without taking the cost into account.

Along a BPE merge path it is monotone by construction. Let \tau_k apply the first k merges. Adding merge k+1 rewrites some number a_{k+1}(\psi) \ge 0 of adjacent pairs into single tokens and changes nothing else, so

TPB has no interior minimum in k. Merging New with ␣York, then that result with ␣City, improves it every time. Carried to the end, one token per document gives \text{TPB} = 1/\mathbb{E}|\psi|, which TPB scores as near-perfect. TPB does not charge for over-merging. Every metric defined below does.

Summary. TPB fixes how long the representation is and says nothing about what it costs. It has no optimum to find, only a direction. Each metric below adds one of the costs TPB omits.

1.3 Unigram bits per byte

Give the tokens a distribution, but no order. The population unigram marginal is the expected count of a type divided by the expected sequence length:

Under a code built from u, a document costs \sum_j -\log_2 u(t_j) bits, whose expectation is \mathbb{E}|\tau(\psi)| \cdot H(u). Dividing by expected bytes gives an exact factorization, not an approximation, because an order-zero cost is additive over positions:

This is the first metric that can get worse. A merge pushes the two factors in opposite directions: the sequence gets shorter (decreasing TPB), and the type distribution spreads over a larger vocabulary with mass moved onto rarer types (increasing entropy). The product is not monotone in k, so an interior optimum exists and can be located (branch 3 measures it: 4K, 16K, 32K and 131K types for fitting corpora of 100KB to 6MB). In those measurements u is estimated from token counts on the fitting corpus, with Dirichlet smoothing, and the code is scored on the tokenizer-validation partition.

Two costs remain uncharged. Both belong to the vocabulary rather than to the text.

Estimating u. H(u) treats the unigram distribution as known. It is not known. It is computed from a finite corpus, and a receiver who does not already hold those frequencies cannot decode a message coded with them. \text{TPB}\cdot H(u) has no term for that.

Communicating \Delta. The vocabulary is itself learned from the fitting corpus. A decoder needs it, and \text{BPB}_{\text{uni}} charges nothing for sending it. A larger vocabulary is never penalized for being larger, only for how it spreads probability mass across types.

Summary. \text{BPB}_{\text{uni}} is the first metric here that can increase when merging goes too far, so unlike TPB it has an optimum to find. That penalty arrives only through H(u). Merging moves probability mass onto more and rarer types, which raises the bits per token, so the product can rise even while TPB falls. It is not a charge for vocabulary size. Two vocabularies of very different sizes cost the same here if they give the same TPB and the same entropy. It is also still order zero, so it scores each token without reference to its neighbors. Charging for the vocabulary itself is what §1.4 adds. Dropping the order-zero assumption is what §1.5 adds.

1.4 Two-part MDL

The minimum description length principle: among hypotheses h, prefer the one that minimizes the total bits a receiver who knows nothing needs in order to reconstruct the data.

Every L in this section is a count of bits. Here D is the fitting corpus and h_k is the first k merges. Raising k lowers L(D \mid h_k), because the token sequence gets shorter, and raises L(h_k), because there are more merges to transmit. If the two cross, the minimum is a stopping rule that consumes no held-out data at all.

1.4.1 Computing the corpus part

The problem. Transmit the tokenized fitting corpus to a receiver who knows neither the token frequencies nor the vocabulary, and count every bit it takes. MDL ranks vocabularies by that count, so the count has to include everything the receiver needs.

Fix a merge count k. This part computes L(D \mid h_k) alone. Encode the fitting corpus with the first k merges, write n_\delta for the number of times type \delta occurs in the resulting token sequence, and write N = \sum_\delta n_\delta for the length of that sequence. The cost of the merge list itself is the separate term L(h_k), computed in 1.4.2.

First attempt: entropy

The obvious code uses the corpus's own empirical frequencies \hat{p}_\delta = n_\delta / N. Its length is

It is not a description length. A description length has to be self-contained: a receiver holding the message, and nothing else agreed in advance, must reconstruct the corpus. Decoding with \hat{p} requires the count table. An honest sender transmits that table and then the token sequence, so N H(\hat{p}) is the length of the second half only, quoted as if it were the whole message.

Why that matters. The missing bits number about \tfrac{V-1}{2}\log_2 N, so the omission grows with the vocabulary. A larger vocabulary hides more of its own cost. Substituting the plug-in entropy for a real code length would bias \arg\min_k toward more merges, which is the failure the criterion exists to prevent.

How large the omission is. Take the 6MB fit at the vocabulary the criterion selects, V = 44{,}361. The tokenized corpus is N = 894{,}031 tokens. Computed properly, the corpus term is 1.26\times 10^{7} bits and the tokenizer term L(h_k) is 1.24\times 10^{6} bits. The omitted \tfrac{V-1}{2}\log_2 N is 4.38\times 10^{5} bits, or 54.8 kB, 53.5 KiB, 0.055 MB, 0.052 MiB.

What to compare it against. Per byte of the fitting corpus the omission is 0.073 bits, which looks small, and that is the wrong denominator. Against the corpus term it belongs to it is 3.5%. Against the tokenizer term it is 35%, and the tokenizer term is the only part of \text{MDL}(k) that penalizes further merging. Against the curve: on the 6MB fit in the branch 3 figure, MDL moves by 0.009 bits per byte across the whole range V = 41{,}285 to 52{,}016, the range that brackets its minimum. The omission is eight times that. Leaving it out would move the selected vocabulary.

Second attempt: the KT mixture

The fix is a single distribution over token sequences, fixed before any data is seen, that does not require knowing the frequencies. Put a prior \pi on the unknown token distribution p and integrate:

The integral runs over \mathcal{P}(\Delta), every probability distribution on the vocabulary. With \pi = \mathrm{Dirichlet}(\tfrac12,\ldots,\tfrac12), the Jeffreys prior for a multinomial, it is closed form. This is the Krichevsky-Trofimov code:

Sender and receiver agree on Q in advance, so L_{\text{KT}} is a complete description length with no side information. Three properties make it suitable here.

• It is a predictor. Q factorizes into the sequential rule q(t_{n+1} = \delta \mid t_{1:n}) = (n_\delta + \tfrac12)/(n + V/2), add-one-half smoothing, so L_{\text{KT}} is the accumulated log loss of an online predictor that has never seen the counts in advance.
• It pays the estimation cost and no more. For every sequence, L_{\text{KT}} \le N H(\hat{p}) + \tfrac{V-1}{2}\log_2 N + O(1): exactly the term the plug-in figure skipped, which is the minimax redundancy rate (the smallest excess over the best fixed code, in the worst case over sources) for a multinomial source.
• It is cheap along the merge path. L_{\text{KT}} depends on the data only through the counts, and one merge changes three counts (the two merged types down, the new type up) plus N. Replaying the merge list updates it in O(1) per merge, so the curve exists at every k with no grid and no refitting. The replay is checked against direct re-encoding of the corpus; token counts match exactly.

1.4.2 Computing the tokenizer part

Merge i names an ordered pair of types that already exist, drawn from the 256 + i types defined at the moment it is created. A fixed-width index costs \log_2(256+i) bits per element, two elements per merge:

Without this term, a tokenizer could memorize whole documents as single types and score arbitrarily well on the data cost. This term is what charges for over-merging.

1.4.3 The criterion

\arg\min_k \text{MDL}(k) selects a vocabulary from the fitting corpus alone. Against \text{BPB}_{\text{uni}} it makes the same order-zero assumption about token order and differs in two ways: it charges for learning u and for the merge list, and it is computed on the data the tokenizer was fitted to rather than on a held-out sample. No language model is involved. It is a tokenizer-side criterion, and its numbers are not comparable to the LM numbers in the tables below.

Summary. MDL charges for the token sequence, for learning the frequencies, and for the merge list. It is the first metric here that charges for the size of the vocabulary itself, which is what stops a longer merge list from always looking better. It is therefore the first criterion here that can pick a vocabulary on its own, with no held-out text and no LM. It is still order zero, and it is measured on the corpus the tokenizer was fitted to. Branch 3 reports what it picks.

1.5 Held-out LM bits per byte

Replace the order-zero model with a trained one. A language model q is fitted to the tokenization of an LM-training partition and scored on a disjoint validation partition of held-out documents:

Because \tau is invertible and q is a genuine distribution over token strings, the pair is an actual code for the original bytes, and LM-BPB is its rate. A tokenizer that shortens the sequence but makes each token harder to predict pays for it here. One that lengthens the sequence and makes it easier to predict is rewarded. Branch 5 and the 544M scale check both contain conditions where TPB and LM-BPB rank two tokenizers in opposite orders.

Two implementation details, identical across every condition, keep the measured quantity slightly below the ideal code length. Documents are scored in context windows of c tokens with the context reset at each window, so the first token of a window is conditioned on nothing and is not scored (about 1 token in 129 at c = 128), and no end-of-document symbol is scored. Both discounts are small and apply to all conditions. They are not exactly equal across conditions, because conditions differ in TPB.

Summary. LM-BPB drops the order-zero assumption. It is the only metric here that charges for everything a model has to learn, on text neither the tokenizer nor the model has seen. It costs one LM training run per condition, which is why the cheaper metrics above are still worth computing.

1.6 Metric comparison

Read down the table. TPB asks how short the representation is. \text{BPB}_{\text{uni}} asks what that representation costs when order is ignored. MDL asks what it costs when neither the frequencies nor the tokenizer are known in advance. LM-BPB asks what remains unpredictable after a model has used all the context it can. The paper's argument is that the first metric is the one the field reports and the last is the one that matters.

1.7 Metric disagreement, measured

The three 6MB English conditions put numbers on the previous table. TPB and \text{BPB}_{\text{uni}} are measured on the tokenizer-validation partition (\text{BPB}_{\text{uni}} with counts from the fitting corpus, Dirichlet-smoothed, as in §1.3). LM-BPB is the paper-protocol mean over three seeds. MDL has no column: it is computed along one merge path to choose k, and these three tokenizers are three different paths, so it does not compare them.

The three metrics pick three different winners. TPB inverts the LM ordering: raw produces the fewest tokens and the worst LM. \text{BPB}_{\text{uni}} repairs part of that (the pretokenized variants stop being punished for emitting more tokens) but separates raw from GPT-2 by 0.0001 bits per byte, where the LM separates them by 0.117. The entire pretokenization gain is invisible at order zero, so it lives in token-to-token dependence, which is also why no order-zero criterion, MDL included, can arbitrate between pretokenization schemes. Source: pretok-6mb-*-intrinsic.json, campaign-all-runs.csv.

Comparison methods

Part 1 defines what is measured. This part defines the four derived quantities used to decide whether a difference between two measurements means anything.

2.1 Paired effect sizes

Every LM run is a pair (tokenizer condition, seed). Differences are formed within a seed and then averaged, never across seeds:

Both at fixed V = 16{,}384. \Delta_{\text{data}} measures what more tokenizer-fitting text gives. \Delta_{\text{pretok}} measures what the GPT-2 regex gives at equal fitting text. Positive means the second condition is better. The reported \pm is the standard deviation of the three paired differences, not of the three levels.

2.2 Excess risk over an oracle

For a fitting method M and a fitting budget of n bytes, with the same algorithm, the same vocabulary size and the same LM protocol throughout, and only n changing:

The 50MB fit is that method's oracle: a stand-in for the tokenizer the method would produce with unlimited text. \Delta_M(n) is therefore the method's estimation error at budget n, measured against itself rather than against another method. Two readings follow, and they answer different questions:

• \Delta_M(n) \to 0 means the method's fitting problem is solved at n bytes. More tokenizer data cannot help it further.
• \text{BPB}(M, 50\text{MB}) < \text{BPB}(M', 50\text{MB}) while \Delta_M \approx \Delta_{M'} at every n means M is a better hypothesis class, not a better estimator. It is better in the limit, and no better when fitting text is scarce.

The paper's open question, whether pretokenization acts as a statistical regularizer, is exactly the difference between these two readings. Branch 4 measures it.

2.3 Error bars

A comparison at 100KB has two random inputs, not one: which 100KB of text the tokenizer happened to see, and which seed the LM happened to get. Writing \text{BPB}_{ij} for LM seed j on tokenizer sample i, a one-way random-effects model separates them:

\text{MS}_{\text{between}} is r times the variance of the per-sample means. \text{MS}_{\text{within}} is the variance across seeds, pooled within samples. \sigma_{\text{within}} is the seed-only bar the paper originally reported. \sigma_{\text{between}} is the bar a reader needs when the claim is about a tokenizer fitted on n bytes, because a replication would draw a different n bytes.

2.4 The Occam bound

The paper's Theorem 4.1 bounds held-out risk by empirical risk plus a penalty in the description length of the hypothesis. For h described in L(h) bits, m independent blocks, per-block loss bounded by \ell_{\max}, with probability at least 1-\delta:

Instantiated here with L(h) an explicit integer-length Shannon code over the full vocabulary plus a fixed-width code-length table, m the number of fitting documents, \ell_{\max} the realized maximum per-document bits per byte, and \delta = 0.05. Vacuous has a precise meaning here: the right-hand side exceeds 8 bits per byte, the cost of transmitting the raw UTF-8, so the bound excludes nothing.

Data and protocols

All splits are whole-document, deduplicated, and content-hashed in tracked manifests. Each partition has a single role: tok-fit fits the tokenizer, tok-val selects tokenizer-side hyperparameters, LM-train trains the language model, LM-val is where every LM-BPB number on this page is measured, and test has never been read.

Where a table below says only "BPB", it means LM-BPB (§1.5) on that manifest's LM-val partition, at the best-validation checkpoint of the stated protocol. MDL numbers (§1.4) are bits per byte of a fitting corpus and belong to a different axis.

Do the effects survive a real training budget?

Two facts about the tiny model set up the design. It is far from converged at step 1,000, and past its optimum it memorizes: on the 6MB training partition it reaches train 0.44 against validation 3.45 BPB by step 43k. One 50,000-update run per condition therefore yields the exact result of every shorter protocol, because the best checkpoint within a budget is a prefix property of a single run.

Both effects survive and both shrink. The 1,000-update protocol reproduces exactly, which validates the tracked runs. Converged training cuts \Delta_{\text{data}} by 3.2× and \Delta_{\text{pretok}} by 1.6×. At 27M parameters the ordering of the two effects reverses: how much text the tokenizer saw nearly stops mattering (+0.013), while the choice of pretokenization does not (+0.062). The estimation effect decays with LM scale. The hypothesis-class effect persists.

B1c Vocabulary sweep at convergence

The left arm flattens: V = 1\text{K} improves from 2.256 to 2.146, because at 1,000 updates its longer token sequences were undertrained. That gives a flat optimum across 1K to 4K (2.143 to 2.157) and monotone degradation from 8K up (2.163, 2.239, 2.322, 2.648 at 8K / 16K / 32K / 65K). Vocabularies at or above 8K reach their validation best within 250 to 1,000 steps and then overfit the 1.5MB LM training corpus. The U keeps its location. Its right arm at this scale is downstream LM estimation, not tokenizer estimation. The fixed-vocabulary experiments, not the sweep, carry the identification.

Consequence for the paper. Quote converged numbers everywhere. Present the sweep as motivation, with the resource confound named. At scale, boundary design matters more than fitting-data scarcity.

Which error bar belongs on these comparisons?

At 100KB the tokenizer sample contributes 4.4× the standard deviation the seed does. Part of that is mechanical: whole-document sampling makes realized sizes vary (101 to 154KB at the 100KB scale) and BPB tracks realized bytes monotonically there, so roughly half the spread is byte-budget variation and half is genuine sampling noise. Either way the practical rule is that a comparison between tokenizers fitted on about 100KB carries \pm 0.02 to 0.03 BPB, not \pm 0.005.

Applied to branch 1: the converged \Delta_{\text{data}} = +0.053 stays about 2.3× above this bar and survives. The scaled +0.013 does not, so any claim about fitting data at the 27M protocol needs its own sample replication before it can be asserted.

Can a vocabulary be chosen with no held-out data?

Three reference points make the argmin readable. The count-two frontier is the first merge in the list that applies at most twice in the fitting corpus. Past it, merges are supported by almost no evidence. The held-out order-zero optimum is the vocabulary minimizing a Dirichlet-smoothed unigram code length measured on the tokenizer-validation partition, so it is the \text{BPB}_{\text{uni}} minimum of §1.3 with held-out data available. The Occam-bound argmin is the vocabulary minimizing the right-hand side of §2.4.

The curve is U-shaped at every scale and its minimum moves right with fitting data, stopping just below the count-two frontier every time, without ever seeing held-out text. It is conservative by 0.3 to 1.5 octaves (factors of two) against the held-out order-zero optimum. Self-check: replaying the merge list and re-encoding the corpus directly give identical token counts at every tested vocabulary.

The instantiated Occam bound fails. Its right-hand side lands between 32 and 139 bits per byte against a trivial byte code at 8, and its argmin is the base vocabulary 256 at every scale, because \ell_{\max}\sqrt{L(h)\ln 2 / 2m} with document-level blocks is far larger than the empirical rate. The bound is valid but carries no information. What MDL does not predict is the downstream LM optimum, and the branch 1c sweep prices the difference. At 1.5MB MDL selects 19,745 types. The converged sweep on the same corpus gives 2.239 LM-BPB at 16K and 2.322 at 32K, against 2.143 to 2.157 on the 1K to 4K plateau, so training the tiny LM at the MDL pick costs roughly +0.1 BPB. That is not a defect to repair: the downstream optimum at this scale is set by how much LM training data there is, MDL is blind to the LM by construction, and the two criteria answer different questions. A vocabulary for a data-limited LM should be chosen with the LM in the loop.

Framing shipped to the paper. MDL selection is the operative criterion. The Occam bound is motivation for the two-part structure, not a selector.

Is pretokenization a regularizer or a better hypothesis class?

The three \Delta columns agree at matched scales, so pretokenization does not shrink merge-estimation error: it is not a regularizer. The oracle column spreads by 0.130 to 0.136 BPB, so the entire pretokenization advantage appears only in the limit: it is a better hypothesis class. Separately, raw BPE reaches its own asymptote by 6MB, about 370 fitting bytes per vocabulary type. More tokenizer text stops helping well before any production scale.

The paper's open question therefore has a negative answer at fixed vocabulary. The regularization explanation survives only in the vocabulary-sweep sense, and possibly at extreme scarcity: raw BPE's excess at 100KB is 0.166, in a regime where the constrained methods cannot even reach 16,384 types.

Does any of this depend on English?

• \Delta_{\text{data}} = +0.173 (SD 0.005), against +0.167 in English at the same protocol. The estimation phenomenon is not English-specific.
• \Delta_{\text{pretok}} = +0.045 at 6MB (SD 0.0005), about 40% of the English effect and positive at every scale. Most of the English gain is whitespace handling. Class boundaries account for the rest.
• GPT-2 emits 10% more tokens per byte at every scale and still has lower BPB at every scale. This is the §1.2 failure mode: TPB and LM-BPB rank the two tokenizers in opposite orders, and only one of the two rankings corresponds to a shorter description of the text.
• Count-one stopping (truncating the merge list at the first merge applying once, 7,381 types at 100KB) gains +0.016 over the full 16,384-type list. The language-agnostic repair transfers.
• SentencePiece unigram (same 6MB, V = 16,384, byte-exact round trips on all 416 documents, run August 30) reaches 2.3430 \pm 0.0045, below GPT-2 BPE by +0.0148 paired (per-seed +0.0191, +0.0107, +0.0147, positive for every seed). The branch 6 algorithm advantage is not English-specific, at about 35% of its English size, and SP again pairs the most tokens per byte (0.262) with the lowest BPB.

What is lost to a stronger baseline?

Shrinkage at equal total budget makes no difference. Splitting the same 100KB into a 39.5KB fitting stream and a 60.7KB scoring stream (weight 0.75, selected on tokenizer validation as before) gives 2.1512 ± 0.0017 against 2.1535 ± 0.0044 for plain raw BPE on the full 100KB. The tracked shrinkage gain (2.049) was its extra 282KB of independent text, not the estimator.

SentencePiece unigram, fitted on the same 6MB at the same 16,384 types, has lower BPB than every BPE variant in the study, including the 50MB GPT-2-pretokenized oracle, with byte-exact round trips verified on all 385 documents (identity normalization, no dummy prefix, byte fallback). The pretrained GPT-2 tokenizer reaches a lower BPB still under the fixed-body protocol (the transformer body is identical in every condition and only the embedding table grows with V, so its 50,257 types also give the LM more parameters). Algorithm choice moves BPB by 0.042, more than any within-BPE intervention at 6MB (pretokenization variants differ by about 0.01, and text beyond 6MB gives about 0.02).

Paper consequence: add both baselines, and either extend the statistical framing to unigram fitting (it is likelihood-based, so it is a natural fit) or scope the claims to BPE explicitly. A reviewer who runs either baseline would otherwise discover this first.

Does the ordering hold at 544M parameters?

Pretokenization persists. GPT-2 pretokenization leads raw by 0.018 to 0.023 BPB from 0.5B tokens onward, flat to the end of training. The final full-validation gap is +0.018 (0.9009 against 0.8834). The first evaluation, at 0.26B tokens, shows +0.080 and is not converged; it is excluded from the chart. Read against branch 1, \Delta_{\text{pretok}} now traces +0.117, +0.071, +0.062, +0.018 across four protocols, though the last point also changes corpus (FineWeb-Edu rather than WikiText-103) and drops to one seed, so it is a scale trend with two confounds, not a clean fourth point.

The algorithm gap collapses. At the tiny protocol SentencePiece unigram beat GPT-2 BPE by 0.042 BPB. At 544M the final full-validation gap is +0.0016 (0.8834 against 0.8817), a 26× shrinkage, and the subset trajectory falls from +0.002 at 1.5B tokens to +0.0007 over the last five evaluations. No seed replicate exists at this scale, and at 27M the seed SD was about 0.002, so the remaining gap is within a plausible seed bar. The boundary-design choice survives scale and the fitting-algorithm choice does not, which inverts branch 6's headline at the scale that matters.

The TPB column is the §1.2 contradiction at production scale: raw byte BPE emits 11% fewer tokens per byte than GPT-2 and 12% fewer than SentencePiece, so it is better on the metric the field reports and worse on the number of bits needed to describe the text. SentencePiece has both the most tokens per byte and the lowest BPB of the three.

Final-test protocol

Three partitions have never been read: confirmatory test (1.01MB English), wikipedia-zh test (1.00MB Chinese), and the FineWeb-Edu final_test slice (40.7MB). Every number on this page comes from validation partitions, and validation numbers steered analysis choices, so they are not unbiased estimates. The test partitions exist to buy one unbiased pass.

Protocol. One pass, run once, after the paper's claims are frozen and on an explicit go. No new training: the existing best checkpoints are evaluated once on the test partition beside their validation partition. Each claim below is a sign of a paired per-seed difference. A claim is confirmed when that sign holds for every seed pair on test. Magnitudes are reported beside the validation magnitudes without re-tuning, and a disagreement is reported as a disagreement, not re-run.

The converged-tiny and scaled effect sizes reuse the confirmatory and scaled test partitions with the same rule. The scaled \Delta_{\text{data}} claim enters this table only if the sample replication below is run first; without it, branch 2 says the claim cannot be asserted at any confidence worth testing.

Remaining runs

Changes already in the paper

• Abstract and introduction: converged-audit summary, the class-versus-estimation answer, and the MDL contribution, all in \claudetext.
• Section 4: new subsection "An instantiated two-part stopping rule" with the KT construction, the exact-replay computation, argmin results, the vacuous-bound result, and a pgfplots figure of all four MDL curves.
• Section 5: a "Robustness protocols" paragraph; converged U-curve results in 5.1; effect-robustness table and tokenizer-sample variance in 5.2; new subsections for excess risk (with table), external baselines, and the Chinese replication; the equal-budget shrinkage control appended to 5.3.
• Discussion, related work, conclusion: reframed division of labor; Gowda & May and Krichevsky-Trofimov citations added; limitations updated to what actually remains.
• Not yet written: the 544M scale check (a Section 5 paragraph, the algorithm-gap result, and a softening of the "results reach 27M parameters and 96MB" limitation once the SentencePiece run lands); the metric-disagreement table of §1.7 as a display table; the final-test pass and whatever the queued runs change.

Artifacts and scripts

experiments/artifacts/summaries/ · 2026-08-campaign-findings.md (index) · budget-robustness-findings.md · tokvar-findings.md · mdl-stopping-findings.md · excess-risk-findings.md · zh-replication-findings.md · branch6-baselines-findings.md · campaign-all-runs.csv (111 runs) · mdl-*.csv · excess-risk.csv · tokvar-variance.csv · budget-robustness.csv · mdl-stopping.pdf
experiments/scripts/ · runners run_{budget_sweep,scaled_protocol,vocab_sweep_converged,tokvar,oracle_lm,zh,branch6,master_chain}.sh · fits fit_{pool_tokenizers,mdl_artifacts,zh_tokenizers,equal_budget_shrinkage,sentencepiece}.py|sh · analysis mdl_stopping.py, render_mdl.py, summarize_{budget_sweep,tokvar,excess_risk}.py · data download_wikipedia.py · external LM train_transformer_external.py · scale check scale06b_{prepare_slices,fit_tokenizers,encode,train,chain}.py|sh

Gitignored but regenerable: the WikiText download is prefix-stable by construction, every BPE fit is deterministic, and manifests carry content hashes. Lint and the 40-test suite pass. Both English and Chinese final-test partitions, and the 40.7MB FineWeb-Edu test slice, remain unread.