# How to read the magnitudes This is the prose version of the report's "How to read the magnitudes" and "H_s vs H_s^log" tabs (`report.html`). ## Setup: population entropies and their estimators Fix a phrase and let `p` be its distribution over **contexts**. We observe a finite set of contexts, treated as an i.i.d. sample `c_1, …, c_N ~ p`. Per context `c`: - **`p(c) ∈ (0,1]`** — the LM's length-normalized score of the phrase in `c`. - **`q(c)`, `r(c)`** — mean scores of the verb-/noun-literal pools; **`m(c) = max{q(c), r(c)}`** is the better single component word. - **`U(c) = min{p(c), m(c)}`** (covered part), **`S(c) = max{0, p(c) − m(c)}`** (synergistic excess) — so `U(c) + S(c) = p(c)` pointwise — and **`s(c) = log p(c) − log m(c)`** (log-synergy). - A context is **synergistic** when `p(c) > m(c)`, i.e. `s(c) > 0`: the idiom is more concentrated there than either component word alone. Each reported quantity is defined as a **population expectation under `p`** — an entropy (or, for syn_frac, a probability), e.g. `H(p) := E_{c~p}[−log p(c)]`. These expectations cannot be evaluated directly, so every number in the tables and figures is the corresponding **Monte-Carlo estimator**: the sample mean over the observed contexts, written with a hat, ``` Ĥ(p) := (1/N) Σ_i −log p(c_i), ``` and likewise for the others. The entropies themselves are not "means over contexts" — the sample means are their estimators. The bootstrap CIs quantify the estimators' sampling error. ## The metrics - **`H(p) = E[−log p(c)]`** — base entropy of the phrase over contexts; smaller = more concentrated. Idioms have lower `Ĥ(p)` than literal VPs in every config. - **`H_u = E[−log U(c)]`** — surprisal of the covered (redundant) part. Since `U(c) ≤ p(c)`, `H_u ≥ H(p)` always; the informative quantity is the ratio. - **`H_u / H(p) ≥ 1`** — the primary comparison, estimated by `Ĥ_u / Ĥ(p)`. Equals 1 iff no observed context is synergistic; grows with synergy. - **`H_s = E[−log S(c)]`** — the synergy entropy: the surprisal of the synergistic excess. The original, principled definition — `S` is the quantity that makes the split `U + S = p` additive. It *decreases* as synergy grows, and `Ĥ_s = +∞` as soon as one context has `S(c) = 0`. - **`H_s^log = E[max{0, s(c)}]`** — log-space synergy: the average surprisal reduction the idiom achieves over its best component word. Always finite; increases with synergy; satisfies the exact identity `Ĥ_u = Ĥ(p) + Ĥ_s^log`. (A signed variant `E[s(c)]` can go negative.) - **`H_s^reg = E[−log max{S(c), ε·p(c)}]`, `ε = 0.01`** — `H_s` with the excess floored at `ε·p`: a non-synergistic context contributes `−log p(c) + log(1/ε)` (large but finite) instead of `+∞`. Decreases with synergy; continuous in the number of non-synergistic contexts. - **`syn_frac = P_{c~p}(S(c) > 0)`** — the probability that a context is synergistic, estimated by the observed fraction. `syn_frac = 1` is exactly the condition for `Ĥ_s` to be finite. ## Direction conventions `H(p)`, `H_u`, `H_s`, `H_s^reg` are expectations of surprisals `−log(·)`, so they **decrease** as their argument grows; `H_s^log` and `syn_frac` **increase** with synergy. | metric | range | direction | one-liner | |---|---|---|---| | `H(p)` | ≥ 0 | smaller = more concentrated | spread of the phrase over its contexts | | `H_u/H(p)` | ≥ 1 | increases with synergy | primary comparison: concentration beyond the best component word | | `syn_frac` | [0,1] | increases with synergy | estimated P(context is synergistic) | | `H_s^log` (and `/H`) | ≥ 0 | increases with synergy | mean surprisal reduction vs the best word; `= H_u − H(p)` | | `H_s^reg` (and `/H`) | > H(p) / > 1 | decreases with synergy | `H_s` with the excess floored at `ε·p`; finite, continuous | | `H_s` (and `/H`) | > H(p) or +∞ | decreases with synergy | surprisal of the excess; `+∞` iff some context is non-synergistic | ## H_s and H_s^log: the exact relationship On a synergistic context (`s(c) > 0`) the excess factorizes against the idiom's own score: ``` S(c) = p(c) − m(c) = p(c) · (1 − e^{−s(c)}) ⟹ −log S(c) = −log p(c) + φ(s(c)), φ(s) := −log(1 − e^{−s}). ``` This is a deterministic, pointwise identity, so the sample mean preserves it: ``` Ĥ_s = Ĥ(p) + (1/N) Σ_i φ(s(c_i)) if all s(c_i) > 0, else +∞ ``` (verified in the sweep data to machine precision). `φ` is strictly decreasing, with `φ(s) → +∞` as `s ↓ 0` and `φ(s) ≈ e^{−s} → 0` as `s → ∞`. So per context, `Ĥ_s^log` averages `s` itself while `Ĥ_s` averages the strictly decreasing transform `φ(s)` of the *same* variable, plus the base surprisal. This accounts for every observed behavior of `H_s`: (i) its direction is inverted (more synergy ⇒ smaller `Ĥ_s`); (ii) it has a pole at zero synergy (one context with `s ≤ 0` ⇒ `Ĥ_s = +∞`); (iii) it mixes concentration with synergy (the `Ĥ(p)` offset), whereas `Ĥ_s^log = Ĥ_u − Ĥ(p)` isolates the synergy term. **The AM–GM bound.** For every context `U(c), S(c) ≥ 0` and `U(c) + S(c) = p(c)`, so `U·S ≤ ((U+S)/2)² = p²/4`, hence ``` −log U(c) − log S(c) ≥ −2 log p(c) + 2 log 2, ``` with equality iff `U(c) = S(c)`, i.e. `m(c) = p(c)/2`, i.e. `s(c) = log 2`. Averaging (sample mean or population expectation alike): ``` Ĥ_u + Ĥ_s ≥ 2·Ĥ(p) + 2 log 2 ≥ Ĥ(p) ``` — the `bound_ok` check in `main.py`, which holds for every phrase in every config. The two surprisal terms therefore do **not** sum to `H(p)`: the additive split `E[U] + E[S] = E[p]` lives in linear space. Substituting the identity `Ĥ_u = Ĥ(p) + Ĥ_s^log` gives the pure synergy form `Ĥ_s + Ĥ_s^log ≥ Ĥ(p) + 2 log 2`, whose pointwise version is `s + φ(s) ≥ 2 log 2`, minimized exactly at `s = log 2`. **Where `Ĥ_s` is finite.** Exactly when every observed context is synergistic (`syn_frac = 1`). In the medial · geo configs this holds for most idiom phrases but few non-idiom phrases (e.g. gemma2-9b: 13/18 idioms vs 3/18 non-idioms) — itself a comparison in the expected direction. See the "H_s vs H_s^log" tab of `report.html` for the figure (the `φ` curves and the empirical distribution of `s` per context). ## Is `H_u/H(p)` just measuring `H(p)`? There is an exact identity `Ĥ_u = Ĥ(p) + Ĥ_s^log`, hence ``` Ĥ_u / Ĥ(p) = 1 + Ĥ_s^log / Ĥ(p) = 1 + ratio_s_log. ``` So the ratio is `1 +` (synergy term) `÷` (base entropy). Since idioms also have lower `Ĥ(p)`, dividing by `Ĥ(p)` is not "controlling for" it — a ratio rescales rather than partials out, so a smaller denominator alone can inflate it. The step-by-step derivation and the statistical tests are in the "Is it just H(p)?" tabs of `report.html` (rerun via `python code/confound_check.py`): - An **ANCOVA** `H_s^log ~ idiom + H(p)` shows the idiom effect on synergy survives controlling for `H(p)` (pooled β = +0.49 → +0.42, ~13% attenuation, p ≈ 1e-25). - An exact **decomposition** of the ratio gap attributes ~92% to the synergy numerator `H_s^log` and ~8% to the `H(p)` denominator. The clean practice is to report the additive synergy `H_s^log = H_u − H(p)` and test it with the ANCOVA, treating idioms' lower `H(p)` as a real, separate finding rather than folding it into a ratio.