# How to think about these magnitudes

Every quantity is an **average over an idiom's observed contexts**. For one context `c`:

- **`p(c)`** — the LM score of the **idiom** in context `c` (how likely / concentrated the idiom is there).
- **`q(c)`, `r(c)`** — the LM scores of the **verb-literal** and **noun-literal** pools; **`m(c) = max{q(c), r(c)}`** is the *better single component word*.
- **Synergy** (the operational definition for this whole project): the idiom is **more concentrated than either word alone predicts**, i.e. `p(c) > m(c)`. Idioms should show more of this than literal VPs — that is the hypothesis.

### Surprisal basics (read this first)
`−log(x)` is a **surprisal**: smaller `x` → bigger `−log(x)`; `x → 0` → `+∞`; `x = 1` → `0`. So *any* "`−log` of a score" reads as **"bigger = rarer / less concentrated."** That single fact explains every direction below — and why the original `H_s` points "backwards."

---

## Step by step, one metric at a time

### `H(p)` — base entropy  ·  range `≥ 0`  ·  **↓ smaller = idiom more concentrated**
`H(p) = mean_c −log p(c)`. How spread the idiom is across contexts. **Low** `H(p)` = the idiom lives in a few tight contexts. In every config idioms have **lower** `H(p)` than literal VPs — they are more fixed. This is the denominator for all the ratios.

### `H_u` — unique / redundant entropy  ·  range `≥ H(p)`  ·  *(read via the ratio)*
`H_u = mean_c −log min{p, m}`. Because `−log` is decreasing, `−log min{p,m} = max(−log p, −log m)` — the surprisal of whichever of (idiom, best word) is **less** concentrated. The raw nats aren't very interpretable on their own; the **ratio** below is the signal.

### `H_u / H(p)` — **THE headline**  ·  range `≥ 1`  ·  **↑ bigger = MORE synergy / more idiomatic**
- `= 1`: the idiom is no more concentrated than its best single word → **no synergy**.
- `> 1`: the idiom concentrates **beyond** what either word explains → **synergy**.

Idioms sit significantly **higher** than literal VPs in **all** configs — the operational synergy signature. If you only look at one number, look at this one.

### `syn_frac` — synergy coverage  ·  range `0…1`  ·  **↑ bigger = MORE synergy**  ← most intuitive
Fraction of the idiom's contexts that are synergistic (`p > m`). `1.0` = every context synergistic; `0` = fully compositional everywhere. This is the plain-English "how idiomatic across contexts" number — and it is *exactly* "how many contexts escaped the `−log(0)` blow-up in the original `H_s`."

### `H_s` — synergy entropy (original)  ·  range `≥ 0` **or `+∞`**  ·  **↓ bigger = LESS synergy; `+∞` = some compositional context**  ⚠ brittle
`H_s = mean_c −log max{0, p − m}`. The "excess" `p − m` is the **amount** of extra concentration (more = more synergy), but `−log` **inverts** it:

| synergy at context | excess `p − m` | contribution `−log(excess)` |
|---|---|---|
| lots | large | **small** |
| a little | ~0⁺ | **large** |
| none (`p ≤ m`) | `0` | **`+∞`** |

So a **big** `H_s` means **little** synergy, and **`+∞` means at least one fully-compositional context** — i.e. *less* idiomatic, not more. Because one zero poisons the whole average, `H_s` is finite only when **every** context is synergistic, so it is `+∞` for most phrases and is essentially unusable per-phrase. The two variants below fix this.

### `H_s^log` — log-space synergy  ·  range `≥ 0`  ·  **↑ bigger = MORE synergy**  ·  finite always  ← recommended magnitude
`H_s^log = mean_c max{0, log p − log m}` = the average **surprisal reduction** the idiom buys over its best word. `0` = no context synergistic; grows with how much *more* concentrated the idiom is. Right direction, never blows up. (Signed twin `mean(log p − log m)` may go **negative** for a net anti-synergistic phrase.) `H_s^log / H(p)` is the normalized version.

### `H_s^reg` — regularized H_s  ·  range `≥ H(p)`  ·  **↓ bigger = LESS synergy**  ·  finite & continuous
Same construction as `H_s`, but the excess is **floored at `ε·p`** (`ε = 0.01`) instead of `0`. A non-synergistic context then contributes `−log(ε·p) = (−log p) + |log ε|` — a large-but-**finite** penalty (`|log ε| ≈ 4.6` nats above that context's own surprisal) instead of `+∞`. It equals `H_s` exactly on fully-synergistic phrases, and otherwise **rises smoothly with the number of compositional contexts**, so you can finally tell a phrase with one bad context from one with five (the original `H_s` calls both `+∞`).

> Worked example (gpt2): `have a ball` has 1 non-synergistic context → `H_s = +∞`, `H_s^reg = 6.43`. `get the sack` has 2 → `H_s = +∞`, `H_s^reg = 7.80`. Same `+∞` under the original, cleanly separated under the regularized version.

---

## Direction cheat-sheet

| metric | range | direction | one-liner |
|---|---|---|---|
| `H(p)` | ≥ 0 | **↓** smaller = more concentrated | how spread the idiom is over contexts |
| `H_u/H(p)` | ≥ 1 | **↑** bigger = MORE synergy | **headline** — idiom beyond its best word |
| `syn_frac` | 0…1 | **↑** bigger = MORE synergy | fraction of contexts that are synergistic |
| `H_s^log` (and `/H`) | ≥ 0 | **↑** bigger = MORE synergy | avg surprisal reduction vs best word (recommended) |
| `H_s^reg` (and `/H`) | ≥ H(p) / ≥ 1 | **↓** bigger = LESS synergy | finite, continuous twin of the original H_s |
| `H_s` (orig, and `/H`) | ≥ 0 or +∞ | **↓** bigger = LESS synergy | `+∞` ⇔ a compositional context exists (brittle) |

**Mental model.** `H_u/H(p)`, `syn_frac`, and `H_s^log` all point **up with synergy** and are the ones to trust. `H_s` and `H_s^reg` are *surprisals of the excess*, so they point **down with synergy** (and the original additionally explodes to `+∞`). They aren't a true partial-information decomposition — the two `−log` terms don't even sum to `H(p)` (they sum to `≥ 2H(p) + 2log2`) — so treat "synergy" here as the concrete claim *idiom more concentrated than either word predicts*, best measured by `H_u/H(p)`.

---

## Is `H_u/H(p)` just measuring `H(p)`?

There is an exact identity: **`H_u = H(p) + H_s^log`**, hence

```
H_u / H(p) = 1 + H_s^log / H(p) = 1 + ratio_s_log.
```

So the headline ratio is literally `1 +` (synergy term) `÷` (base entropy). Since
idioms also have **lower** `H(p)`, dividing by `H(p)` is **not** "controlling for"
it — a ratio rescales rather than partials out, so a smaller denominator alone can
inflate it. The full step-by-step derivation, plus the statistical test that
disentangles the two, is the **"Is it just H(p)? (derivation)"** section of
[`report.html`](report.html). The verdict (rerun via `python code/confound_check.py`):

- An **ANCOVA** `H_s^log ~ idiom + H(p)` shows the idiom effect on synergy
  **survives controlling for `H(p)`** (pooled β = +0.49 → +0.42, ~13% attenuation,
  p ≈ 1e-25).
- An exact **decomposition** of the ratio gap attributes **~92%** to the synergy
  numerator `H_s^log` and only **~8%** to the `H(p)` denominator.

So the ratio is **not** an `H(p)` artifact — but the clean practice is to report the
additive synergy `H_s^log = H_u − H(p)` and test it with the ANCOVA, treating the
lower `H(p)` of idioms as a real, *separate* finding rather than folding it into a ratio.