H_u / H_s model sweep

\(H(p)\) differs sharply between idioms and non-idioms (idioms lower) and \(H(p)\) sits in the denominator of the headline ratio. So before claiming idioms have more synergy, we must rule out that "idioms have higher \(\hat H_u/\hat H(p)\)" is merely "idioms have smaller \(\hat H(p)\)". This step fixes the exact algebra; step 2 turns it into a test; step 3 reports the numbers.

1 · Setup: population quantities and their estimators

Fix an idiom and let \(p\) be its distribution over contexts. We do not observe \(p\); we observe a finite set of contexts which we treat as an i.i.d. sample \(c_1,\dots,c_N \sim p\). For a context \(c\), \(p(c)\in(0,1]\) is the LM's length-normalized score of the idiom there; \(q(c),r(c)\) are the scores of the verb-/noun-literal pools, and \(m(c):=\max\{q(c),r(c)\}\) is the better single component word (all strictly positive). The surprisal of a score \(x\in(0,1]\) is \(-\log x \ge 0\).

The quantities of interest are population expectations under \(p\) (estimands):

We cannot evaluate these expectations directly, so for each phrase we report the Monte-Carlo estimators — the sample means over the observed contexts. This is why we average:

By the law of large numbers each is consistent and unbiased for its estimand, \(\mathbb{E}\bigl[\hat H(p)\bigr]=H(p)\) (and likewise for \(\hat H_u,\hat H_s^{\log}\)). Everything downstream — the tables, bootstrap CIs and regressions — operates on these per-phrase estimators, and the CIs quantify their sampling error.

2 · Lemma (pointwise)  \(-\log\min\{a,b\} = \max(-\log a,\,-\log b)\)

For \(a,b>0\): \(\min\{a,b\}\) is the smaller value and \(-\log\) is strictly decreasing, so the log of the smaller value is the larger surprisal. Hence \(-\log\min\{a,b\} = \max(-\log a,\,-\log b)\). \(\;\blacksquare\)

3 · Theorem  \(H_u = H(p) + H_s^{\log}\)  (and identically \(\hat H_u = \hat H(p) + \hat H_s^{\log}\))

Apply the Lemma with \(a=p(c)\), \(b=m(c)\), then the elementary identity \(\max(\alpha,\beta)=\alpha+\max(0,\beta-\alpha)\) with \(\alpha=-\log p(c)\), \(\beta=-\log m(c)\). This gives, for every context \(c\) (a deterministic, pointwise statement):

Because it holds pointwise, applying \(\mathbb{E}_{c\sim p}[\cdot]\) to both sides gives the population identity \(H_u = H(p)+H_s^{\log}\); applying the sample mean \(\tfrac1N\sum_i[\cdot]\) gives the same identity for the estimators:

The split is therefore exact at both the population and the estimate level (verified in the data to ~1e-15). Corollary, dividing the estimators by \(\hat H(p)\):

So the headline ratio is exactly \(1\) plus the synergy estimator rescaled by the base-entropy estimator. It is therefore not independent of \(\hat H(p)\) — which is exactly why step 2 is needed.

The confound, stated precisely

Let \(D=1\) for idioms, \(0\) for non-idioms. Empirically \(D\) shifts both \(\hat H(p)\) (down) and \(\hat H_u/\hat H(p)\) (up). By the identity in step 1, a group difference in the ratio can come from (i) a larger numerator \(\hat H_s^{\log}\), or (ii) a smaller denominator \(\hat H(p)\), or both. The substantive, theory-backed claim is (i) — the idiom concentrates beyond what its best word predicts. We must isolate (i) from (ii).

Why dividing is not controlling

Forming the ratio does not remove the \(\hat H(p)\) effect. "Controlling for \(\hat H(p)\)" means estimating the group effect at a fixed value of \(\hat H(p)\) — a conditional (partial) effect. A ratio instead rescales by \(\hat H(p)\): for equal absolute synergy, the phrase with smaller \(\hat H(p)\) gets a larger ratio, so the ratio can move purely because the denominator shrank. We therefore test the numerator directly, two ways.

Method A — ANCOVA (partial effect at fixed H(p))

Let phrase \(j\) have estimators \(S_j := \hat H_s^{\log}\) and \(H_j := \hat H(p)\) (from step 1), and \(D_j\in\{0,1\}\). Across the \(2{\times}18\) phrases fit by ordinary least squares:

\(\beta\) is the idiom–nonidiom difference in synergy holding \(H(p)\) fixed. We test \(H_0:\beta=0\) with \(t=\hat\beta/\operatorname{se}(\hat\beta)\), \(\mathrm{df}=n-3\), where \(\operatorname{se}=\sqrt{\bigl[\hat\sigma^2 (X^{\top}X)^{-1}\bigr]_{\beta\beta}}\) and \(\hat\sigma^2=\mathrm{RSS}/\mathrm{df}\). Comparing \(\hat\beta\) to the uncontrolled slope from \(S=\alpha+\beta D\) shows how much of the raw synergy gap \(H(p)\) accounts for. The pooled fit adds model fixed effects (dummies).

Method B — exact decomposition of the ratio gap

With group means \(\bar S_i,\bar S_n,\bar H_i,\bar H_n\), add and subtract \(\bar S_n/\bar H_i\):

The first term is the gap that would remain if both groups shared idioms' \(H(p)\); the second is the gap that would remain if both groups shared non-idioms' synergy. Each term's share of \(\Delta R\) is the % of the ratio gap that source explains.

Caveat — H(p) is partly a mediator

\(H_s^{\log}\) and \(H(p)\) are negatively correlated (more concentrated ⇒ more synergy; see the correlation column in step 3). If low \(H(p)\) is itself a consequence of idiomaticity, then \(H(p)\) is a mediator, not a pure confounder, and the ANCOVA \(\beta\) under-states the total idiom effect. So the controlled \(\beta\) is a conservative lower bound; the true effect lies between it and the uncontrolled slope.

Per-config (medial · geo) and pooled across the five models; the two methods are defined in step 2. All entries are the per-phrase MC estimators from step 1 (group means \(\bar S,\bar H\) are over phrases). Rerun for any mode/reduction with python code/confound_check.py --mode MODE --red RED.

Decomposition of the ratio gap (Method B). Per-config group means, the ratio gap ΔR, and the share carried by the synergy numerator vs the H(p) denominator.

ANCOVA on H_s^log (Method A). Idiom coefficient β without and with H(p) as a covariate, the % the effect shrinks when H(p) is controlled, the H(p) slope γ, and the H_s^log–H(p) correlation. β stays large and significant at fixed H(p).

Conclusion

Practice. Report the synergy as the additive H_s^log = H_u − H(p) and test it with the ANCOVA above (or its non-finite-safe siblings), rather than treating H_u/H(p) as if it controlled for H(p). The ratio is a fine convenience index, but it conflates "more synergy" with "lower base entropy". Rerun with python code/confound_check.py --mode MODE --red RED.