Cohen's h: When 'Statistically Significant' Doesn't Mean 'Material'

I wrote previously about using Wilson confidence intervals to monitor static behavioural assumptions — prepayment rates, attrition, defaults, and the like. It's still a genuinely good tool: transparent, auditable, and it beats "eyeballing the latest number" by a wide margin. If you haven't read it, it's worth a look before this one.

But Wilson intervals have a quirk that shows up the moment your sample size gets large, and it's worth naming directly.

The Wilson interval's width shrinks roughly with $1/\sqrt{n}$. That's exactly what you want statistically — more data, more precision. But it also means that once $n$ gets into the tens or hundreds of thousands (a large retail mortgage book, a high-volume SaaS subscriber base, a national survey), the interval becomes razor-thin.

Say your static assumption is an 8.0% prepayment rate, and this quarter's observed rate is 8.3%. With 500 mortgages, that 0.3 percentage-point wobble is well within noise — the interval might be [5.9%, 10.9%], and 8.0% sits comfortably inside. Green light.

Now run the same 8.0% vs 8.3% comparison with 300,000 mortgages. The interval might be [8.15%, 8.45%]. Your 8.0% assumption is now outside the band. Red flag, escalate, recalibrate.

This is a well-known statistical trade-off: statistical significance is a function of sample size, not of practical importance. At large n, everything becomes "significant" eventually, because the test is answering "is there any detectable difference at all," not "is this difference big enough to matter." A governance process built purely on Wilson intervals at scale ends up generating a flood of technically-correct-but-practically-noisy alerts, which is exactly the kind of thing that erodes trust in the monitoring process itself.

The Wilson interval answers one question well: is the observed rate statistically distinguishable from the assumption, given the data I have?

That's a different question from: is the gap between the two rates big enough that I should actually care?

For the second question, you want an effect size — a measure of the distance between two proportions that does not depend on sample size at all. That's exactly what Cohen's h gives you.

Cohen's h is defined using the arcsine (variance-stabilising) transformation. For two proportions $p_1$ and $p_2$:

The arcsine transform is the trick that makes this work: it stretches proportions near 0% and 100% (where a raw percentage-point difference is "bigger" in relative terms) and compresses the middle, so that a given value of $h$ represents roughly the same perceptual gap in proportions, regardless of whether you're comparing 2% vs 3% or 45% vs 46%. Critically, $h$ has no $n$ in it anywhere — sample size never enters the calculation.

Cohen's own rule-of-thumb bands for interpreting $|h|$ are:

Going back to the mortgage example: 8.0% vs 8.3% gives $\varphi_1 = 0.583$, $\varphi_2 = 0.605$, so $h \approx 0.022$. That's negligible by any reasonable standard — regardless of whether n is 500 or 500,000. Cohen's h correctly tells you this is not a material change, even in the exact scenario where the Wilson interval was screaming "recalibrate."

This isn't an argument to throw out Wilson intervals — it's an argument to pair them with an effect-size check. A sensible two-gate governance rule looks like this:

This keeps the Wilson interval's strength — an objective, auditable "is anything happening at all" trigger — while stopping it from flooding your ALM or risk committee with statistically-real-but-practically-irrelevant fails whenever the book is large. It also protects the other direction: with a small sample, a large, business-relevant swing might still sit inside a wide Wilson band and get waved through. Checking $h$ alongside the interval catches that case too, since $h$ doesn't care how wide your confidence band happens to be.

Adjust the baseline rate, the observed rate, and the sample size below. Watch how the p-value (which reacts to sample size) and Cohen's h (which doesn't) can tell very different stories about the same gap.

Push the sample size slider up while keeping the two rates close together, and you'll see the p-value collapse toward zero while $h$ barely moves. That's the whole point: $h$ is measuring the thing that actually determines whether you should care.

Wilson intervals remain the right first-pass tool for objective, auditable monitoring of a static behavioural assumption — I stand by that earlier post. But at scale, "statistically significant" and "worth escalating" stop being the same thing. Pair the Wilson check with Cohen's h (or another sample-size-independent effect size) so your governance process fails on drift that's actually material, not just drift that's technically detectable.

If your book, cohort, or dataset is large enough that quarterly monitoring keeps flagging assumptions that only moved by a whisker, this is very likely why — and it's a five-minute fix.