Simpson's paradox

What it is

Simpson's paradox is when combining groups flips a pattern. Something can be true in every group separately, yet look the opposite when all the groups are added together. It can also work the other way round.¹˒²

It happens when a hidden third factor is spread unevenly across the groups.³ Statisticians call such a factor a confounder. See the Confounding variable entry.

The Correlation vs causation entry gives two real examples. One is about graduate admissions at Berkeley in 1973. Overall, women were admitted less often than men, but department by department there was little sign of bias against them.⁴ The other is about two kidney-stone operations, where the operation that did better for small stones and for large stones looked worse overall.⁵˒³ This entry explains how the flip works, with a simple imaginary example.

How it works: two basketball players

Imagine two basketball players, A and B. Here are their shooting records, with made-up numbers.

Player A is better at close shots. Player A is also better at long shots. Yet overall, Player B looks far better.

How? Most of B's shots were easy close ones. Most of A's shots were hard long ones. The overall figure mixes together how good each player is with which shots they took. Shot difficulty is the hidden factor. Once you split the shots by difficulty, the real picture appears.

Why it matters

Simpson's paradox isn't just a maths puzzle. It turns up in medicine, university admissions, sports statistics, and anywhere groups are compared. Whenever one option is used mostly in harder situations, its overall record can look worse than it deserves.

It also shows why randomized trials are so useful. When chance decides who gets which treatment, the hard and easy cases end up spread evenly between the groups. That stops this kind of flip from happening. See the Randomized controlled trial entry.

Should you trust the split or the combined numbers?

It depends on the story behind the data, not on the arithmetic.³

If a hidden factor affects both which group something falls into and how it turns out, like shot difficulty, then the split numbers tell the fairer story. But splitting by the wrong factor can mislead too. Before choosing, ask: what could be causing what?

In everyday life

• Hospital league tables. Imagine a hospital that takes on the sickest patients. Its overall survival rate may look worse than a hospital that treats milder cases, even if it does better for every type of patient.
• School results. A school's overall exam results depend on which students it takes in, not only on how well it teaches.
• Job applications. As at Berkeley, overall acceptance rates can hide what is happening group by group.⁴

The common trap

The first trap is trusting an overall figure without asking what it combines.

The second trap is thinking that splitting the data is always better. It's only better when you split by a factor that really is confusing the picture.

Some ways out:

• Ask: "Is one option used more often in harder cases?"
• Look at the figures group by group, where they are available.
• Think about which factor could be affecting both the grouping and the result.

Try it

Make up your own Simpson's paradox. Take two imaginary shops and two kinds of customer complaint, easy and hard. Give Shop A a better success rate for both kinds, but give it far more hard complaints. Work out the overall success rates. Can you make Shop B look better overall?

Related concepts

Correlation vs causation · Confounding variable · Randomized controlled trial · Sample size

Sources

1. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B, 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.x

2. Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. Journal of the American Statistical Association, 67(338), 364–366. https://doi.org/10.1080/01621459.1972.10482387

3. Julious, S. A., & Mullee, M. A. (1994). Confounding and Simpson's paradox. BMJ, 309(6967), 1480–1481. https://doi.org/10.1136/bmj.309.6967.1480

4. Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. https://doi.org/10.1126/science.187.4175.398

5. Charig, C. R., Webb, D. R., Payne, S. R., & Wickham, J. E. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. BMJ, 292(6524), 879–882. https://doi.org/10.1136/bmj.292.6524.879