Correlation vs causation

What it is

A correlation means two things tend to go up or down together. Causation means that changing one thing actually changes the other.

A correlation can happen because one thing causes the other. But it can also happen for other reasons. So a correlation on its own isn't proof of a cause.¹˒² It is often the first clue, though. The skill is to ask: "What else could explain this?"

Why it matters

Almost every day we hear that something is "linked to" better health or a longer life. Some of those links are real causes. Many are not.

If we treat every link as a cause, we change habits, spend money, and worry for nothing. If we ignore every link, we miss real warnings. The case that smoking causes lung cancer was built mainly from studies that simply compared smokers with non-smokers.³˒¹

News reports often make this harder. In one study of 462 university press releases, a third claimed a stronger cause than the research showed. When a press release exaggerated, 81% of the news stories based on it did too. When it didn't, only 18% did.⁴ So when you read "linked to," "associated with," or "tied to," it usually means a correlation was found, not a cause.

Four ways a link can fool you

1. A hidden third factor. Something else causes both things. Scientists call this a confounder, and the problem confounding. Ice cream sales and drownings rise and fall together. Ice cream doesn't make anyone drown. Hot weather brings both. Across 17 European countries, more stork pairs went along with more human births. Bigger countries simply have more of both.⁵

2. The arrow points the other way. This is called reverse causation. Austin Bradford Hill, a scientist who studied patterns of disease, asked it neatly: "which is the cart and which is the horse?" Does a certain diet lead to disease? Or do the early stages of a disease change what people eat?¹ Here is a picture of how this can happen. In a study of British men, non-drinkers seemed to die sooner than light drinkers. That looks as if a little alcohol protects you. But the pattern showed up only in men who were already ill, and many of them had stopped drinking because they were ill.⁶ The illness came first. (The wider debate about moderate drinking is still going. This example only shows how reverse causation works. It is not a verdict on alcohol.)

3. Pure chance. Compare enough things, and some will match by luck. Across 23 countries, the amount of chocolate people ate closely tracked the number of Nobel prize winners per person.⁷ People who search huge piles of data for matches are guaranteed to find some striking but meaningless ones.⁸

4. How the groups were split. Sometimes a pattern flips when you split the data into groups. This is called Simpson's paradox.⁹˒¹⁰ In 1973, the University of California, Berkeley admitted about 44% of men who applied to its graduate programmes but only 35% of women. That looks like bias against women. But department by department, there was little sign of it, and if anything a small tilt in favour of women. Women more often applied to departments that were hard for everyone to get into.¹¹

A real-life reversal: hormone therapy

For years, many studies watched women who chose to take hormone replacement therapy (HRT) after menopause. These women had about half as much heart disease. Fifteen of 16 such studies pointed the same way. The researchers had tried hard to account for other differences between the women, and they thought a hidden third factor was unlikely.¹²

Then came randomized trials. In a randomized trial, chance (like a coin toss) decides who gets the treatment and who doesn't. That way, people's own choices, health, or wealth can't decide it. One trial, in women who already had heart disease, found no overall benefit.¹³ Another, in healthy women aged 50 to 79, found a small increase in heart disease and was stopped early.¹⁴

Why the gap? Experts still debate it. Women who chose HRT tended to be better off throughout their lives, and that may have made the pills look better than they were.¹⁵ Later work suggests that much of the gap also came from who was studied and when they started. The earlier studies mostly followed long-term users and missed harms early on. The big trial mostly studied women well past menopause.¹⁶ The lesson is not "HRT is bad." That trial was about preventing heart disease, and treating menopause symptoms is a separate question. The lesson is that the earlier studies pointed the wrong way, and a randomized trial was needed to find out.

The common trap

The trap is believing that a strong, "significant," or carefully checked link must be a cause.

A p-value is a number researchers use to judge whether a pattern could easily be a fluke. A small p-value means "probably not just luck." The stork link had a small p-value (0.008).⁵ But a small p-value says nothing about a hidden third factor or which way the arrow points. Hill himself said such tests only help guard against chance.¹ And adjusting for other factors can't fix what nobody measured. That is part of what went wrong with HRT.¹²˒¹⁴

Some ways out:

• Run through the four stories. Could a third factor cause both? Could the arrow point the other way? Could it be chance? Could the way people were grouped create the pattern?
• Ask whether it was randomized. A randomized trial is the strongest single tool, but not the only one. Trials can be wrong too, especially small ones, and they only answer the question for the people they studied. Still, in one review of highly cited medical research, findings from non-randomized studies were later overturned or shrunk more often (5 of 6) than findings from randomized trials (9 of 39). The numbers are small, though.¹⁷
• When a trial isn't possible, use Hill's viewpoints. Hill suggested several ways to look at a link. How strong is it? Cigarette smokers had nine to ten times the lung cancer death rate of non-smokers.¹ Do different studies agree? Does more exposure go with more effect? Did the cause come before the effect? Hill said these are viewpoints, not a checklist. None of them can prove a cause, and none is strictly required.¹ The one true requirement is that the cause comes first.²
• Don't swing to "correlation means nothing." In 1950, among 649 men with lung cancer, only 2 were non-smokers. Among 649 similar men without lung cancer, 27 were.³ Evidence like that, checked from many angles, can show a real cause.

Try it: the kidney-stone puzzle

Here are real results for two operations to remove kidney stones.¹⁸˒¹⁹ Look only at the bottom row first. Which operation is better?

Keyhole surgery wins overall. Now look at the rows above. Open surgery did better for small stones and for large stones. Can you explain the flip in one sentence?

The answer: open surgery was mostly used on large stones, which are harder to treat. Its overall score was dragged down by its harder cases, not by the operation.¹⁹ Stone size was the hidden third factor.

One more thing to spot: the open operations were done in 1972–80 and the keyhole ones in 1980–85. Other things changed over those years too, so even this comparison isn't perfectly fair.¹⁹

Then try it on the news. Find a headline that says "linked to." Write down four possible stories: X causes Y; Y causes X; something else causes both; or it's chance. Then open the study. Was it randomized? If not, which of your stories did it rule out?

Related concepts

Confounding variable · Indicators of causation · Reverse causation · Simpson's paradox · Randomized controlled trial · Placebo effect · Confirmation bias · Multiple lines of evidence

Sources

1. Hill, A. B. (1965). The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine, 58(5), 295–300. https://doi.org/10.1177/003591576505800503

2. Rothman, K. J., & Greenland, S. (2005). Causation and causal inference in epidemiology. American Journal of Public Health, 95(S1), S144–S150. https://doi.org/10.2105/AJPH.2004.059204

3. Doll, R., & Hill, A. B. (1950). Smoking and carcinoma of the lung: Preliminary report. BMJ, 2(4682), 739–748. https://doi.org/10.1136/bmj.2.4682.739

4. Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., Venetis, C. A., Davies, A., Ogden, J., Whelan, L., Hughes, B., Dalton, B., Boy, F., & Chambers, C. D. (2014). The association between exaggeration in health related science news and academic press releases: Retrospective observational study. BMJ, 349, g7015. https://doi.org/10.1136/bmj.g7015

5. Matthews, R. (2000). Storks deliver babies (p = 0.008). Teaching Statistics, 22(2), 36–38. https://doi.org/10.1111/1467-9639.00013

6. Shaper, A. G., Wannamethee, G., & Walker, M. (1988). Alcohol and mortality in British men: Explaining the U-shaped curve. The Lancet, 332(8623), 1267–1273. https://doi.org/10.1016/S0140-6736(88)92890-5

7. Messerli, F. H. (2012). Chocolate consumption, cognitive function, and Nobel laureates. New England Journal of Medicine, 367(16), 1562–1564. https://doi.org/10.1056/NEJMon1211064

8. Vigen, T. (2015). Spurious correlations. Hachette Books. ISBN 978-0-316-33943-8. Website: https://www.tylervigen.com/spurious-correlations (no DOI; book, not peer reviewed)

9. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.x

10. Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. Journal of the American Statistical Association, 67(338), 364–366. https://doi.org/10.1080/01621459.1972.10482387

11. Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. https://doi.org/10.1126/science.187.4175.398

12. Stampfer, M. J., & Colditz, G. A. (1991). Estrogen replacement therapy and coronary heart disease: A quantitative assessment of the epidemiologic evidence. Preventive Medicine, 20(1), 47–63. https://doi.org/10.1016/0091-7435(91)90006-P

13. Hulley, S., Grady, D., Bush, T., Furberg, C., Herrington, D., Riggs, B., & Vittinghoff, E. (1998). Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. JAMA, 280(7), 605–613. https://doi.org/10.1001/jama.280.7.605

14. Writing Group for the Women's Health Initiative Investigators. (2002). Risks and benefits of estrogen plus progestin in healthy postmenopausal women: Principal results from the Women's Health Initiative randomized controlled trial. JAMA, 288(3), 321–333. https://doi.org/10.1001/jama.288.3.321

15. Lawlor, D. A., Davey Smith, G., & Ebrahim, S. (2004). Commentary: The hormone replacement–coronary heart disease conundrum: Is this the death of observational epidemiology? International Journal of Epidemiology, 33(3), 464–467. https://doi.org/10.1093/ije/dyh124

16. Hernán, M. A., Alonso, A., Logan, R., Grodstein, F., Michels, K. B., Willett, W. C., Manson, J. E., & Robins, J. M. (2008). Observational studies analyzed like randomized experiments: An application to postmenopausal hormone therapy and coronary heart disease. Epidemiology, 19(6), 766–779. https://doi.org/10.1097/EDE.0b013e3181875e61

17. Ioannidis, J. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. https://doi.org/10.1001/jama.294.2.218

18. Charig, C. R., Webb, D. R., Payne, S. R., & Wickham, J. E. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. BMJ, 292(6524), 879–882. https://doi.org/10.1136/bmj.292.6524.879

19. Julious, S. A., & Mullee, M. A. (1994). Confounding and Simpson's paradox. BMJ, 309(6967), 1480–1481. https://doi.org/10.1136/bmj.309.6967.1480