Skip to main content
    Path to Science

    Thinking Dictionary

    Multiple lines of evidence

    What it is

    Multiple lines of evidence means supporting a conclusion with several different kinds of evidence. Each kind could go wrong, but in its own way. If they still all point to the same answer, the conclusion becomes much harder to explain away than any single piece of evidence.¹˒²

    Scientists who study the causes of disease have a name for this: triangulation.¹

    Why it matters

    Any one study, test, or story can be wrong for reasons you can't see. A measuring device might be off. The people studied might be unusual. Or it might just be luck.

    When several kinds of evidence agree, and each has different weak spots, it becomes unlikely that luck or one hidden flaw explains them all.¹ That is why the most useful lines of evidence have unrelated weaknesses. Ideally, their possible errors would even push in opposite directions.¹

    The key word is independent. Ten studies that all share the same flaw are closer to one line of evidence than to ten. Simply repeating the same kind of study can repeat the same mistake. That is why some researchers argue that repeating a study, on its own, isn't enough. You also need different methods.²

    A real case: the water in Flint, Michigan

    In April 2014, the city of Flint, in the United States, switched its drinking water to the Flint River. The water was not treated to stop it from eating away at old pipes. Treatment like that is called corrosion control.³˒⁴ What followed shows how separate clues, each easy to dismiss alone, added up.

    • What residents saw. People reported water that looked discoloured.³ On its own, a complaint about colour or taste is easy to wave away. It can't measure anything.
    • A car factory. According to news reports, in October 2014 General Motors said the river water was corroding engine parts at its Flint plant, and it arranged to use different water. This detail comes from news coverage, not from a scientific study.⁵
    • One house. In 2015, researchers tested the water at a single home. Its lead levels climbed steeply, from 104 to 397 to 707 micrograms per litre.³ But one house could be a strange exception.
    • Children's blood. Researchers led by Mona Hanna-Attisha looked at blood tests of young children. The share of children under five with raised lead in their blood went from 2.4% to 4.9% after the switch. The rise was largest in areas with the most lead in the water. Outside the city, where the water hadn't changed, there was no clear change.⁶ That outside group acted as a built-in comparison.

    Each clue alone had an easy excuse: one odd house, a matter of taste, factory chemistry. Together, and with the comparison outside the city, they were very hard to dismiss. The public health emergency came after the blood-lead study in September 2015.³

    In everyday life

    • Is your phone slower since the update? Your impression is one line. The phone's own battery or speed figures are another. A friend with the same phone and update is a third. Each could be wrong in a different way, so agreement among them means more.
    • Lots of reviews. Fifty glowing reviews may feel like fifty lines of evidence. But if they could all come from the same kind of source, they share the same weak spot. A careful independent test would add more than another fifty similar reviews.
    • Your own health. How you feel, a measurement at home, and a test at the doctor's are different kinds of evidence. When they agree, you can be more confident. When they don't, that is worth a closer look.

    The common trap

    The trap is counting evidence instead of weighing it. "Lots of studies say so" sounds strong. But if the studies share a design flaw or a funder, they may count as little more than one.

    The opposite trap is expecting every line to agree. When lines of evidence clash, that is useful information. It shows which weakness to check next.¹

    And don't throw out personal reports. In Flint, what residents noticed was a real early warning. Its weakness was that it couldn't measure anything by itself.

    This tool has a long history. In 1965, Austin Bradford Hill suggested several viewpoints for judging whether a link between two things is a cause. Several of them are about whether different kinds of evidence fit together. He was clear that they were not a checklist.⁷

    Try it

    Pick a claim you believe, such as "my phone's battery is worse since the update." Write down every kind of evidence you have: your feeling, the battery figures, a friend's phone, a review site. Next to each, write one way it could be wrong. Now circle the ones whose weak spots are different from each other. How many truly independent lines do you really have?

    Related concepts

    Correlation vs causation · Confirmation bias · Credible sources · Senses vs instruments · Iteration · Cherry-picking

    Sources

    1. Lawlor, D. A., Tilling, K., & Davey Smith, G. (2017). Triangulation in aetiological epidemiology. International Journal of Epidemiology, dyw314. https://doi.org/10.1093/ije/dyw314

    2. Munafò, M. R., & Davey Smith, G. (2018). Robust research needs many lines of evidence. Nature, 553(7689), 399–401. https://doi.org/10.1038/d41586-018-01023-3

    3. Pieper, K. J., Tang, M., & Edwards, M. A. (2017). Flint water crisis caused by interrupted corrosion control: Investigating "ground zero" home. Environmental Science & Technology, 51(4), 2007–2014. https://doi.org/10.1021/acs.est.6b04034

    4. Masten, S. J., Davies, S. H., & McElmurry, S. P. (2016). Flint water crisis: What happened and why? Journal AWWA, 108(12), 22–34. https://doi.org/10.5942/jawwa.2016.108.0195

    5. MLive. (2014, October 13). General Motors shutting off Flint River water at engine plant over corrosion worries. MLive. https://www.mlive.com/news/flint/2014/10/general_motors_wont_use_flint.html

    6. Hanna-Attisha, M., LaChance, J., Sadler, R. C., & Champney Schnepp, A. (2016). Elevated blood lead levels in children associated with the Flint drinking water crisis: A spatial analysis of risk and public health response. American Journal of Public Health, 106(2), 283–290. https://doi.org/10.2105/AJPH.2015.303003

    7. Hill, A. B. (1965). The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine, 58(5), 295–300. https://doi.org/10.1177/003591576505800503