Skip to main content
    Path to Science

    Thinking Dictionary

    Error bars and confidence intervals

    What it is

    Many results are reported as a single number: "48% support the plan." But no measurement is perfectly exact. An error bar or a confidence interval shows the range where the true value probably lies.

    On a graph, an error bar is usually a small line sticking up and down from a point or a bar. In writing, the range might appear as "48%, plus or minus 3 points." That means the real figure is plausibly anywhere from about 45% to 51%.

    The lesson is simple: before trusting a precise-looking number, ask "plus or minus what?"

    What "95% confidence" means

    Polls often say their margin of error is calculated "at 95% confidence." Here is a plain way to understand it. Imagine the same survey were done 100 times, each time with a new random group of people. In about 95 of those surveys, the result would land within the margin of error of the true value.¹

    People often read it a little differently: "there's a 95% chance the true value is inside this range." That sounds almost the same, but strictly speaking it isn't what the method promises. This is a common misreading.²˒³

    Why it matters

    A range is more honest than a single, falsely exact number. Many experts recommend that research findings be reported with confidence intervals, not just as "significant" or "not significant."⁴ And there is little cost in being honest about uncertainty: in experiments, giving a range of numbers lowered people's trust only slightly.⁵

    Confidence intervals are also widely misunderstood, even by experts. In one study, 120 researchers and 442 students were shown six false statements about a confidence interval. On average, both groups agreed with more than three of them. The researchers did hardly any better than the students.²

    The gap between two numbers is less certain than each number

    This is one of the most useful points about margins of error. Imagine a poll in which each candidate's support has a margin of plus or minus 3 points. You might think a 4-point lead is clear. It isn't. The margin of error for the gap between the two candidates is roughly twice as big, about plus or minus 6 points.¹

    The same goes for changes over time. If two polls each have a margin of plus or minus 3, the margin for the change between them is about plus or minus 8 points.¹ So a lead or a change that looks bigger than the margin of error may still be within the noise.

    You may also hear a rule of thumb that if two error bars overlap, there is no real difference. Treat that as a rough simplification at best. It can mislead, partly because error bars don't all show the same thing.

    Not all error bars mean the same thing

    Error bars on graphs can show different things. Some show how spread out the measurements were. Others show a confidence interval. They can look the same but mean quite different things. Always read the caption or the small print to find out what the bars show.

    And remember that a margin of error covers only one kind of error: the random wobble from studying some people rather than everyone. It doesn't include problems such as people refusing to take part, or questions worded in a leading way.¹ A narrow interval can still be wrong if the method was biased.

    In everyday life

    • Election polls. "Party A leads by 2 points" may mean very little if the margin for the gap is about 6.
    • Health studies. A news story says a treatment cut risk by 20%. Look for the range. "Somewhere between 2% and 38%" tells a very different story from "between 18% and 22%."
    • Product tests. A battery "lasts 10 hours." Plus or minus what? Under what conditions?

    The common trap

    The trap is treating a precise number as a trustworthy number. "48.3%" looks more exact than "about 48%, give or take 3." But the second is more honest.

    Another trap is thinking a narrow range means a correct result. A biased method can give a narrow range around the wrong answer.

    Some ways out:

    • Always ask "plus or minus what?"
    • When comparing two numbers, remember that the uncertainty on the gap is bigger.
    • Check what the error bars on a graph actually show.

    Try it

    Find a poll in the news, and look for the margin of error. It is often in the small print. Write each party's or candidate's figure as a range. For example, 48% plus or minus 3 becomes 45% to 51%.

    Now look at the "lead" in the headline. Remember that the margin for the gap between the top two is about twice the stated margin. Is the lead really a lead?

    Related concepts

    Scientific uncertainty · Signal vs noise · Random vs systematic error · Probabilistic reasoning · Accuracy vs precision · Sample size

    Sources

    1. Mercer, A. (2016, September 8). 5 key things to know about the margin of error in election polls. Pew Research Center. https://www.pewresearch.org/short-reads/2016/09/08/understanding-the-margin-of-error-in-election-polls/

    2. Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21(5), 1157–1164. https://doi.org/10.3758/s13423-013-0572-3

    3. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3

    4. Cumming, G. (2014). The new statistics. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

    5. van der Bles, A. M., van der Linden, S., Freeman, A. L. J., & Spiegelhalter, D. J. (2020). The effects of communicating uncertainty on public trust in facts and numbers. PNAS, 117(14), 7672–7683. https://doi.org/10.1073/pnas.1913678117