Skip to main content
    Path to Science

    Thinking Dictionary

    Random vs systematic error

    What it is

    In science, an error is not necessarily a mistake. It is the unavoidable gap between a measurement and the true value. Errors come in two main kinds.¹˒²

    • Random error is the unpredictable wobble in any measurement. Sometimes a reading is a bit too high, sometimes a bit too low. If you take enough readings and average them, random error mostly cancels out.
    • Systematic error, also called bias, is a steady push in one direction. Think of a scale that always reads 1 kg too heavy, or a survey that only reaches certain kinds of people. Taking more readings won't fix it. You have to find the cause and remove it.¹˒²

    Why it matters

    It is tempting to think that more data always means a better answer. That is only true for random error. With systematic error, more data just gives you the same wrong answer with more confidence.

    The way to catch systematic error is to check your method against something whose true value you already know. This is called calibration.¹ If your scale reads 1.1 kg for a bag you know weighs 1 kg, you have found a systematic error.

    A famous failure: the 1936 election poll

    In 1936, the United States held a presidential election between Franklin Roosevelt and Alf Landon. A popular magazine, the Literary Digest, ran an enormous poll. It mailed out more than 10 million ballots and got back over 2.3 million. That is fewer than one in four.

    The poll predicted that Landon would win, with 55% of the vote against 41% for Roosevelt. In fact, Roosevelt won by a landslide, 61% to 37%.³

    How could a poll of over two million people get it so wrong? Because its errors were systematic. A bigger sample doesn't help if the sample is skewed. The popular explanation is that the magazine only polled wealthy people who owned cars and telephones. That is only part of the story. A later study found that the problem was both who was sent ballots and, especially, who chose to send them back. If everyone who received a ballot had replied, the poll would at least have picked the right winner.³

    The lesson is a lasting one. Size is no protection against bias.

    Margins of error only cover random error

    Modern polls report a margin of error, such as "plus or minus 3 points." This covers only the random error that comes from asking some people rather than everyone. It does not include other problems, such as people who refuse to take part or questions that are worded in a leading way.⁴ So a small margin of error tells you nothing about bias.

    In everyday life

    • Your bathroom scale. If it reads a little differently each time, that's random error. Weigh yourself a few times and take the average. If it always reads 2 kg too high, that's systematic error. Averaging won't help. You need to check it against a known weight.
    • Online reviews. Thousands of reviews may still be skewed. The people who bother to write a review may be the ones who loved or hated the product. That is a bit like the 1936 poll.
    • A kitchen oven. If your oven always runs 20 degrees hotter than the dial says, every recipe will cook too fast. Baking more cakes won't reveal it. An oven thermometer will.

    The common trap

    The trap is believing that "more data means more accuracy." That is true only for random error. A huge sample with a hidden bias can be more misleading than a small, fair one, because its size makes it look trustworthy.

    Some ways out:

    • Ask how the data were collected. Who was included? Who was left out? Who chose not to respond?
    • Check against something known. Calibrate your instrument, or compare a survey with a result you can verify.
    • Don't be impressed by size alone. Ask about bias first.

    Try it

    You need a kitchen scale and something whose weight you know, such as a sealed 500 g packet.

    1. Weigh the packet 10 times, lifting it off between readings. Write down each number. The spread of the numbers is random error.
    2. Now put the scale on a folded towel, or on a slight slope. Weigh the packet 10 more times.
    3. Compare the two averages.

    If the average moved, you have created a systematic error. Notice that taking more readings in the second position wouldn't bring the average back. Only fixing the cause will.

    Related concepts

    Accuracy vs precision · Signal vs noise · Error bars and confidence intervals · Senses vs instruments · Correlation vs causation · Sample size

    Sources

    1. Joint Committee for Guides in Metrology. (2012). International vocabulary of metrology (VIM) (JCGM 200:2012, 3rd ed.). BIPM. https://www.bipm.org/documents/20126/2071204/JCGM_200_2012.pdf

    2. Menditto, A., Patriarca, M., & Magnusson, B. (2007). Understanding the meaning of accuracy, trueness and precision. Accreditation and Quality Assurance, 12(1), 45–47. https://doi.org/10.1007/s00769-006-0191-z

    3. Squire, P. (1988). Why the 1936 Literary Digest poll failed. Public Opinion Quarterly, 52(1), 125–133. https://doi.org/10.1086/269085

    4. Mercer, A. (2016, September 8). 5 key things to know about the margin of error in election polls. Pew Research Center. https://www.pewresearch.org/short-reads/2016/09/08/understanding-the-margin-of-error-in-election-polls/