Skip to main content
    Path to Science

    Thinking Dictionary

    Sample size

    What it is

    Sample size is how many people, animals, or measurements a study includes. Scientists often write it as n. A study "with n = 50" included 50 people or measurements.

    Small samples are easily swayed by luck. They can show big effects that later disappear, or miss effects that are real. Larger samples give steadier answers. But a large sample can't fix a sample that was chosen in a biased way.

    Why it matters

    Think of flipping a coin. Flip it 10 times, and you might easily get 7 heads. That doesn't mean the coin is unfair. Flip it 1,000 times, and you will get much closer to half heads. Small samples bounce around more, so they produce extreme results more often, just by chance.

    The same is true in research. A poll of 1,067 people chosen at random has a margin of error of about plus or minus 3 points. But if you look at a small group within that poll, say around 160 people, the margin grows to about plus or minus 8 points.¹ So be careful with headlines about small groups inside a larger survey.

    As a rule of thumb from basic statistics, to halve the margin of error you need about four times as many people.

    Why small studies can mislead

    A study's statistical power is its ability to detect an effect that is really there. Small studies have low power. That means they often miss real effects.

    But there is a less obvious problem too. When a small, low-powered study does find a striking result, that result is less likely to be true. And even when it is true, the study tends to overestimate how big the effect is.² One influential argument, though its headline claim is debated, lists small studies as one of several factors that make published findings less likely to be true.³

    This is why exciting results from small studies often shrink or vanish when someone repeats the study with more people.

    Our gut trusts small samples too much

    People tend to expect a small sample to look just like the whole population. Psychologists Amos Tversky and Daniel Kahneman called this belief, half jokingly, the "law of small numbers."⁴ In everyday life, it shows up when we judge a whole restaurant by one meal, or a whole country by one visit.

    Bigger isn't automatically better

    A large sample reduces random error. It can't fix a biased sample. In 1936, a magazine poll collected over two million replies and still predicted the wrong winner of the US presidential election, because of who received the ballots and who chose to reply.⁵ The Random vs systematic error entry tells that story in full.

    So when judging a study, ask two separate questions: How big was the sample? And how was it chosen?

    In everyday life

    • "A study of 12 people found…" That doesn't mean the study is worthless. Small studies can be useful first steps. But treat the result as a clue, not a conclusion.
    • Product reviews. A product with three five-star reviews isn't necessarily better than one with 2,000 reviews averaging 4.4 stars. Three reviews can easily be lucky.
    • Your own experience. "My grandfather smoked and lived to 95" is a sample of one. It can't tell you about the risk for most people.
    • Small groups in polls. "Young voters swing sharply" may rest on a small subgroup with a large margin of error.¹

    The common trap

    The first trap is trusting striking results from small studies. They are the ones most likely to shrink later.²

    The second trap is the opposite: thinking a huge sample guarantees the right answer. It doesn't if the sample is biased.⁵

    The third trap is reading "no significant effect" as "no effect." A study that is too small may simply have missed it.

    Some ways out:

    • Look for the sample size. It is often written as "n =".
    • Be extra cautious with dramatic findings from small studies.
    • Ask how the sample was chosen, not just how big it was.
    • Wait for the result to be repeated with more people.

    Try it

    Flip a coin 10 times and write down how many heads you got. Do this five times. Then flip it 50 times, and do that five times. You can also use a free online coin-flip simulator.

    Work out the share of heads for each set. Compare how much the share jumps around for the 10-flip sets and for the 50-flip sets. Which ones stayed closer to half?

    Related concepts

    Random vs systematic error · Error bars and confidence intervals · Randomized controlled trial · Signal vs noise · Probabilistic reasoning · Cognitive shortcuts · Cherry-picking · Simpson's paradox · Regression to the mean

    Sources

    1. Mercer, A. (2016, September 8). 5 key things to know about the margin of error in election polls. Pew Research Center. https://www.pewresearch.org/short-reads/2016/09/08/understanding-the-margin-of-error-in-election-polls/

    2. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475

    3. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

    4. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105–110. https://doi.org/10.1037/h0031322

    5. Squire, P. (1988). Why the 1936 Literary Digest poll failed. Public Opinion Quarterly, 52(1), 125–133. https://doi.org/10.1086/269085