Skip to main content
    Path to Science

    Thinking Dictionary

    Cherry-picking

    What it is

    Cherry-picking means showing only the evidence that supports a point and leaving out the rest. Imagine picking only the ripest cherries from a tree, then saying the whole tree looks like that.

    The evidence that is shown may be completely real. What misleads is what has been left out. Cherry-picking is close to Confirmation bias: it is the selection part, whether done on purpose or by accident.

    Why it matters

    In everyday arguments, cherry-picking is easy to do and hard to spot. In research, it takes two main forms:

    • Publication bias. Studies with disappointing results are less likely to be published. So the published research looks more positive than the full picture.¹
    • Outcome-reporting bias. Within a study, the results that didn't work out are dropped, or swapped for ones that did. Before a trial starts, researchers usually write a plan, called a protocol, that says what they will measure. Comparing the published results with that plan shows whether anything changed.¹

    These problems don't need anyone to fake data. Real results, selectively shown, are enough to mislead.

    How big the problem is

    Antidepressant trials. Researchers looked at 74 trials of antidepressants registered with the US drug regulator, the FDA. Almost a third (31%) were never published. In the published research, it looked as if 94% of the trials had positive results. The FDA's own analysis found that only about half (51%) did. Selective publishing made the drugs look about a third more effective, on average, than the full set of trials showed.²

    Changing the goalposts. A review of earlier studies found that in 40% to 62% of trials, at least one main outcome had been changed, added, or left out compared with the original plan.¹ In another project, researchers checked every trial published over six weeks in five top medical journals. For 58 of the 67 trials (87%), they found misreported outcomes and wrote letters asking for corrections. Only 40% of those letters were published.³ Even top journals don't reliably catch the problem.

    In everyday life

    • Charts in the news or in adverts. Imagine a share price that is "up 50% this month." Over the whole year, it may still be down. The choice of start and end dates is a form of cherry-picking. The Signal vs noise entry explains why short time windows can mislead.
    • Product reviews. A company's website shows only its glowing reviews.
    • Arguments. Someone cites three studies that support them, and doesn't mention the ten that don't.

    The common trap

    The first trap is thinking "every fact in it is true, so it must be honest." True facts, carefully selected, can still mislead.

    The second trap is thinking cherry-picking is always deliberate. It often happens without anyone choosing to hide anything. For example, studies that found nothing may simply never get written up.

    The third trap is thinking that more cited studies always means a stronger case. Always ask what was left out.

    Some ways out:

    • Ask what's missing. What would the full picture look like?
    • Look for the whole body of evidence, ideally a review that gathers all the studies on a question. See the Multiple lines of evidence entry.
    • Check the time window. Would a longer period tell a different story?
    • Check the plan. For a trial, compare what it reports with what it planned to measure.

    Try it

    Find a chart in a news story or an advert. Ask yourself why it starts and ends where it does. Then look up the same data over a longer period. Does the story still hold?

    Related concepts

    Confirmation bias · Multiple lines of evidence · Peer review · Misinformation vs disinformation · Sample size · Signal vs noise

    Sources

    1. Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., et al. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE, 3(8), e3081. https://doi.org/10.1371/journal.pone.0003081

    2. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358(3), 252–260. https://doi.org/10.1056/NEJMsa065779

    3. Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., et al. (2019). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20. https://doi.org/10.1186/s13063-019-3173-2