How to Read a Health Claim

What a headline leaves out

We start with a plain question: what kind of evidence sits behind this claim? A headline compresses a press release, and a press release compresses a paper. The losses are not random. A 2014 BMJ study of 462 press releases, issued in 2011 by 20 UK universities, found that 40% contained more direct advice than the journal article, 33% overstated causal links drawn from correlational research, and 36% inferred effects in humans from animal, cell or simulation studies. When a release made the animal-to-human leap, 86% of the related news stories did too, against 9.6% when it did not. The study is observational, so it shows an association and not proof of cause, and its figures date from 2011. We think the pattern persists, but this source cannot show that.

Questions to ask before believing it

The US National Center for Complementary and Integrative Health publishes a checklist for understanding health news stories. It asks whether the work was tested on people, because animal or laboratory findings may not yet matter for health. It asks whether enough people were studied, whether the size of the difference matters to you and not only to scientists, whether the story discussed limitations, and whether participants resemble you. It asks whether the study ran long enough to show long-term effects, and whether the funder could profit from the findings. It also reminds readers that one study rarely proves anything. The page carries no date, and it is a minimum, not a substitute for specialist appraisal. The Cambridge Medical Library describes the Behind the Headlines service, in which clinicians and scientists review health stories and the underlying studies and write impartial assessments.

Bias is not the same as chance

Two different problems can make a study mislead. Imprecision, or random error, is the noise that comes with small samples. Bias is systematic error that pushes results in one direction. The Cochrane Handbook, version 5.2.0, chapter 8 defines bias as a systematic error that can cause a study to overestimate or underestimate the true effect, and treats imprecision as a separate concept. It sets out seven domains for judging risk of bias, including how participants were allocated, whether participants and assessors were blinded, whether outcome data were missing and whether outcomes were selectively reported. It also cites one meta-epidemiological study in which effects were exaggerated by 41% in trials with inadequate allocation concealment judged high risk, and by 30% in those judged unclear. A large study can be biased, and a careful small study can be imprecise. Readers should ask both questions.

A worked example of absolute and relative risk

The same result can be described in two ways, and the choice changes the impression. The Handbook gives the standard definitions in its chapter on relative and absolute risk reductions: relative risk reduction is 100% x (1 - RR), the risk difference is often called the absolute risk reduction, and the number needed to treat is a simple transformation of the risk difference.

Here is an example, illustrative, not real data. Suppose 8 in 1,000 people in a comparison group have an event over five years, against 6 in 1,000 in the other group.

• RR = 6/8 = 0.75
• Relative risk reduction = 100% x (1 - 0.75) = 25%
• Absolute risk reduction = 8/1,000 - 6/1,000 = 2/1,000 = 0.2 percentage points
• Number needed to treat = 1 / 0.002 = 500

A headline could truthfully say "cuts risk by 25%". It could equally say "2 fewer events per 1,000 people". Only the second tells a reader how much changes for one person.

Common mistakes

The first is reading a relative figure without the baseline. A 25% reduction of a rare event is small in absolute terms, and the same 25% of a common event is large. The second is treating one study as settled. The third is accepting a causal verb, such as "causes" or "prevents", for a study that only observed groups. The fourth is skipping the population: results in one group may not carry to another. The fifth is confusing a large effect on a convenient measure with a change that matters to people. We hold ourselves to the same tests when we write, and we expect to be caught when we fail them.

Going deeper

The core arithmetic fits on one page. With risk defined as events divided by people in a group:

• Risk = events / people in the group
• RR = risk in the intervention group / risk in the comparison group
• Relative risk reduction = 100% x (1 - RR)
• Absolute risk reduction = risk in comparison group - risk in intervention group
• Number needed to treat = 1 / absolute risk reduction, with the reduction written as a proportion

Three habits help when reading a primary paper. Look for confidence intervals, as the BMJ study reports for its percentages, for example 40% (33% to 46%). Check that the outcomes reported are the outcomes the study set out to measure. Notice which effect measure is used: the BMJ study reports odds ratios of 6.5, 20 and 56 for how exaggeration travels from release to story, and odds ratios can look far larger than the percentages beneath them, so we quote the percentages first.

We invite you to check every link above against the source. If we have misread one, we want to know.

Informational only, not medical advice.