Statistical Significance

What statistical significance actually is

Statistical significance is a way of asking, “Could this result just be a fluke?” When a study finds a difference between two groups, researchers test how likely that difference is to show up by random chance if there were really no effect at all. If chance is an unlikely explanation, the result is called statistically significant.

The usual cutoff is a p-value below 0.05, meaning there’s less than a 5% chance of seeing a result this big if nothing real were going on.

How it works

Researchers start with the assumption that there’s no true effect, then check how surprising their data would be under that assumption. A small p-value says the data would be surprising if nothing were happening, so they conclude something probably is. A larger p-value means chance can’t be ruled out.

The 0.05 line is a convention, not a law of nature. It’s a tradeoff that tries to balance missing real effects against being fooled by random noise. Some fields use stricter cutoffs depending on what’s at stake.

What it isn’t

This is the part that trips people up. Statistical significance does not mean a result is large, useful, or important. A study with thousands of people can find a “significant” effect that’s far too small to matter in real life. The significance is about chance, not size; effect size and confidence intervals tell you how big the result is.

It also isn’t proof. A significant finding can still be a false alarm, especially in small studies or when many comparisons are run at once.

Related terms you’ll see next

• P-value
• Confidence interval
• Effect size
• Randomized controlled trial

Why it matters when you read about mental health

Headlines love the word “significant,” and it’s easy to read it as “big deal.” But a treatment can be statistically significant and still barely help anyone. When you see the claim, it’s worth asking two more questions: how large was the effect, and how wide was the confidence interval? Those answers tell you whether a significant result actually means something for real people.

<blockquote>
  <p><strong>Statistical Significance:</strong> Statistical significance means a result is unlikely to be due to chance alone. It does not tell you whether the result is large or important.</p>
  <p>&mdash; <a href="https://shrinktionary.com/terms/statistical-significance/">Shrinktionary</a>, a plain-English mental health dictionary, medically reviewed by a board certified psychiatrist.</p>
</blockquote>