Statistical Significance

Imagine if everyone in the medical profession had no idea what temperature and its consequences meant. Ignorance of such a fundamental concept would result in confussion and carnage. Statistical Significance is like temperature, but significantly more significant and consequential. Not just consequential to the medical profession, but to anyone or any profession doing experiments, making claims on data, and or theorizing.

I have written and talked about Statistical Significance often. And to many, I might be preaching to a choir. But when something is important, there is no harm repeating it. So here goes: This article is yet another attempt at a laypersons guide to Statistical Significance.

Superpowers

“I have a superpower”, suppose I tell you.

“What superpower?” you say doubtfully.

“I can predict the future.”, I say.

“The future? Like what the price of bitcoin will be in a week?”

“Well, not quite.”, I clarify. “I can predict the future in one special case.”

“What special case?”

“Well, if you toss a coin, I can predict its outcome before you toss the coin.”

Evidence

“How can I believe you?” you continue, sceptically.

“Well, ask our mutual friend Amali. She tossed a coin twice last Sunday. And I was able to predict it correctly both times.”

“Really?”

“Yes. I said heads and then tails. And that’s what happened. Amali’s coin landed heads and then tails.”

Statistical Significance

The probability of guessing the outcome of a coin toss randomly, and getting it right, is 50% (assuming the coin was balanced or fair). The probability of getting it right twice is 50% x 50% or 25%. Hence, the probability that “the evidence for me having a superpower being a result of randomness” is 25%.

The “probability that the evidence was a result of randomness” is the intuitive definition of statistical Significance. For a more formal definition (involving more concepts like “Null Hypotheses”, “Alternative Hypotheses” and “P-Values”), you can read this excellent Wikipedia article on the topic.

A step-by-step guide to using Statistical Significance

  • When someone makes a claim (especially dubious people like Astrologers and Economists) ask for evidence. When you get the evidence, calculate the Statistical Significance
  • If the Statistical Significance is low (say less than 5%), you can accept the claim; the chance of the evidence being a result of randomness is low.
  • If the statistical Significance is high (say more than 5 or 10%), take one of the following steps.
  • Ask for more evidence. If you get more evidence, repeat the steps.
  • If the claimant refuses evidence and says something like, “well, this data is not the only evidence, many other theories supported by more data also support my claim”, ask for this additional evidence and repeat the steps.
  • If the claimant dithers further, then summarily dismiss the claimant and the claims. Both are frauds.
  • Alternatively, make sure that the claimant suffers severe repercussions if their claim turns out to be false. For example, demand that they forfeit their entire consultation fee plus an additional 50% of their claim turns out to be wrong. In other words, make sure they have #SkinInTheGame.
Source: https://www.explainxkcd.com/wiki/index.php/882:_Significant

--

--

--

I am a Computer Scientist and Musician by training. A writer with interests in Philosophy, Economics, Technology, Politics, Business, the Arts and Fiction.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What Is A Data Scientist? From a Product Manager Point of View

Eva’s first days of Snowflake+Tableau visualizations

HyperLogLog: A Simple but Powerful Algorithm for Data Scientists

The 4 Questions of Statistics

Data Science Project Frameworks for Managers

How to be a Data Analyst — Data Viz with Google Data Studio

Weekly coding progress (Week 12)

Decoding Jacinda Ardern — (Analysis using Python)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nuwan I. Senaratna

Nuwan I. Senaratna

I am a Computer Scientist and Musician by training. A writer with interests in Philosophy, Economics, Technology, Politics, Business, the Arts and Fiction.

More from Medium

Our Brain’s Inference Skills, on Fear™

Hypothesis Testing: What It’s All About+Examples

Conscious learning

Real Statistics: A Radical Approach