Statistical Significance
Yet another layperson’s guide
Imagine if everyone in the medical profession had no idea what temperature and its consequences meant. Ignorance of such a fundamental concept would result in confussion and carnage. Statistical Significance is like temperature, but significantly more significant and consequential. Not just consequential to the medical profession, but to anyone or any profession doing experiments, making claims on data, and or theorizing.
I have written and talked about Statistical Significance often. And to many, I might be preaching to a choir. But when something is important, there is no harm repeating it. So here goes: This article is yet another attempt at a laypersons guide to Statistical Significance.
Superpowers
“I have a superpower”, suppose I tell you.
“What superpower?” you say doubtfully.
“I can predict the future.”, I say.
“The future? Like what the price of bitcoin will be in a week?”
“Well, not quite.”, I clarify. “I can predict the future in one special case.”
“What special case?”
“Well, if you toss a coin, I can predict its outcome before you toss the coin.”
Evidence
“How can I believe you?” you continue, sceptically.
“Well, ask our mutual friend Amali. She tossed a coin twice last Sunday. And I was able to predict it correctly both times.”
“Really?”
“Yes. I said heads and then tails. And that’s what happened. Amali’s coin landed heads and then tails.”
Statistical Significance
The probability of guessing the outcome of a coin toss randomly, and getting it right, is 50% (assuming the coin was balanced or fair). The probability of getting it right twice is 50% x 50% or 25%. Hence, the probability that “the evidence for me having a superpower being a result of randomness” is 25%.
The “probability that the evidence was a result of randomness” is the intuitive definition of statistical Significance. For a more formal definition (involving more concepts like “Null Hypotheses”, “Alternative Hypotheses” and “P-Values”), you can read this excellent Wikipedia article on the topic.
A step-by-step guide to using Statistical Significance
- When someone makes a claim (especially dubious people like Astrologers and Economists) ask for evidence. When you get the evidence, calculate the Statistical Significance
- If the Statistical Significance is low (say less than 5%), you can accept the claim; the chance of the evidence being a result of randomness is low.
- If the statistical Significance is high (say more than 5 or 10%), take one of the following steps.
- Ask for more evidence. If you get more evidence, repeat the steps.
- If the claimant refuses evidence and says something like, “well, this data is not the only evidence, many other theories supported by more data also support my claim”, ask for this additional evidence and repeat the steps.
- If the claimant dithers further, then summarily dismiss the claimant and the claims. Both are frauds.
- Alternatively, make sure that the claimant suffers severe repercussions if their claim turns out to be false. For example, demand that they forfeit their entire consultation fee plus an additional 50% of their claim turns out to be wrong. In other words, make sure they have #SkinInTheGame.
