What is statistical significance?
The significance is a statistical term that tells how sure you are that a difference between two tests exists and is not due to a too small test sample size.
For example, if you determine that 60% of your customers prefer blue shoes vs 40% that prefer red shoes based on a total of 10 customers, it has not the same level of reliability that if you obtain the same results based on 10k customers.
To summarize, showing the statistical significance of a test is
How to calculate the statistical significance?
There are many calculators available online. I have chosen two of them because they work in a different way.
- This calculator from Hubspot : it gives you afterwards the reliability percentage of your test based on the data resulting from your test.
- This sample-size calculator by Optimizely that tell you how long you should run your test to obtain results with a statistical significance
What confidence level should be used for the statistical significance?
I often see two confidence percentage values used for the statistical significance: 99% and 95%. Those values are often used in research and online marketers tend to use them also.
But online marketing is not a science. I honestly don’t think that such percentages of confidence level are necessary for every test. It is important to evaluate the risks in order to determine if a high confidence level is needed.
As given in example in this article, If you have a 75% chance to make a million dollars and 25% chance to lose $1,000 then odds are you’re up for the bet. But if you have a 55% chance of making a million and a 45% chance of losing a million then chances are you’ll pass on the opportunity. If the proposed opportunity has a high risk and low reward then you should be more concerned about taking the bet. But if the bet has a low risk for a high reward then maybe it worths taking it.
How to apply this to online marketing? Well, it depends on what your test is based on. If it is to select ad copies, to chose between call to actions on your landing page, or to pick the colour of a button, then having a confidence level around 80% is fine.
When can I ignore statistical significance?
Well, never. If you are professional you should be familiar with statistical significance and evaluate the risk of not measuring it. In some cases, if a clear trend is observed, you could decide to stop the test earlier.
The result of the test will become estimated based on your available data. You have to evaluate if the variation between the two tests is somehow relevant and if the precision of your data sufficient enough to pick a winner.
This can be used for example when your current landing page conversion rate is really low: if you measure that your new landing pages tested have already better performances after a small time frame, you will accept to go-live with your new landing pages and this, even before reaching the statistical significance.