How To Calculate Statistical Significance (And Its Importance)

If you're trying to determine the effectiveness of something, consider calculating statistical significance. Though it's known for being taught in statistics coursework, it can be used for a variety of different industries including business.

In this article, we define statistical significance, its importance and how to calculate it by hand.

Related: Analytical Skills: Definitions and Examples

What is statistical significance?

Statistical significance refers to the likelihood that a relationship between two or more variables is not caused by random chance. In essence, it's a way of proving the reliability of a certain statistic. Its two main components are sample size and effect size. In the use of statistical hypothesis testing, a data set's result can be deemed statistically significant if you have reached a certain level of confidence in the result. In statistical hypothesis testing, this means the hypothesis is unlikely to have occurred given the null hypothesis. According to a null hypothesis, there is no relationship between the variables in question.

Related: Decision-Making Methods for the Workplace

Why is statistical significance important?

In regards to business, statistical significance is important because it helps you know that the changes you've implemented can be positively attributed to various metrics. For example, if you've recently implemented a new application to help your office work more efficiently, statistical significance provides you with the confidence in knowing that it made a positive impact on your company's overall workflow. That is, the app's impact was statistically significant and provided value. If it turns out the app wasn't statistically significant, this means your business dollars and the app are at risk. Make sure to measure the statistical significance for every result to get a more comprehensive calculation and result.

To help you make business decisions in the future, consider using business relevance along with statistical significance. This will ensure your decisions are not based on statistical significance alone.

Related: How Analyzing Data Can Improve Decision-Making

How to calculate statistical significance

Calculating the statistical significance is rather extensive if you calculate it by hand and this is why it's typically calculated using a calculator. When you calculate it by hand, however, it will help you more fully understand the concept. Here are the steps for calculating statistical significance:

1. Create a null hypothesis.

2. Create an alternative hypothesis.

3. Determine the significance level.

4. Decide on the type of test you'll use.

5. Perform a power analysis to find out your sample size.

6. Calculate the standard deviation.

7. Use the standard error formula.

8. Determine the t-score.

9. Find the degrees of freedom.

10. Use a t-table.

1. Create a null hypothesis

The first step in calculating statistical significance is to determine your null hypothesis. Your null hypothesis should state that there is no significant difference between the sets of data you're using. Keep in mind that you don't need to believe the null hypothesis.

2. Create an alternative hypothesis

Next, create an alternative hypothesis. Typically, your alternative hypothesis is the opposite of your null hypothesis since it'll state that there is, in fact, a statistically significant relationship between your data sets.

3. Determine the significance level

Your next step involves determining the significance level or rather, the alpha. This refers to the likelihood of rejecting the null hypothesis even when it's true. A common alpha is 0.05 or five percent.

4. Decide on the type of test you'll use

Next, you'll need to determine if you'll use a one-tailed test or a two-tailed test. Whereas the critical area of distribution is one-sided in a one-tailed test, it's two-sided in a two-tailed test. In other words, one-tailed tests analyze the relationship between two variables in one direction and two-tailed tests analyze the relationship between two variables in two directions. If the sample you're using lands within the one-sided critical area, the alternative hypothesis is considered true.

5. Perform a power analysis to find out your sample size

You'll then need to do a power analysis to determine your sample size. A power analysis involves the effect size, sample size, significance level and statistical power. For this step, consider using a calculator. This type of analysis allows you to see the sample size you'll need to determine the effect of a given test within a degree of confidence. In other words, it'll let you know what sample size is suitable to determine statistical significance. For example, if your sample size ends up being too small, it won't give you an accurate result.

6. Calculate the standard deviation

Next, you'll need to calculate the standard deviation. To this, you'll use the following formula:

standard deviation = √((∑|x−μ|^ 2) / (N-1))

where:

∑ = the sum of the data

x = individual data

μ = the data's mean for each group

N = the total sample

Performing this calculation will let you know how to spread out your measurements are about the mean or expected value. If you have more than one sample group, you'll also need to determine the variance between the sample groups.

7. Use the standard error formula

Next, you'll need to use the standard error formula. For our purposes, let's say you have two standard deviations for your two groups. The standard error formula is as follows:

standard error = √((s1/N1) + (s2/N2))

where:

s1 = the standard deviation of your first group

N1 = group one's sample size

s2 = the standard deviation of your second group

N2 = group two's sample size

8. Determine t-score

For the next step, you'll need to find the t-score. The equation for this is as follows:

t = ((µ1–µ2) / (sd))

where:

t = the t-score

µ1 = group one's average

µ2 = group two's average

sd = standard error

9. Find the degrees of freedom

Next, you'll need to determine the degrees of freedom. The formula for this is as follows:

degrees of freedom = (s1 + s2) - 2

where:

s1 = samples of group 1

s2 = samples of group 2

10. Use a t-table

Finally, you'll calculate the statistical significance using a t-table. Start by looking at the left side of your degrees of freedom and find your variance. Then, go upward to see the p-values. Compare the p-value to the significance level or rather, the alpha. Remember that a p-value less than 0.05 is considered statistically significant.

Examples of statistical significance

Consider the following examples of statistical significance:

Example 1

Let's say you want to attract more customers to your business, so you decide to run an ad campaign. In doing so, you consider how many advertisements should be made in print and how many should be made digitally. You rely on past ad campaigns to forecast how many you'll need of each. If you determine that your p-value is above 0.05 or 5%, you'd end up with a result that is not statistically significant. This means that there's a greater than 5% chance that the relationship between the two types of ads was left up to chance. Therefore, this result would indicate that it's not reasonable to use the previous ad campaign as a guide.

Example 2

Next, let's say you've created a new company website design with the hopes of attracting more customers. You've determined that there was a statistically significant increase in the number of customers since the new website's implementation. Your calculation of the statistical significance resulted in a p-value of 3% or 0.03. Given that it's below 0.05, this is a statistically significant result meaning that the increase in customers was not left to random chance.