Difference in Proportions Hypothesis Test Calculator

Conducting a Hypothesis Test for the Difference in Proportions

When two populations rates are related, you can compare them by analyzing the difference between their proportions.

A hypothesis test for the difference in sample proportions can help you make inferences about the relationships between two population proportions.

Note: Difference in proportions hypothesis tests are commonly used in “A/B Tests” in which a researcher compares one rate to another. For example, a digital marketer might use an A/B test to compare a conversion rate from one web advertisement to another version of the same advertisement.

Testing for a Difference in Proportions

For the results of a hypothesis test to be valid, you should follow these steps:

1. Check Your Conditions
2. State Your Hypothesis
3. Determine Your Analysis Plan
4. Analyze Your Sample
5. Interpret Your Results

Check Your Conditions

To use the testing procedure described below, you should check the following conditions:

• Independent Samples – Your samples should be independent of each other.
• Binary Outcomes - When conducting a hypothesis test for the difference in two proportion, each sample point from each sample should consist of only one of two outcomes. We often label one outcome a “success” and the other a “failure,” but it does not matter which of the two outcomes gets which label.
• Success-Failure Rate - Each sample size should be large enough that you see at least 10 “success” and 10 “failures” in each sample. For example, if one of your sample proportions has a 20% or 0.2 “success” rate, then you would need to check that the sample size is at least 50 [20 = 50 * 20%] to meet this condition. This condition helps ensure that the sampling distributions from which you collect your samples reasonably follow the Normal Distribution.
• Simple Random Sampling - You should collect your samples with simple random sampling. This type of sampling requires that every occurrence of a category or event in a population has an equal chance of being selected when taking a sample.
• Sample-to-Population Ratio - For each sample, the population should be much larger than the sample you collect. As a rule-of-thumb, a sample size should represent no more than 5% of its population.

State Your Hypothesis

You must state a null hypothesis and an alternative hypothesis to conduct a hypothesis test.

The null hypothesis is a skeptical claim that you would like to test.

The alternative hypothesis represents the alternative claim to the null hypothesis.

Your null hypothesis and alternative hypothesis should be stated in one of three mutually exclusive ways listed in the table below.

D is the hypothesized difference between the populations' proportions that you would like to test.

Determine Your Analysis Plan

Before conducting a hypothesis test, you must determine a reasonable significance level, α, or the probability of rejecting the null hypothesis assuming it is true. The lower your significance level, the more confident you can be of the conclusion of your hypothesis test. Common significance levels are 10%, 5%, and 1%.

To evaluate your hypothesis test at the significance level that you set, consider if you are conducting a one or two tail test:

• Two-tail tests divide the rejection region, or critical region, evenly above and below the null distribution, i.e. to the tails of the null sampling distribution. For example, in a two-tail test with a 5% significance level, your rejection region would be the upper and lower 2.5% of the null distribution. An alternative hypothesis of P₁ - P₂ ≠ D requires a two-tail test.
• One-tail tests place the rejection region entirely on one side of the distribution i.e. to the right or left tail of the null distribution. For example, in a one-tail test evaluating if actual population proportion difference D is above the null distribution with a 5% significance level, your rejection region would be the upper 5% of the null distribution. P₁ - P₂ < D and P₁ - P₂ > D alternative hypotheses require one-tail tests.

The graphical results section of the calculator above shades rejection regions blue.

Analyze Your Sample

After checking your conditions, stating your hypothesis, determining your significance level, and collect your sample, you are ready to analyze your hypothesis.

Sample proportions follow the Normal Distribution with the following parameters (i.e. numbers that define the distribution):

• The Difference in the Population Proportions, D - The true difference in the proportions is unknown, but we use the hypothesized difference in the proportions, D, from the null hypothesis in the calculations.
• The Standard Error, SE - The standard error of the difference in the sample proportions can be computed as follows:
  
  SE = ((p₁ x (1 – p₁))/ n₁ + (p₁ x (1 – p₁))/ n₂)^(1/2),
  
  with n being the sample size. It defines how differences in sample proportions are expected to vary around the null difference in proportions sampling distribution given the sample sizes and under the assumption that the null hypothesis is true.

In a difference in proportions hypothesis test, we calculate the probability that we would observe the difference in sample proportions (p₁ - p₂), assuming the null hypothesis is true, also known as the p-value. If the p-value is less than the significance level, then we can reject the null hypothesis.

You can determine a precise p-value using the calculator above, but we can find an estimate of the p-value manually by calculating the z-score as follows: z = (p₁ - p₂ - D) / SE

The z-score is a test statistic that tells us how far our observation is from the difference in proportions given by the null hypothesis under the null distribution. Using any z-score table, we can look up the probability of observing the results under the null distribution. You will need to look up the z-score for the type of test you are conducting, i.e. one or two tail. A hypothesis test for the difference in two proportions is sometimes known as a two proportion z-test because of the use of a z-score in analyzing results.

Interpret Your Results

The conclusion of a hypothesis test for the difference in proportions is always either:

1. Reject the null hypothesis
2. Do not reject the null hypothesis

If you reject the null hypothesis, you cannot say that your sample difference in proportions is the true difference between the populations. If you do not reject the null hypothesis, you cannot say that the null hypothesis is true.

A hypothesis test is simply a way to look at evidence and conclude if it provides sufficient evidence to reject the null hypothesis.

Example: A/B Test (Hypothesis Test for the Difference in Two Proportions)

Let’s say you are in charge of email marketing for a clothing brand. Your goal is to sell clothes online, and to sell clothes online, you have to get your email recipients to open your emails.

As part of a new email campaign, you have written two versions of an email subject line: an A version and a B version. But you do not know which one will be more effective.

So, you decide to run an “A/B Test” of your subject lines using a difference in proportions hypothesis test to analyze your results. Your goal is to see if either subject line will have a higher "open" rate.

Your email database consists of 100,000 contacts, and you decide to run the test on 5,000 of them with 50% of the sample group receiving subject line A and 50% receiving subject line B. Let’s go through the steps you would take to run the test.

1. Check the conditions - Your test consists of binary outcomes (i.e. open and no open), your sample sizes are large enough to meet the success-failure condition but not too large to violate the sample-to-population ratio condition, and you collect your samples using simple random sampling.
2. State Your Hypothesis - Your null hypothesis is that the email subject lines are the same (i.e. P₁ - P_2> = 0) and your alternative hypothesis is that they are not the same (i.e. P₁ - P_2> ≠ 0).
3. Determine Your Analysis Plan - You believe that a 5% significance level is reasonable. As your test is two-tail test, you will evaluate if the difference in open rates between the samples would occur at the upper or lower 2.5% [2.5% = 5%/2] of the null distribution.
4. Analyze Your Sample - After collecting your samples (which you do after steps 1-3), you find that subject line A had a sample open rate, p₁, of 20%. Subject line B has a sample open rate, p₂, of 17%. Using the calculator above, you find that a difference in sample proportions of 3% [3% = 20% - 17%] would results in a z-score of 2.73 under the null distribution, which translates to a p-value of 0.63%.
5. Interpret Your Results - Since your p-value of 0.63% is less than the significance level of 5%, you have sufficient evidence to reject the null hypothesis.

In this example, you found that you can reject your original claim that the subject lines have the same performance. The test does not guarantee that your subject line A has a higher open rate than subject line B, but it does give you strong reason to favor subject line A.