To find statistical evidence for conversion rate differences, follow these steps:
- Define Groups: Establish an experimental group (receiving the change) and a control group (no change).
- Calculate Conversion Rates: Determine the proportion of conversions for each group. This is typically done by summing conversion events (e.g., 'True' = 1, 'False' = 0) and dividing by the total number of users in each group.
- Perform Hypothesis Test: Calculate a test statistic (e.g., a z-score or t-score) that measures the difference between the two conversion rates, standardized by their variability.
- Determine P-value: Obtain the p-value associated with the test statistic. This represents the probability of observing a difference as large as, or larger than, the one found, assuming there is no true difference between the groups (null hypothesis).
- Conclude Significance: Compare the p-value to a predetermined significance level (commonly 0.05). If the p-value is less than 0.05, conclude that the observed difference is statistically significant; otherwise, it is not.