ABConvert includes tools to help determine statistical significance in your test results. This involves hypothesis testing to identify if differences between test groups are statistically significant. For reliable insights, ensure your tests have sufficient data—typically at least 10,000 views and 200 orders.
Understanding Hypothesis Testing
One-Sided Hypothesis: Tests for an effect in one specific direction (e.g., whether a new price increases sales). Use this when you have a specific prediction about the direction of the effect.
Two-Sided Hypothesis: Allows for an effect in either direction (e.g., whether a new price changes sales, either up or down). Choose this when you are unsure about the direction of the effect or want to detect any change.
Interpreting Results
While ABConvert provides a "Conclusion" in the Statistical significance section to help you know if your test is statistically significant at a glance, it might be useful to understand each metric used to get that conclusion:
Lift: The percentage change in a certain metric between test groups, such as the conversion rates between original versus test group. Lift helps determine the effectiveness of a change or variant in your A/B test.
Confidence: Represents the probability that the results of an experiment are not due to random chance. It indicates how certain you can be about your test results. Common confidence levels are 90%, 95%, and 99%.
P-value: Indicator of statistical reliability. A low p-value (typically <0.05) suggests significant differences between groups.
ABConvert provides a robust set of tools to help you determine the statistical significance of your A/B test results, ensuring that any observed differences between test groups are not due to random chance.