🚦Test design
A/B testing is a powerful method for optimizing your store performance by comparing two or more variations of a page, product, price, or more. To ensure the success of your A/B tests with ABConvert, follow these best practices when designing a test.
1. Define a clear hypothesis
Before starting a test, it's essential to have a clear understanding of what you want to achieve. Whether you're testing price sensitivity, content engagement, or shipping options, your goals should be specific and measurable.
Start with a specific question or problem you want to address.
For example:
"Will increasing the free shipping threshold lead to a higher average order value?"
"Will changing the product images on the product detail page increase add-to-cart rates?"
"Will redirecting visitors from a specific ad to a dedicated landing page improve conversion rates?"
Formulate a testable hypothesis.
This should be a clear statement about what you expect to happen, i.e. your objective. For example:
"Increasing the free shipping threshold from $50 to $75 will result in a higher average order value."
"Using lifestyle images instead of product-only images on the product detail page will increase add-to-cart rates."
"Visitors coming from Facebook ads who are redirected to a landing page tailored to that ad campaign will have higher conversion rates than those sent to the general product page."
Clearly defining your hypothesis will help you design better experiments and determine what metrics to focus on during analysis.
2. Choose the right test type
Price test: Use this to experiment with different product prices. You can test individual products or run store-wide price adjustments.
Shipping test: This allows you to test variations in shipping rates and free shipping thresholds.
URL redirect test: This enables you to test the effectiveness of sending traffic to different landing pages or PDPs.
Template test: Compare page layouts to see effects on key metrics.
Theme test: Use this to compare the performance of different Shopify themes.
Visual editor test: Edit text, replace images, and modify HTML elements directly on your live pages without code.
Checkout test: rename, hide or reorder delivery methods and payment methods; add custom checkout block in checkout page
Select the test type that aligns with your goals. For example, if you want to understand how price affects conversion rates, use the Price test.
3. Set up proper traffic splits
Ensure that you allocate traffic appropriately between test groups. ABConvert allows you to set traffic splits for up to five groups. A common practice is to start with a 50/50 split for two variations (control vs. variant) and adjust based on the number of variations being tested.
4. Use traffic allocation, audience filters or targeting options if needed
ABConvert offers powerful audience filters options. These allow you to run tests on specific segments of your audience based on Country, Device (Desktop/Mobile), Visitor type (New/Returning), UTM parameters, Cookie values, or custom JavaScript conditions.
This ensures that your tests are relevant to specific user groups and helps in gathering more actionable insights.
ABConvert also introduces traffic allocation % and force assign targeting, allowing you to gradually roll out a test to a small percentage of your audience, or force specific groups into a variant.
5. Check your test with preview mode
Always verify your tests before making them live to real customers. Our Preview mode makes this easy:
Tests start in a Draft state.
Move them to Preview state to test them out. This writes to your theme but only visible via preview link, ensuring your live customers are unaffected while you testing around on your real website.
Use the floating widget on your store to toggle between test groups.
Once verified, set the test from Preview to Active.
6. Monitor key metrics
Once your test is live, monitor key performance metrics using our Analytics Dashboard. Some key metrics you can focus on:
A Probability to Win calculation to easily see which variant is pulling ahead.
Multi-dimension breakdowns allowing you to slice your results by country, device, UTM, and more.
Configurable columns so you can focus on the metrics that matter most.
Profit calculating based on your COGS setting in ABConvert setting page.
7. Ensure enough sample
To make informed decisions based on your A/B test results, ensure that your test has enough sample. ABConvert recommends running tests until you have at least 10,000 visitors and 200 orders in total. This ensures that your results are not due to random chance but reflect real user behavior.
The Analytics dashboard provides a quick and easy way to see whether your test results are statistically significant, including the new Probability to Win chart.
Give the test enough time to collect meaningful data. Avoid making changes to the test or the website during the testing period, as this can skew results.
8. Analyze results and draw conclusions
After completing an A/B test, refer back to your hypothesis and thoroughly analyze the results using the Analytics dashboard:
Compare your target metric such as: conversion rates, across different variations.
Look at secondary metrics like average order value or profit per visitor (PRV).
Use multi-dimension breakdowns to see if a variant performed better on mobile vs desktop, or in specific countries.
Use insights from the test to decide which version to implement to your store, or inform future optimization efforts.
By following these best practices when designing A/B tests with ABConvert, you'll be able to gather actionable insights that can significantly improve your store's performance across pricing strategies, content engagement, shipping options, and more.