All Collections
Analytics
How ABConvert analytics works
How ABConvert analytics works

This article explain how ABConvert analytics works and how you should interpret it

Jeffrey at ABConvert avatar
Written by Jeffrey at ABConvert
Updated over a week ago

Reading analytics is a core part of performing A/B testing. In ABConvert, we have a dedicated analytics dashboard to report data to our user in real-time.

In this article, we dive into how our analytics works and how to interpret your data

Overview

In this article, we will explain three major parts of our analytics:

  1. Dashboard components

  2. Data collecting process & calculation of metrics

  3. Interpreting data

Dashboard components

There are three components in the analytics dashboard:

  1. Summary section - report the summary of stats

  2. Statistical tool - examine the statistical significance for your data

  3. Breakdown table - display the granular level of your data

Summary section

In this section, we report summary statistics of your test.

In the first card, user can see the visitors amount and the trend with graph. This allows our user to monitor traffic.

In the the second card, we report data in the form of conversion funnel. Users can see at which point their customers drop off and observe the change of rate for different variant.

In the third card, user can see the average metrics of revenue and profit. By looking at these metrics, user can know which variant is better for increasing profit or AOV.

Statistical tools

In order to make fair decision based on A/B test result, we provide our user a statistical analysis tool to examine whether the result is significant or not.

In brief, in each A/B test, we want to know whether version A is better than version B. But we don't know it just be a result of luck, therefore we need to do a statistical hypothesis test.

The hypothesis will be:

If version A is equal to version B.

Ideally, we want version A is not equal to version B, either better or worse, so we can make decision based on this result.

Hence, we provide a statistical significance tool for our user to check the statistical result.

In the Statistical Tool section, there are several parts:

  1. Hypothesis settings

  2. Result table

Hypothesis settings

In the part, user can set up the way the do hypothesis.

By selecting one-sided or two-sided hypothesis, we are actually saying this:

We want to test if version A is equal to version B.

  1. If failed, version A is worse then version B (one-sided)

  2. If failed, version A is not equal to version B (two-sided)

Since the hypothesis result is actually a Yes or No response, setting the hypothesis to be one-sided or two-sided might be depending on user's objective.

As for setting confidence level, we are essentially deciding how difficult is it to reject the hypothesis.

If the choose the lower confidence level (e.g. 90%), chance are that we think the version A and version B are different but in reality it's not.

The higher the level, the larger difference we need in a test to make decision.

Result Table

In the part, we can be able to see result generated from hypothesis test.

There are the following metrics:

  1. Lift

  2. Confidence

  3. P-value

  4. Conclusion

Lift

This metric tell you how many percentage the version B increase on your objective rate compared to version A. Ideally, you want to see a positive left.

Confidence

This metric tell you how confident you are to the result. If your confidence is higher than confidence level, than the result is result trustworthy and the decision can be made from the conclusion.

P-value

This metric is generate from the T test of the hypothesis. The lower the value, the more significant the result is. Read more here.

Conclusion

We come up with the conclusion regarding an objective based on the metric.

The underlying logic is that when we observe positive Lift and the result is significant (on confidence level), we conclude it's significantly better.

When the lift is negative, and the result is significant, we conclude it's significantly worse.

Otherwise, it's either not significantly better or not significantly worse.

Breakdown Table

In this section, you can be able to see the raw data on different granularity.

There are three level of granularity:

  1. Test group level

  2. Product level

  3. Variant level

Users can change the level to look at different data to understand the test result.

They can also select the column like any dashboard if they don't need some information.

We also let our user to add costs to their product if it's not on the Shopify backend. Once we have the product costs, we can calculate the profit.

Data collecting process & calculation of metrics

There are two methods of data collecting process in our analytics:

  1. Visitor based data

  2. Product view based data

These two methods are design to support different types of store since each store has different way to attract traffic to different pages.

Visitor based data

In this method, we focusing on visitor. Each visitor is defined as a visit in 30 minutes. For example, if a customer come back to your website several times in 30 minutes, it's counted as one visitor.

Each visitor can only contribute to at most one event for a event type. These are the event data we collect:

  1. Add to cart

  2. Checkout

  3. Order

So in this method, one visitor can only contribute to one add to cart, one checkout, and one order, even though they may add to cart several times and order more than one product.

Therefore, the conversion rate of the following metric will be calculated by:

  1. Add to cart rate: # number of add to cart / # number of visitor

  2. Checkout rate: # number of checkout/ # number of visitor

  3. Order rate: # number of checkout / # number of visitor

If a store have more traffic to home page and most people add to cart on home page, then this method is more suitable, since we can observe the visitor behaviour regardless of pages type.

Also for users who tend to add more products to a test, this metric might be similar to their own Shopify analytics.

Product view based data

In this method, we focus on product view. Each product is defined as each time a user enter a product page at the same browser. If they open a browser window, can visit a product page several time, then it will count as one product view.

Each product view can only contribute at most one event for a product. These are the event data we collect:

  1. Add to cart

  2. Checkout

  3. Order

In this method, one visitor can view different products and add many products to cart. Then, it will be counted as many product views and many add to carts.

Therefore, the conversion rate of the following metric will be calculated by:

  1. Add to cart rate: # number of add to cart of product / # number of product views

  2. Checkout rate: # number of checkout of product / # number of product views

  3. Order rate: # number of order of product / # number of product views

This method is suitable for users who want to observe product level changes of different version and those who mainly direct to product page.

Interpreting data

After a test, we want to make decision based on the data.

In this section, we focus on how to interpret data for price testing.

But the logic can be applied to all test.

There are two step to interpret data:

Step 1

  1. If I lower the price, I want to see the conversion rate significantly increase.

  2. If I increase the price, I don't want to see the conversion rate significantly decrease.

Step 2

In either way, we want to see profit increase. .

Ideally, for lowering price, we want to see conversion rate significantly increase plus profit per view increase.

For increasing price, we want to see conversation rate not significantly decrease plus profit per view increase.

If the conversion rate hasn't changed significant, we will have to check if profit per view change significantly.

If not, then we stick with the original one.

Conclusion

In this article, we walk you through how ABConvert's analytics works and offer some guidance on making decision.

If you have any questions, feel free to email [email protected]. I hope you enjoy this article.

Did this answer your question?