Reading analytics is a core part of performing A/B testing. In ABConvert, we have a dedicated analytics dashboard to report data to our user in real-time.

In this article, we dive into how our analytics works and how to interpret your data

In this article, we will explain three major parts of our analytics:

There are three components in the analytics dashboard:

In this section, we report summary statistics of your test.

In the first card, user can see the visitors amount and the trend with graph. This allows our user to monitor traffic.

In the the second card, we report data in the form of conversion funnel. Users can see at which point their customers drop off and observe the change of rate for different variant.

In the third card, user can see the average metrics of revenue and profit. By looking at these metrics, user can know which variant is better for increasing profit or AOV.

In order to make fair decision based on A/B test result, we provide our user a statistical analysis tool to examine whether the result is significant or not.

In brief, in each A/B test, we want to know whether version A is better than version B. But we don't know it just be a result of luck, therefore we need to do a statistical hypothesis test.

The hypothesis will be:

Ideally, we want version A is not equal to version B, either better or worse, so we can make decision based on this result.

Hence, we provide a statistical significance tool for our user to check the statistical result.

In the Statistical Tool section, there are several parts:

In the part, user can set up the way the do hypothesis.

By selecting one-sided or two-sided hypothesis, we are actually saying this:

We want to test if version A is equal to version B.

Since the hypothesis result is actually a Yes or No response, setting the hypothesis to be one-sided or two-sided might be depending on user's objective.

As for setting confidence level, we are essentially deciding how difficult is it to reject the hypothesis.

If the choose the lower confidence level (e.g. 90%), chance are that we think the version A and version B are different but in reality it's not.

The higher the level, the larger difference we need in a test to make decision.

In the part, we can be able to see result generated from hypothesis test.

There are the following metrics:

This metric tell you how many percentage the version B increase on your objective rate compared to version A. Ideally, you want to see a positive left.

This metric tell you how confident you are to the result. If your confidence is higher than confidence level, than the result is result trustworthy and the decision can be made from the conclusion.

This metric is generate from the T test of the hypothesis. The lower the value, the more significant the result is. Read more here.

We come up with the conclusion regarding an objective based on the metric.

The underlying logic is that when we observe positive Lift and the result is significant (on confidence level), we conclude it's significantly better.

When the lift is negative, and the result is significant, we conclude it's significantly worse.

Otherwise, it's either not significantly better or not significantly worse.

In this section, you can be able to see the raw data on different granularity.

There are three level of granularity:

Users can change the level to look at different data to understand the test result.

They can also select the column like any dashboard if they don't need some information.

We also let our user to add costs to their product if it's not on the Shopify backend. Once we have the product costs, we can calculate the profit.

There are two methods of data collecting process in our analytics:

These two methods are design to support different types of store since each store has different way to attract traffic to different pages.

In this method, we focusing on visitor. Each visitor is defined as a visit in 30 minutes. For example, if a customer come back to your website several times in 30 minutes, it's counted as one visitor.

Each visitor can only contribute to at most one event for a event type. These are the event data we collect:

So in this method, one visitor can only contribute to one add to cart, one checkout, and one order, even though they may add to cart several times and order more than one product.

Therefore, the conversion rate of the following metric will be calculated by:

If a store have more traffic to home page and most people add to cart on home page, then this method is more suitable, since we can observe the visitor behaviour regardless of pages type.

Also for users who tend to add more products to a test, this metric might be similar to their own Shopify analytics.

In this method, we focus on product view. Each product is defined as each time a user enter a product page at the same browser. If they open a browser window, can visit a product page several time, then it will count as one product view.

Each product view can only contribute at most one event for a product. These are the event data we collect:

In this method, one visitor can view different products and add many products to cart. Then, it will be counted as many product views and many add to carts.

Therefore, the conversion rate of the following metric will be calculated by:

This method is suitable for users who want to observe product level changes of different version and those who mainly direct to product page.

After a test, we want to make decision based on the data.

In this section, we focus on how to interpret data for price testing.

But the logic can be applied to all test.

There are two step to interpret data:

In either way, we want to see profit increase. .

Ideally, for lowering price, we want to see conversion rate significantly increase plus profit per view increase.

For increasing price, we want to see conversation rate not significantly decrease plus profit per view increase.

If the conversion rate hasn't changed significant, we will have to check if profit per view change significantly.

If not, then we stick with the original one.

In this article, we walk you through how ABConvert's analytics works and offer some guidance on making decision.

If you have any questions, feel free to email [email protected]. I hope you enjoy this article.