Ad Split Testing Statistical Analysis Tool

What is the ad testing tool?
Why would I want to use this testing tool?
What does this ad testing tool compare?
If I don't have conversion tracking setup on my AdWords ads, can I still use this ad testing tool to test my CTR?
What if I am testing more than two ads? Can I use this ad testing tool to test all combinations?

What is Ad Variations?
Is there a limit on the length of the ad variation name?
What information do I need to run my test?
Where can I find my Clicks, Impressions and Conversion Rate?
Why do I need to choose a confidence level?
What if I want a different confidence level?
How do I get the results?

I've run the test, but all I see is a sentence telling me if my ads are different. How do I see the detailed results?
What is Additional Testing?
What is the minimum number of clicks, impression and conversions needed to run this test?
What's the difference between the two percentages in absolute difference?
How do you compute the absolute difference?
How do you compute relative improvement?
Is the confidence interval on the difference based on the absolute difference or relative improvement?
What is the confidence interval?
What does the confidence interval tell me if I find a statistical difference?
What does the confidence interval tell me if I don't find a statistical difference?
What is a p-value?
What determines if my difference is significant?
How do I select my alpha value?

How does this ad tester determine the number of impressions or clicks needed to find a statistical significance?
I've noticed that when the difference between my ads is small I need more impressions or clicks than when they are large. Can you explain this?
Why do you show the upper confidence interval if it doesn't matter in terms of this test?
What if I want to use this confidence interval as a two sided interval?
Why did you choose one sided confidence intervals?
What type of test you are using to conduct this analysis?
What delta, power and hypothesized proportions are used to determine the number of impressions or clicks needed?
Why did you use such a large power?

It is a binomial distribution statistical calculator used to compare the results of two ads. Not only will it tell you if one ad performs better than another. It will also tell you confidence intervals, p-values, and the sample size needed to finish the test.

If you are doing a/b split testing on your AdWords ads and you want detailed statistical analysis, this is the tool for you.

This tool was designed to test two ad variations. It will test the results of click through rate (CTR) and optionally Conversion Rate (CR).

Yes, you can use this ad testing tool. You are only required to enter your clicks and impressions.

This tool is designed to test only two ads. It is not recommended to use this tool to test all combinations. With every additional combination tested, you increase the chance of making incorrect conclusions. If you ran more than two ads in your test, you can test the ad with the lowest rate to the ad with the highest rate. If they are not different, than neither are the ads in the middle. If they are different and you want to do further analysis, than you should use advanced statistical software tools designed to test more than two proportions or consult with a statistician.

This is the name given to each ad in your test. It will be used in the following results. You can use Ad1 and Ad2. However, the recommendation is to use a better descriptor. For example, if one ad listed the feature "Free Shipping" and the other listed "24/7 Customer Service" than you could use the words "Free Shipping" and "Customer Service" instead of Ad 1 and Ad 2. If the free shipping ad is better than the customer service ad than the results would state, "The Free Shipping ad is better than the Customer Service ad." Isn't that either than having to remember which ad was Ad 1?

Yes there is a limit but it allows enough room for you to be descriptive enough to tell the difference between the two ads in the results.

All you need is the number of clicks, number of impressions and conversion rate.

From the campaign management tab choose Campaign Summary - Campaign Name - Ad Group Name. Next select the Ad Variations Tab. From here you can find all the information you need to run your test.

The confidence level is used when computing the confidence intervals and determining if a significant difference has been found.

The available confidence levels are 98%, 95%, 90% and 85%.

Once you have entered the clicks, impressions, conversion rate and selected your confidence level, you are ready to run the test. This can be done by selecting the orange button titled "Test these ads" in Step 3.

Under the sentence you will see grey text that states "Show me details..." Select the text to see the detailed results.

This lists the amount of additional data that is needed to find a difference in the ads if one hasn't already been found. If the ads are statistically different based on the confidence level chosen in step 2, it will state that no additional data is needed. Otherwise, it will list the number of impressions needed for CTR or the number of clicks for conversion rate. This number is for each ad not the total for both ads.

You need a minimum of at least 5 for each possible outcome. An outcome is a click, non click, conversion, and non conversion. For example if you are testing conversion rates, you would need a minimum of 20 impressions with 10 getting clicks and a 50% conversion rate. Giving you 5 conversions and 5 non conversions, thus meeting the minimum of 5 in all four outcomes. This is required because of the type of testing being performed. If you don't meet these requirements and run the test anyway, an error message will be displayed explaining that you have not met the assumptions of this testing. This means you do not have 5 data points for all possible outcomes. Getting 5 outcomes does not guarantee a statistical significance.

The first one is the difference between the ads and the second is the relative improvement compared to the ad with the lowest rate.

The absolute difference is the ad with the highest rate minus the ad with the lowest rate.

Relative improvement is the absolute difference divided by the ad with the lowest rate times 100. For example, if you have a CTR of 2% for ad 1 and 4% for ad 2, that would be 100% improvement.

(4% - 2%)/2% * 100 = 100%

The confidence interval on the difference is based on the absolute difference.

Since the data you collected is a sample of the population, the results are just estimates of the population. The confidence interval is a range that is believed to contain the true population rate. If you have a 95% confidence interval, than there is 95% confidence that the interval contains the true rate. Since the true rate is unknown, we don't know if it does contain the true mean. A larger confidence level will produce a larger confidence interval (increasing the chance of containing the true rate).

Each value is a one sided confidence interval on the difference. The lower one sided confidence interval is the most important of the two. If you find a statistical difference based on a 95% confidence level, it tells you there is 95% confidence that the difference between the ads could be as small as the value given. If the lower confidence interval is 2% and the absolute difference is 5%, than you know with 95% confidence that the winning ad difference could really only be 2%. If the lower confidence interval is sufficiently large, than the upper confidence interval doesn't really matter in terms of this test.

If you didn't find a statistical difference, than the lower confidence interval tells you how much worse the ad with the higher rate could be than the ad with the lower rate. If this value is negative, you know that your ads are not statistically different. For example, if you chose a 95% confidence level and the lower confidence interval is -1%, than the true difference between the ad with the highest rate could actually be 1% less than the ad with the lowest rate(i.e. not better).

Since the data is just a sample of the population, there isn't 100% certainty that the ads are really different (Type I error). The p-value is the probability of concluding that the ads have different CTR or conversion rates when they are really equal. A larger confidence level decreases your alpha value thus decreasing your chance of making a type I error but requires a large sample size to find a difference.

A test result has a statistically significant difference if the preselected alpha value is less than the computed p-value.

When you choose your confidence level in Step 2 you are setting your alpha value. Alpha is 1 - (confidence level/100). If you choose a 95% confidence level, alpha is 0.05.

If the p-value is less than the specified alpha value, than the tester will state no additional testing. Otherwise, the tool uses a sample size calculator to determine the amount of additional clicks or impressions needed in order for the ad with the highest rate to be better than the ad with the lowest rate. Please note that additional data collection does not guarantee that a statistical difference will be found, especially if the rates move closer together during the testing.

Yes, the following example will help illustrate the point. If you had 100 impressions for both ads, with one ad getting 99 clicks and the other only getting 1 click, you wouldn't need statistics to tell the ads have different CTR. Now let's say one ad has 10 clicks and the other 11. The second case isn't as obvious. You need more compelling results to say 11% is better than 10% than you do 99% is better than 1%. That is why when the difference between the ads are smaller, you will need a larger samples size.

If someone is interested in the upper limit and not the lower, it will be available to them.

You can use this as a two sided interval but the confidence level will be different. If you chose a 95% confidence level, than the confidence interval given would be 90%. Remember that a 95% confidence level has an alpha value of 5%. So the confidence level for the two sided interval is 1 - 2*alpha value.

Since the test performed is a one-sided test, than the lower one sided confidence interval must be used to be in agreement with the conclusions based on the p-value. The two-sided confidence is 1 - 2*alpha value.

This ad testing tool is using the normal approximation to the binomial distribution. The proper screening is in place to ensure that the assumptions are met.

For the statistician reading this, the calculator is using the normal approximation to the binomial to compute a one sided sample size. The alpha level is determined in step 2 of the calculator. The power is set to 0.5. The estimated proportions and resulting delta used are the current observed rates entered in step 1.

If the observed rates are equal to the hypothesized rates used to compute the sample size, a power of 0.5 is the largest power that will produce a p-value less than or equal to the specified alpha value from step 2. In this calculator, the hypothesized rates are always the observed rate. If after further data collection the rates are the same, a power of 0.5 is needed to ensure the p-value would be less than 0.05. Plus power is guarding against the probability of concluding the two ads are the same when they are really different. Once the p-value drops below the specified alpha value, the power is no longer relevant and focus should be on the range of the confidence interval.