Skip to Content

The author

Steve Baker

Chief Analyst

When most experts talk about testing adverts on Google Adwords, they recommend that you change only one thing in the advert, so that your results aren’t confused by conflicting changes (if one change improves your click through rate, and one makes it worse, then the overall impact could be better or worse, and you’ve no idea what worked at what didn’t). And that’s all ducky. But it assumes that the lines of your advert work independently of each other. What happens if this isn’t the case. Consider the two adverts below: A) ABC Mortgages Free Advice And Valuations Competitive Rates Suppose that you’ve been running A) for a while, and try a new third line œQuick And Friendly Service: B) ABC Mortgages Free Advice And Valuations Quick And Friendly Service You get the following results: A) 10000 impressions, 1000 clicks, click through rate = 10% B) 10000 impressions, 800 clicks, click through rate = 8% So you keep advert A) Then you try it against another advert, with a second line of œYour Local Mortgage Broker: C) ABC Mortgages Your Local Mortgage Broker Competitive Rates You get the following results: A) 10000 impressions, 1000 clicks, click through rate = 10% C) 10000 impressions, 800 clicks, click through rate = 8% Again, you keep advert A). But have you missed something here? We’ve no idea if advert A) is better than an advert with both changes: D) ABC Mortgages Your Local Mortgage Broker Quick And Friendly Service It’s quite possible that the two new lines work well together, as they have a similar tone. This being the case, you could potentially get the following results: A) 10000 impressions, 1000 clicks, click through rate 10% D) 10000 impressions, 1200 clicks, click through rate 12% This is quite possible, but you can’t run this test instead of the others, in case one of the lines was better, and the other worse. And if you run it as well as the others, it feels a bit like you’re wasting time. And if you wanted to try varying all three lines, you’d need to run seven different tests, instead of just three. In principle, the solution is quite simple. There’s an easy way to test your three adverts in the time of two tests, or if you are changing all three lines, eight tests in the time it would take to run four tests: Simultaneously, run all possible combinations of the adverts against each other at once. The above tests ran for a total of 20,000 impressions each, but how many impressions would it take to get the best of the four options? Only 40,000 impressions, because you don’t need to waste 50% of your impressions on the control, only 25%. This is the same amount of time that you’d have spent just testing A against B and A against C. So your results would have looked something like: A) 10000 impressions, 1000 clicks, click through rate 10% B) 10000 impressions, 800 clicks, click through rate 8% C) 10000 impressions, 800 clicks, click through rate 8% D) 10000 impressions, 1200 clicks, click through rate 12% In fact, you can save a bit more time by ditching one of the adverts early on, if it’s clearly not going to be the best. So perhaps testing one change at a time isn’t necessarily the best idea , the stronger the synergy between the lines of your advert, the more likely you are to miss positive changes. However, there is a price to pay for this approach (isn’t there always?). The test is slightly more likely to come up with the wrong answer (it’s slightly less significant), so you’ll probably want to run it a little bit longer to get the same degree of confidence in your results. In summary, if you think there’s likely to be a strong synergy between two lines of your advert, it’s worth testing all four combinations of the two lines at the same time. If there’s no reason to believe that the two new lines work particularly well together, but poorly individually, then there’s no benefit to running the four options at once, and you should run two a:b tests as usual. For example, if you have the advert: Dave’s Confectionary Buy Our Chocolates! Free Next Day Delivery And you want to try a second line of œGet Luxury Chocolates Here!, and/or a third line of œGift Wrapping Available, you have to decide whether these lines are likely to complement each other, but are unremarkable individually. Here, it looks like the lines are pretty much unrelated, with one being an alternative call to action, and the other suggesting a different feature. So two standard (one change) a:b tests would be quicker or more reliable (depending on how long you wait for your results) than running all four possibilities together. Any comments? Disagree? Let me know¦

When most experts talk about testing adverts on Google Adwords, they recommend that you change only one thing in the advert, so that your results aren’t confused by conflicting changes (if one change improves your click through rate, and one makes it worse, then the overall impact could be better or worse, and you’ve no idea what worked at what didn’t). And that’s all ducky. But it assumes that the lines of your advert work independently of each other. What happens if this isn’t the case. Consider the two adverts below: A) ABC Mortgages Free Advice And Valuations Competitive Rates Suppose that you’ve been running A) for a while, and try a new third line œQuick And Friendly Service: B) ABC Mortgages Free Advice And Valuations Quick And Friendly Service You get the following results: A) 10000 impressions, 1000 clicks, click through rate = 10% B) 10000 impressions, 800 clicks, click through rate = 8% So you keep advert A) Then you try it against another advert, with a second line of œYour Local Mortgage Broker: C) ABC Mortgages Your Local Mortgage Broker Competitive Rates You get the following results: A) 10000 impressions, 1000 clicks, click through rate = 10% C) 10000 impressions, 800 clicks, click through rate = 8% Again, you keep advert A). But have you missed something here? We’ve no idea if advert A) is better than an advert with both changes: D) ABC Mortgages Your Local Mortgage Broker Quick And Friendly Service It’s quite possible that the two new lines work well together, as they have a similar tone. This being the case, you could potentially get the following results: A) 10000 impressions, 1000 clicks, click through rate 10% D) 10000 impressions, 1200 clicks, click through rate 12% This is quite possible, but you can’t run this test instead of the others, in case one of the lines was better, and the other worse. And if you run it as well as the others, it feels a bit like you’re wasting time. And if you wanted to try varying all three lines, you’d need to run seven different tests, instead of just three. In principle, the solution is quite simple. There’s an easy way to test your three adverts in the time of two tests, or if you are changing all three lines, eight tests in the time it would take to run four tests: Simultaneously, run all possible combinations of the adverts against each other at once. The above tests ran for a total of 20,000 impressions each, but how many impressions would it take to get the best of the four options? Only 40,000 impressions, because you don’t need to waste 50% of your impressions on the control, only 25%. This is the same amount of time that you’d have spent just testing A against B and A against C. So your results would have looked something like: A) 10000 impressions, 1000 clicks, click through rate 10% B) 10000 impressions, 800 clicks, click through rate 8% C) 10000 impressions, 800 clicks, click through rate 8% D) 10000 impressions, 1200 clicks, click through rate 12% In fact, you can save a bit more time by ditching one of the adverts early on, if it’s clearly not going to be the best. So perhaps testing one change at a time isn’t necessarily the best idea , the stronger the synergy between the lines of your advert, the more likely you are to miss positive changes. However, there is a price to pay for this approach (isn’t there always?). The test is slightly more likely to come up with the wrong answer (it’s slightly less significant), so you’ll probably want to run it a little bit longer to get the same degree of confidence in your results. In summary, if you think there’s likely to be a strong synergy between two lines of your advert, it’s worth testing all four combinations of the two lines at the same time. If there’s no reason to believe that the two new lines work particularly well together, but poorly individually, then there’s no benefit to running the four options at once, and you should run two a:b tests as usual. For example, if you have the advert: Dave’s Confectionary Buy Our Chocolates! Free Next Day Delivery And you want to try a second line of œGet Luxury Chocolates Here!, and/or a third line of œGift Wrapping Available, you have to decide whether these lines are likely to complement each other, but are unremarkable individually. Here, it looks like the lines are pretty much unrelated, with one being an alternative call to action, and the other suggesting a different feature. So two standard (one change) a:b tests would be quicker or more reliable (depending on how long you wait for your results) than running all four possibilities together. Any comments? Disagree? Let me know¦