Steve's thoughts
Decoding the Quality Score
Improving your click through rate improves your Quality Score. This you probably already know. But how high is a high click through rate? What is a decent click through rate for a given position? How do you know if your Quality Score is being dragged down by the Account Quality Score or your adverts?
These are questions that everyone that manages an Adwords account has asked at some point. So I decided to find out. As a large agency, we are in the position of having far more data to analyse than most people, and by manipulating this data, we’ve found some very clear and consistent trends, that have given us unique insights into what click through rate Google are looking for in order to consider you ‘relevant’, and how they take your position into account.
So here’s what I did. I pulled out the data for the last 30 days for every keyword, in every Ad Group, in every campaign, in every account that we manage. I stripped out any Phrase and Broad Match keywords (since QS is calculated on Exact Match), Google’s Search Partners data (since this is ignored in the calculation) and brand terms (which would skew the click through rate analysis significantly – more about this later).
I then looked at the average click through rate in each position from 1 – 8 (in increments of 0.1), for each keyword Quality Score. By graphing the results, it wasn’t hard to detect a pattern. Here’s the CTR’s by position for keywords with a QS of 10:
Note that I took the total impressions and clicks for each position, effectively weighting the CTR for each keyword by the number of impressions, so that low volume keywords didn’t have an exaggerated impact on the performance. You can also see why I removed brand names from the above analysis – the position 1 CTR would have been influenced excessively…
From the above graph, it’s possible to get some sense of what click through rate Google expects from a keyword in order to give it a QS of 10. And by fitting a curve through the data, it’s possible to get a reasonable indication of where you should be in order to pick up a Quality Score of 10:
Based on this line, it appears that Google expect the click through rate in any position to be about 65% of the next position up. So where position 1.0 has an average click through rate of 34%, position 2 has an average click through rate of 22.1%.
Whilst this line fits nicely through this data, what happens if you look at other Quality Scores? Will Google expect a similar drop-off in click through rate? If not, then this number is just a convenient fit for this data. Here’s the graph for Quality Scores of 9:
And 7:
And 6:
The same decay of 0.65 has been drawn on each, and in all cases, it appears to fit well. So this appears to be Google’s estimate of what ‘should’ happen to your click through rate every time you drop a position – you lose just over 1/3 of your clicks. Of course, this doesn’t really work once you drop below about 8th, but it’s certainly a useful thing to know. Indeed, the line fits just as well with a Quality Score of 5, 4 or 3 as well.
Based on these fitted lines, you can estimate the following average click through rates by position, and how they are converted into Quality Scores:
Using this, you can potentially ‘health check’ your account. If you have a click through rate of 4.5% in position 4, you should have a Quality Score of around 7 or so. If you are getting less than the predicted Quality Score across the bulk of your keywords (excluding brand, on Google only, on Exact Match), then it’s a sign that your account has other issues, possibly with the landing page, keyword relevance or the overall account quality.
I tried to take this analysis to the next level, by calculating the predicted Quality Score for each keyword, based on its click through rate and average position. Unfortunately, whilst the fits are very good looking at large amounts of data, on a keyword-by-keyword basis, the results can be quite inaccurate, underlining the fact that advert relevance, landing page relevance and account quality also have a significant bearing.
One final point worth noting is that the overall figures for the Quality Score of 7 are a little misleading. Most keywords seem to be given a Quality Score of 7 initially, until Google calculates a value based on their performance. Because far more of our keywords get Quality Scores higher than 7 than get less than 7, the effect of new keywords on the ‘7’ performance is pushing up the click through rate significantly.
Realistically, none of this has any bearing on the way that you manage your account. Regardless of your click through rate, you should always be looking to improve it (though obviously not to the detriment of your conversion rate), and the Quality Scores that Google quotes in the data are only indicative – the true Quality Score figure is probably much more precise, and keywords that appear to have the same Quality Score could actually have quite different values (6.6 and 7.4 could both round to 7).
But in terms of offering insight into what may happen to your traffic volumes if your position were to change, it’s very interesting. If one of the advertisers above you were to drop out, you could potentially be looking at a 50% uplift in clicks. If you’ve got a restrictive budget, this could cause problems. Similarly, a new competitor appearing above you could cost you 1/3 of your clicks…
Nice conclusion Steve and thank you for sharing it.
Thank you very much Steve. We´re working very intense with google adwords but your input
will help us a lot in understanding the numbers much better. And hopefully we will increase our CTR which
is actually not very good
Why is a quality score of 8 missing?
When I saw the final table showing average CTR by Position and QS, I realized I need to go and adjust my bidding on my ads that have QS of 9 and 10 in order to get to position 1 and 2. I was able to find the extra funds by completely stopping my ads that had low Q scores.
Extremely useful post. Thanks.
Very Nice, but what does it means “QS is calculated on Exact Match” are you sure? i have ever read this
@Gojo – That’s a very interesting question. For whatever reason, not one single keyword in any of our accounts has a Quality Score of 8. For whatever reason, in the UK at least, this Quality Score appears to be impossible (or almost impossible) to achieve.
@Dave – I fear that you may have misconstrued the message in my blog. I would never advocate pausing keywords that are delivering cost-effective results, nor would I recommend increasing bids on keywords simply because they have they have a high Quality Score.
Whilst it is important to bear in mind your Quality Score in your optimisation, this should not be your goal – something I recently discussed on a blog here: http://bit.ly/gpW5PQ
The decision of whether to pause a keyword should be based (at least in the main part) on the profitability of that keyword, with a brief consideration of its impact on the Account Quality Score – never the keyword Quality Score.
The decision of how much to bid on a keyword should be based solely on finding the most profitable value (or sweet spot) as discussed here: http://bit.ly/msa5fI. Apologies for the formatting; it’s quite an old post.
@Alberto – It’s not something that Google make a lot of noise about, but it’s been confirmed on various blogs that they write: http://bit.ly/kymqiH. It’s been discussed on a number of other websites over the years, such as these: http://bit.ly/Wupy2 http://bit.ly/m9WU30.
In the last one, Brad Geddes makes a legitimate point – it’s exact matches that determine Quality Score, not your keyword on Exact Match. For example, if you are only bidding on Phrase Match, then the QS is calculated on exact matches to that keyword.
Thanks,
Steve
Steve: I left some comments yesterday, maybe they didn’t get through. Some interesting and thought provoking work. I extended my comments and put them over on our blog – love to hear what you think. Mostly I raise three questions to try and help make sure we get the right conclusions from this cool data.
http://www.clickequations.com/blog/2011/04/quality-score-decoded/
Thanks, Craig
Great analysis Steve, I’ll be looking at my accounts with this in mind over the coming days. Thanks so much for sharing it.
Cheers,
Andy
Super analysis Steve, Did you also make analysis on what these 2 metric do with the avg. cpc’s? This would be an extra interesting metric to add don’t you think?
Hey Steve, I think a QS of 8 is quite possible to achieve here in Australia. I have seen it on numerous accounts i manage. Great blog post.
Hi Steve,
Thank you – good analysis. Trying to unknit exactly how Google’s ‘quality score’ algorithm works is a challenge. It hurts if the quality score is poor due to the PPC costs being ratcheted up.
The worst part is not knowing how much your account history impacts the bigger picture – if, being in-experienced, the CTR has been poor – it seems like there needs to be a long period of PPC premiums being paid, in order to get to the place where the algorithm can shine a good light on your account, and start allowing lower PPC costs.
Great analysis. From your numbers, I must have some kind of problem with my account. See my blog post here:
http://albany-lawyer.blogspot.com/2011/05/adwords-and-quality-score-fiction.html
I have a few keywords in my account with QS 10, 9, and 8, and a large number with 7s. I also advertise in some very competitive arenas and see low QS in them. That might be dragging me down overall, but I have to advertise in those areas.
@Craig – Not sure what happened to your first comment, but I’ve replied on your blog. Apologies for the delay, I’ve been away on holiday.
@Ilja – Much as I’d like to have included some analysis on cost per clicks, it was impossible as I had to aggregate data across totally unrelated keywords. Some had a cpc of a few pence, where others cost many pounds per click – aggregating the cpc’s wouldn’t have given any meaningful results, regrettably.
@Warren – Looking at your data, the number of clicks on each keyword is very low, so Google will only place a very limited weight on the click through rate. As a result, they are probably still relying heavily on the historical performance of that keyword across all advertisers, and penalising your keywords accordingly. If this is the case, then you should see your Quality Score improve over time.
@Steve
Very interesting post. I have use your CTR data for a model in excel – drop me an email and I`ll FW the file.
Note: I am surprised to see a linear CTR at position 1-3 vs 4-10 as I would expect to see this reduce more as the advert moves to RightHandside.
Re: “65%” CTR increase in position increase … Personally, I think it easier to understand a “35%” loss in CTR per dropped position, but depends if you glass is “Half-empty” vs “Half-full”, if you know what I mean
Cheers
Phil.
P.S. Did you check FirstPage Bids vs QS aswell (as strong correlation).
@Phil
I was expecting a discontinuity between positions 3 and 4, but I think that the reason there isn’t one is because these are average positions over a period of time. The results with a position of 3.0 probably appeared 2nd sometimes, 3rd other times, 4th occasionally, etc…
In terms of the First Page Bids, it’s actually based on the Quality Score, so the correlation is to be expected.
Many thanks,
Steve
Wonderful guide! Thank you for this particular, ended up getting an interesting study, inspiring keep it