Skip to Content

The author

Malcolm Slade

Head of Technical SEO

CaffeineNot long after my last post documenting our Google Caffeine sandbox experiences Get ready for some caffeine, Google Caffeine was rolled out on to a single server for further testing. Again, we have taken some time out to test various aspects with the following results.

The Google Caffeine IP Tested

Firstly, if you are unsure as to what Google Caffeine is, I would advise looking at one of our early Caffeine related post; Get ready for some caffeine – November 2009 Google Introduce Caffeine: The Google GTi – August 2009 Pretty much as soon as the Google Caffeine sandbox was closed, people started speculating about various data centres being the first to show Caffeine results. After reviewing many such claims, we found that eventually one data centre did seem to be showing Google Caffeine results, some of the time. This was then later confirmed by Matt Cutts of Google. The IP address was and it could be accessed via where the query variable (q) could be set to your search query (with +s to replace spaces of course).

The Tests

We carried out three tests around Caffeine. Firstly, as Caffeine has always been documented as likely to affect the speed in which results are returned and the size of the returned data set for a result, we looked at those two variables. As Caffeine is likely to affect index sizes, it may also have some impact on rankings due to more pages of your site being found or more pages containing a link to your site being found on other sites etc. So we looked at ranking variance for the third test. The tests were kept simple and carried out using IE8 on a base unit that connected to the internet via a UK IP address that had never been used to search for any of these terms before. In fact, prior to being set up for this test, the box had never even connected to the internet before.

Test 1 - Size of Returned Results Sets

We generated a list of 100 search terms, ranging from head terms such as ‘mortgages’ to long tail terms such as ‘government grants for energy research’. Each search was carried out using and the Google Caffeine IP above. The size of the returned dataset as shown by the ‘of about’ line (Results 1-10 of about 10,200…) was recorded for both and the percentage difference between Caffeine and was calculated for each. As I said, nothing too complicated. test1 When the Google Caffeine sandbox first went live, initial tests carried out by others showed that the data sets returned by Google Caffeine were much larger than those returned by normal Google. When we carried out our own testing nearer to the end of the sandbox’s existence, this difference was almost unnoticeable, with the average difference being that Google Caffeine returned 4% more results (a 4% bigger data set). The latest set of tests using the live Google Caffeine IP were even more unexpected, with most searches bringing back considerably smaller data sets through Google Caffeine (as can be seen from the above graph). On average, Google Caffeine returned 13% fewer results. In some instances, the data set returned was nearly 70% smaller.

Test 2 - Speed of Returned Results

At the same time as testing the size of the returned results sets, we also recorded the time taken to receive those results and display the first page, as shown by the time quoted after the data set size. It’s important to remember for this test that several things could have an impact, not least the fact that we had to refresh the result until it showed Caffeine results. Google is also already pretty fast at returning results, so rather than worry about how much faster Caffeine was or the average speed difference etc., we kept this test really simple and based it on wins. test2 This is the first time that we can make any real conclusions about speed, as the sandbox environment wasn’t a true representation of Google to compare like-for-like. Even now, the result has to be taken with a pinch of salt and certainly wasn’t conclusive, but on average (52% of the time), Google Caffeine was faster. But then, if it is passing back smaller results sets, it should be quicker, shouldn’t it?! However, it could also be said that if Caffeine is indexing more and therefore has a large index to work with to construct the data set to be returned from, it should actually be slower.

Test 3 - Ranking Changes

test3 For our final test, we looked at how rankings differ for 145 terms that relate to our clients. We took the clients ranking position now for the particular term and compared it with where Google Caffeine ranks them for the same term. Just to clarify, a positive number equates to Google Caffeine ranking the client lower than normal Google, while a negative number means the ranking was better on Caffeine. As can be seen from the graph above, the range of ranking changes was limited, with the majority showing improvement. Some did slip, but none suffered any major impact from Caffeine. Of course, we are not aiming to draw any conclusions from this particular experiment. We are, however, sleeping better.


Google Caffeine does seem to be returning smaller sets of results most of the time, which could be due to an improvement in relevance. If Google Caffeine is drawing from a larger index, then it is definitely faster at working out the data set to return. All along, we have expected ranking changes to be minimal, and the latest tests seem to confirm this. In general, Google Caffeine looks to be a success so far. We are expecting the full roll-out of Google Caffeine around February, and will probably carry out another round of testing a few weeks after, when things have settled. Until then, get your fill of decaff. Caffeine is coming.