I have encountered many people who have caused problems when implementing a website test, often breaking the tracking. My usual experience is with Google’s website optimiser and Analytics, often erroneously recording traffic as direct or organic depending upon the circumstance. I have come up against similar issues with alternative vendors for both analytics and website testing solutions and many of the principal checks apply to all situations.
Website testing regularly causes tracking issues and contained below are three basic checks as soon as any new test goes live:
(Assuming website test went live on = www.mywebsite.co.uk/test)
1. Is your site listed under the referrals section within analytics?
Yes – Since Implementing The Test
If a significant portion of your traffic is listed under the referrals section, it is highly likely that there is an issue with your tracking.
Yes – Including A Time Period Previous To Test Implementation
Your tracking has probably had issues for a while, and this should definitely be investigated further. At a guess, likely to be a result of subdomain/cross domain tracking issues or a combination of code placement and site speed.
No
Good, no issues so far.
2. Have you seen a sudden spike in Organic Traffic, Direct Traffic (or in fact, any traffic source) that coincides with the test?
Yes – Since Implementing The Test
It is likely that the source has been lost for an alternative traffic source such as paid search. Try navigating to the page that contains the test.
Now there are two main ways to see both test variations, use two separate browsers, navigate to the test page, and keep clearing cookies on one of them until you have a browser set to each page (an uncouth but still effective approach).
Google Analytics Specific
(The principal still applies to other tracking solutions)
Force analytics to show you a version by the addition of variables, add the following onto your URL and change the value after preview to view different versions:
# utmxid=XXXXXXXXXXXXXXXXXXXXXXXXXXXX;utmxpreview=1
To obtain your utmxid for website optimiser, preview your experiment and take the id from the preview URL. More details can be found here:
http://support.google.com/websiteoptimizer/bin/answer.py?hl=en-GB&answer=63841
e.g.
www.mywebsite.co.uk/test#utmxid=XXXXXXXXXXXXXXXXXXXXXXXXXXXX;utmxpreview=1
This should then allow you to choose which variation you are shown.
Now try adding ‘?utm_source=TEST’ into the URL, after being re-directed by the test, does the utmx cookie still contain TEST as the source value?
If not, it is likely the analytics cookie has not been set before the re-direct occurred, thus losing the original source. You need to ensure that the cookies are set before the re-direct takes place.
No
Perform the test above anyway, just in case a random traffic surge masks any spike you may have noticed (unlikely you say, at least until it happens to you).
- Appear to have a small spike yet cannot find an issue?
Segment your traffic. If a specific traffic segment is causing the spike it is likely that that particular segment has an issue. It may be that your site breaks for all mobile traffic, does not work in IE, Chrome, Firefox or even the tracking above could be breaking on a single segment. Segmentation should help you narrow down any analytics problem to the source, not just for website testing.
Any questions or comments? Please leave them below or on Twitter -@JamesMHolding