Why Google & YouTube (probably) disappointed millions of Michael Jackson fans today
Site Migration Guide - Techniques and Advice to Aid with Successful Site Migration.
25 Jun 2009
Once upon a time, migrating a site to a new platform, technology or domain required very little consideration. Is the host reliable? Can I afford it? Will it run my site? Do I like my new domain name? Fast forward 20 years and with the explosion of e-commerce and search engine usage, the process of migrating a site has become a complex thing that only a fool would brush off or take lightly. This post will discuss several of the issues that should be considered and some basic techniques to aid with site migration.
The cold hard truth is that migrating a site is a risk even if it's simply to a new host. Best case, you take a hit in rankings and traffic for a short period of time, followed by a return and improvement in both. Worst case, your site loses all its traffic and never recovers. As with any risk it needs to be managed effectively and monitored. Here at Epiphany we use a simple 3 stage process to manage most site migrations consisting of data gathering and analysis, the actual migration implementation and the monitoring process or the final result. Before we get into the details of each stage, it is important to understand roughly what happens when you migrate to a new site with new URLs. So... What happens if I just replace my old site with a new site with new URLs? If you simply upload your new site to the server in place of the old then the following is likely to happen (assuming that the old site had some value);
- Search engines will revisit all the pages of the old site that they have in their indexes and receive 404 errors where the page no longer exists. Over a period of time (usually up to 3 months depending on the strength of incoming links and the index rate of the page) the pages will be removed from the search indexes.
- In the interim period while the pages are still in the index but 404, users arriving on the site from a listing for any of the missing pages will also receive a 404 and if not handled correctly will likely return to the SERPs page from where they came.
- Search engines will begin to devalue the links pointing to the pages that now 404 in effect losing all of the links as they no longer point to valid pages. Rankings and traffic may fall away dramatically at this point.
- Search engines will index the new site pages over time and rankings will start to return but due to the loss of many of the page links, they are likely to fall short of the pre migration rankings.
So, hopefully you can see the importance of a structured migration if you wish to avoid any major impact on your rankings and / or traffic.
Stage 1 - Data Gathering and Analysis
The first stage of the process involves ascertaining the strengths of the current site and exactly what you donâ€™t want to lose. To ascertain what you already have, you need to make a list of each page of your site and document certain attributes. Basic things to consider are the Toolbar PageRank of the page, the number of in-bound links to the page from other sites, the amount of visitors using the page as a landing page, whether the page ranks for any of your major traffic driving terms and whether it is indexed. This information can be gathered from Google Analytics, a bit of searching and Webmaster Tools. The result should be something similar to: As you can see, I have added 2 additional columns entitled "Keeper" and "Action". Here you record whether the page has enough value to warrant any action and what the action should be. The metrics for gauging whether a page is a keeper or not is relative to the site. If the site only has 20 pages then it is a relatively small operation to handle all 20 in the best possible fashion. If however the site is 100,000 pages plus, you are going to need to filter your list to make the best use of your resources. If none of your blog articles are indexed then why bother setting up redirects to your new blog section etc. From the table above, I have chosen to keep all of the pages in the top section as they receive a considerable amount of traffic, they are indexed, they rank for important terms and they all have links pointing to them from other sites. In an ideal world any URL marked as a keeper would simply be replicated in the structure of the new site as individual URLs can (be said to) have history. There are, however, instances where this is impossible such as a radical re-categorisation or a major overhaul of the site structure, here you either need to 301 redirect the old URL to the new URL or 301 redirect the old URL to the most appropriate page on the new site. The key here is to 301 as the power of the old URL will, in time, pass to the new - as will visitors still using the old URL. The lower part of the table shows pages that have less value in terms of search. Here the 2 print pages have little to no value as they have no incoming links, no PageRank and no traffic. These can be left to 404 within the new site. It is also arguable that the privacy, contact and delivery pages could be left to 404 but chances are they have a direct counterpart in the new site making redirection simple. As mentioned, you will have to come up with your own metric for disguarding pages but generally if it isnâ€™t indexed it can be canned, if it has no traffic it can be canned and if it has no incoming links, no PageRank and holds no important rankings it can be canned. The main purpose of this exercise is to identify all the definite keepers. Armed with this list of keepers we can move on to stage 2.
Stage 2 - Migration Implementation
As mentioned previously the best option is to have the equivalent page on the new site on the same URL as the old. This of course is rarely possible on changes that are anything more than a simple re-skin. So for every keeper page on the list we need to find either the equivalent on the new site or the most appropriate landing page should there be no equivalent. We then need to implement a 301 redirect to handle each of keeper pages so that any request for them is correctly forwarded via a 301 Permanent Redirect to the desired page on the new site. Next we need to ensure the pages on the new site that we are redirecting to contain the necessary information to maintain any rankings. In the case of like for like pages e.g. /cakes/cheesecakes.php on the old site and /our-range/cakes/cheese.php both of which contain the details of the available cheesecakes, this is as simple as ensuring the Meta Title () is ported from the old to the new or is very similar and the actual textual content is identical or similar. In instances where there is no like for like page e.g. /cakes/cheesecakes.php on the old site which talks about the cheesecake range and /our-range/cakes.php on the new site which talks about all the different cakes available it is necessary to ensure target terms are included in the Meta Title of the new page and within the copy for example by having a paragraph that talks solely about cheesecakes. Next we need to ensure we have implemented an appropriate 404 page that firstly passed a 404 Page Not Found response and secondly provides useful navigation and content to persuade at least some visitors to remain on the site and have a go at finding the content they were expecting. Then we need to create 2 XML sitemaps. One of these should simply contain all of the new URLs within the site. The second of these should contain all the new URLs and all of the old URLs of the keeper pages. We then resubmit the second of these sitemaps to the search engines to speed up the rate in which the search engines re-index the old URLs and find the 301 redirects. Finally, if you are moving to a new domain you can now tell Google all about it through the Webmaster Tools console as illustrated below: This should (why do I feel the need to add a disclaimer here) speed up the process of Google replacing the old site with the new, although at this time, I haven't yet tested it as it only went live a week ago. One other thing you can do is contact the owners of the sites that are now linking to pages that were not on the keeper list and try to get them to modify their link to point to a more appropriate page that actually renders. Having taken all this action you may be thinking what happens now. So... What happens now I have replaced my old site with a new site with new URLs but implemented a proper plan for the migration? What should now happen is:
- The search engines attempt to re-index the pages they already have in their indexes and are told via 301 redirects that the page has permanently moved to a new location. They then update their indexes replacing the old pages with the new.
- Any visitors who have bookmarked old pages will simply be shown the new page as will any visitors who arrived via a listing for an old page while the search engine indexes are still updating.
- The power of the links pointing to the old pages deemed keepers is passed through the 301 redirect and attributed to the new page (Google seems to handle this well while Bing and Yahoo are less reliable).
- Rankings, users and traffic are minimally affected by the whole process.
On to stage 3.
Stage 3 â€“ Monitoring
If you follow the advice above there should be no unforeseen problems but it is still wise to monitor the process just in case. At the start of the process you should run a site: or similar on Google, Bing and Yahoo to record the number of pages that each search engine has from the old site in its index. You should also record your search rankings for all of your major traffic driving terms along with details of the pages that are actually ranking for them on the old site. Once the new site is live you should review this information every 2 days along with your Google Webmaster Tools account to ensure any issues are addressed as they arise. You should also make a record of the actual number of pages within your new site to see how well it is getting indexed. What you should see in a healthy migration is: A fall in the number of indexed pages from the old site in each of the search engine indexes; A rise in the number of indexed pages from the new site in each of the search engine indexes; A slight dip in rankings and traffic followed by a recovery; Ranking pages should switch from old URLs to new as per the 301 redirects rather than different pages all together. Things to look out for are rankings that simply disappear (you may not have taken this ranking into consideration and should do so ASAP); a falling old page count without a rise in the number of new pages indexed (you may have created spidering problems in your new site or copied an inappropriate robots.txt file over from the development server); and/or a large number of 404 errors reported within your Webmaster Tools account. Unless your site is considered very important in the grand scale of things by the search engines (PR 7+ although even this doesn't guarantee importance) it can take up to 3 months for things to settle down and bed in. Search engines have to revisit your pages, update their indexes and redistribute power based on your new structure. All these things take time especially on large sites with deep structures or sites with mediocre value.
This mainly covers the migration of a site to another with different URLs. The same principles can be applied to changes of domains although if the site remains the same it is simply a case of 301 redirecting everything to the new domain with no need to leave anything 404ing. The main issue here is the difference between the old and the new domains with regard to age, history etc. Generally if an aged domain is better than a new one and if the old domain has had similarly themed content on it before then it's a winner. One thing that I hear more often than I should is we can't keep the old URLs because we have changed the underlying technology, this, in the majority of cases, is actually we can't do that as we can't be bothered, we didn't understand the request or we don't actually understand web servers etc. It doesn't matter if your site uses ASP, PHP, Coldfusion or just plain HTML, your web server should still be able to be set up to use .goat, .extension or even no dots at all. When moving host it can be a little tricky to foresee any potential issues. As a rule we recommend hosting in the target country especially if you are not using a TLD specific to the country. There are a number of other things that you can do to aid with targeting a specific country but they go beyond the boundaries of this post. Oh and make sure you know exactly where the server your site is sat on is hosted. I know of many UK sites that rank very well in Google.de but get no UK traffic. I would tell you the host in question but I'm sure you can all put 1 and 1 together to get the answer. Also you could but up a random site on both hosts and monitor load times to ensure the hosts perform appropriately (more important with small hosting outfits and cheap hosting etc.). Why did we create 2 XML sitemaps? Oh yeah! Once you are confident that the old pages have disappeared from the search engine indexes you can switch over to the smaller sitemap which will completely cut the cord. I wouldn't recommend dropping the 301 redirects though as although search engines have updated their data, chances are your link partners haven't, neither have your users. Good luck, take your time and don't panic. Malcolm Slade