Since around August there has been an intermittent problem for .com sites with Google results in country specific listings (e.g. searching Google.co.uk and selecting UK-only results). What has been happening is that the home pages of these sites drop out of the country specific indexes, although they are still there in the global indexes. (To check if you’re affected simply do a site:www.mydomain.com in the two versions of the results) Along with this drop-out the rankings for any search term that is in any way reliant on the home page drop as well, and this has been causing problems for many businesses. A strange symptom that also often appears in these cases is that an https version of the domain address shows up in the indexes, even though there is usually no such version in existence.

There’s been a fair bit of discussion of this in the forums (for instance on webmasterworld) and those affected have reported that their sites periodically reappear and then drop out again, which agrees with what we’ve seen on some affected sites that we monitor. By no means all .com sites are affected and various theories have been tried and proved faulty as to what might be triggering these problems. The general opinion seems to be that at some point Google tried to introduce some sort of geo-specific filter which has gone wrong, and have subsequently been trying to fix it, however none of us really know and Google aren’t saying anything, even in reply to specific questions.

Sadly this is par for the course and is difficult to understand. We saw the same thing last year when what is now called the Big Daddy update took place. At that time many perfectly ethical sites dropped out of the indexes for a number of months while many spammy sites seemed to reappear. Eventually the update was rolled out and most of the good sites came back and reclaimed the rankings they had held previously. One of my own sites suffered this effect, even for some very specific terms for which I was clearly amongst the most relevant sites. At that time there was a great outcry but very little if any useful feedback from Google – just the usual generalities about making sure you do all the normal ethical things.

Now you can understand Google wanting to distance itself from making too many direct answers because they must certainly be inundated with queries from both ignorant users who can’t manage to read the T&Cs, and spammers who are trying to get back into the rankings after being banned. However when people have legitimate questions and there are provable problems with the results you would think that it would benefit them from a technical point of view as well as a PR point of view to be a little more communicative. While the sitemap / webmaster interface is a step in the right direction there are still many questions which are unanswered. SEO clients, who are often big businesses themselves, cannot understand how their SEOs can’t just phone up Google and get answers. The apparent climate of secrecy contributes to the “smoke and mirrors” reputation that SEO has.

The current situation only gives credence to the opinions of those cynics who point to the fact that these major index drops always seem to happen in the run-up to Christmas and are deliberately engineered to boost Adwords revenue. The best way to scotch such suggestions would be for Google to be more open about what is going on and admit any problems when they occur. Even if nothing can be done to speed up the solutions it would ease the minds of those affected, stop them tinkering with their sites in ways that may well be harmful in the long term, and greatly increase their respect for the company that was founded on the idea of “do no evil” but whose reputation is no longer seen as whiter than white.

Google aren’t the only ones to have some strange results at the moment. MSN / Windows Live is also doing some odd things. While checking my own listings I noticed that if you go to msn.com (which these days no longer redirects you to the uk version) and search for “search engine optimisation scotland”, the top ranked site is a holding page for a forthcoming site and contains no content apart from a handful of Google Adsense ads. Surely some mistake!