Sep 22
Q and A: Why have my crawl rates dropped ?
You look familiar. Are you on Twitter? Subscribed to my feed yet?
Hi Kalena,
First I would like to say thanks to you for your answer of my first question regarding server migration and ranking changes. I have one more query. One of my clients has a web design website and he offers web design services only in US and Europe. So he asked me to block other countries so that website couldn’t be accessed in other countries. I did the same and blocked the other countries using htaccess. However, since I have blocked the other countries, I have observed that web site pages are not being crawled so rapidly as it was being earlier. I just want to know if this problem is due to blocking of other countries or there are some other reasons. Thanks in anticipation.
Regards, Manish
Thanks Manish,
I’m not quite sure why your client would wish to block access to his site in the first place. As far as I’m concerned, the more people that come to my sites the better. Even if they come from overseas and I can’t offer them a direct service, there is still the possibility that they will comment or discuss topics from my Blog – raising the awareness, and improving my rankings. If they can’t see it, they can’t promote it. Blocking his site from OS would also eliminate the possibility of backlinks from OS sites, and whilst these may not provide local link benefits, there are plenty of OS sites that I would love top get backlinks from.
Aside from that – assuming that he does have a valid reason for restricting access – one reason why he may very well have seen a reduction in crawl activity (assuming you are monitoring search spider traffic) is that the crawlers may not necessarily originate from his country. Whilst you might assume that crawlers only crawl local sites – this is not always the case, and depending on what country he is based in, blocked OS traffic might significantly reduce, or even eliminate all crawler traffic completely. If the OS users can’t see it – the OS crawlers can’t either
Andy Henderson
Ireckon Web Marketing
Loading ...
September 23rd, 2009 at 11:42 pm
Spot on Kalena,
To get a visual on this, check out google images. Look for google global network spiders topology, or something similar. You’ll clearly see that these spiders are coming in, and going out everywhere. Exclusive access to one country is not a good idea for this reason.