Q and A: Why is Google having trouble indexing our site?

QuestionHello Kalena

One of the sites we manage has a problem.

The homepage at [URL removed] is not getting indexed anymore by Google. The site was made using Sitefinity 3.7 and the hosting is provided by Rackspace. Something similar already happened two times in the past which we resolved using the option “index this page” on the page generated by Sitefinity (1st time) and by re-creating the XML sitemap and linking it directly to Google Webmaster tools (2nd time).

This time we can’t seem to find the reason. We checked if the end-user that works as the back-end has made any changes or if there was any notification from Google Webmaster Tools reports but nothing came up. Here are some more technical details:

1) The site homepage is [URL removed]. But the site root is [URL removed] which is an empty page with a redirect to the home page using a 301 redirect.

2) In Google Webmaster Tools we set up 2 Sitemaps:

  • The first at [URL removed] is indexing the Top pages of the Home page (static)
  • The second [URL removed] gets populated with the pages content generated by Sitefinity (dynamic)

3) Also, from the back-end options, a metatag ROBOTS was set at page level for the top pages, as Google suggests.

4) Google reports 5 blocked URLs when crawling our robots.txt with the message: “Google tried to crawl these URLs in the last 90 days, but was blocked by robots.txt”. This seems suspicious, because I can’t seem to understand what could be blocking it, the robot is pretty simple and not restrictive.

Could you give us an hand? I’ve left a generous donation for your coffee fund.



Hi Jim

First up, thanks for the caffeine donation :-)

As for your problem, oh boy. You’ve got a few different issues going on, so let me address each of them separately:

1) Your XML sitemaps are missing contextual data specified by the Sitemaps protocol. In particular, your < loc > child entries per URL are messed up. I’m surprised this hasn’t generated an error in Webmaster Tools, but I’m pretty sure it would be confusing Googlebot. Go check your sitemaps against the protocol and re-generate them if necessary. Maybe use one of the XML generator tools recommended by Google. Personally, I like XML Sitemaps (yes that’s my affiliate link).

Also, why 2 separate sitemaps for HTML pages? I can understand having separate ones for RSS feeds or structured data stuff, but your standard site pages should all be listed in the one file so you can better manage the content and keep track of indexing history in Webmaster Tools.

2) Your robots.txt file is blocking a number of pages that you have listed in your XML sitemap. So on the one hand you’re telling Google to index pages within a certain directory, but on the other, you’re telling Google they are not allowed to access that directory. This is what the error message is about. You’ve also got conflicting instructions on some of your pages in terms of robots meta tags vs. robots.txt.

3) The 301 redirect on your root directory is your major problem. In fact, that empty landing page is your major problem. Why do you need it? You don’t use Flash and it doesn’t appear to have an IP sniffer for geo-location purposes so I can’t understand why you wouldn’t just put your home page content at the root level and let search engines index it as expected.

The way you have it set up right now is essentially telling Google that you have moved all your content to a new location, when you really haven’t. It’s adding another step to the indexing process and you are also shooting yourself in the foot as every 301 contributes to some lost PageRank. Google clearly doesn’t like the set up or isn’t processing it for some reason. There also appear to be several hundred 301s in place for other pages, so I’m not sure what that’s about. I don’t have access to your .htaccess file, but I can imagine it reads like a book!

4) Unless you specifically need a robots meta tag for a particular page scenario, I would avoid using them on every page. You can achieve the same results with your robots.txt file and it’s easier to manage robot instructions in one location rather than having to edit page by page – avoiding conflicting issues as you have now.

Apart from the obvious issues mentioned above – have you considered switching away from Sitefinity and over to WordPress? I’ve struggled optimizing Sitefinity sites for years – it’s a powerful CMS but it was never built with search engines in mind and always requires clunky hacks to get content optimized. Plus that’s a really out-dated version of Sitefinity.

Given the other issues, it might be time for a total site rebuild?

Best of luck


Like to learn SEO with a view to starting your own business? Access your Free SEO Lessons. No catch!


Spread the joy!

Fast Five in Search – Week 38, 2014



So I answer a lot of questions about search engines on this blog. But did you know that Google also has a Q and A site? This week’s Fast Five is dedicated to some of the more popular questions asked about Google.

Here’s this week’s Fast Five:

1) Does validating my site’s code (with a tool such as the W3C validator) help my site’s ranking in Google?

2) How can I get those links displayed under my site’s listing in Google’s search results like some other sites have?

3) Is the server location important for geotargeting?

4) Why doesn’t my site show rich snippets? I added everything and the test tool shows it’s ok.

and finally…

5) Why is my sitemap file showing a submitted URL count that does not match the number of entries in my sitemap file?

Happy reading!

*Image courtesy of Threadless.


Need to learn SEO but not sure where to start? Download your Free SEO Lesson. No catch!


Spread the joy!

Q and A: Why doesn’t Google index my entire sitemap?

QuestionHello Kalena

I’ve submitted my sitemap to Google several times, and it doesn’t spider more than 57 pages even when I add more pages. I can’t figure out why and would really appreciate your help!

My website is [URL withheld]. The sitemap I submit to google is called sitemap.xml. I’m working on the site currently, and I want google to find the changes and new pages.



Hi Greg

I’ve had a look at your sitemap and your site and I’ve worked out the problem. I think you’re going to laugh :-)

Yes, you have created a XML sitemap containing all your site URLs. Yes, you have uploaded it via your Webmaster Tools account. However, the robots.txt file on your site contains disallow rules that contradict your sitemap.

There are over 30 URLs in your robots.txt with a disallow instruction for Googlebot.  Essentially, you are giving Google a list of your pages and then instructing the search giant not to go near them! Have you re-designed your site lately? Maybe your site programmers made the change during a large site edit or testing phase and forgot to remove the URLs after completion?

All you need to do is edit your robots.txt file to remove the URLs being disallowed and then resubmit your XML sitemap.

All the best.


Need to learn SEO but not sure where to start? Access your Free SEO Lessons. No catch!


Spread the joy!

Q and A: Why Doesn’t Google Rank My Site Higher?

QuestionHi Kalena,

My website has been up and running since 2008.

I regularly add new content & update my blog & facebook pages and yet 5 years on, I am still only attracting 30-40 visits per day and it has remained at this level for 5 years.

I believe my website is user friendly, visually pleasing & provides useful information for the visitor looking for the product I offer, so why doesn’t google rank it higher? I only have 83 pages indexed out of 1,400 – please help!



Hi Natalie

Assuming the website you are referring to is the one associated with your email address, I can provide a few recommendations immediately:

1) I couldn’t find a XML sitemap in the expected location of http://www.[brandwitheld].co.uk/sitemap.xml. Now maybe you have one in a different location, but if not, you’ll want to create one as soon as possible.  An XML sitemap is a file that contains a full list of indexable pages on your web site. It is the preferred method of lettting search engines know about all pages on your site that you want indexed. More information about the protocol and format required is available at Sitemaps.org. You can upload your sitemap via your Google Webmaster Tools account. Haven’t got one of those either? Read on…

2) If your site doesn’t seem to be as visible as you’d like in Google or large chunks of it are not getting indexed, make sure you create a Google Webmaster Tools account and check it for any obvious issues. Google provides an exhaustive amount of insight and advice in Webmaster Tools in terms of technical issues, indexing issues, SEO issues and visitor activity related to your site. If there seems to be something wrong, that should be your first stop.

3) Check your site against these 10 Most Common SEO Mistakes to see if you’re guilty of any of them and address the issues quickly.

4) You claim your site has 1,400 pages, but most of those are product and category pages consisting of dynamically generated versions of the same URL, plus a huge number of pop-ups. The site only has a small number of static HTML pages and therefore only a small amount of content that is visible to search engines.

For example: http://www.[brandwitheld].co.uk/fabrics.html is your main curtain category and then you have 11 different sub-categories under that, such as:

a) http://www.[brandwitheld].co.uk/fabrics.html?cat=browns

b) http://www.[brandwitheld].co.uk/fabrics.html?cat=reds

c) http://www.[brandwitheld].co.uk/fabrics.html?cat=golds

However, the content for each sub-category is dynamically generated from a product database, based on the category parameter indicated after the *?*.  To a search engine, a), b) and c) are seen as the same, single, page, with everything following the *?* generally ignored or treated as duplicate content.

Even worse, those sub-categories then break down into individual styles, but the style information is presented as a javascript pop-up dialogs, which can cause their own issues.

I’d put money on your URL structure being the spanner in the works preventing most of your content from being indexed. No wonder you see so few pages ranking well! I would suggest learning more about how Google treats URL parameters and reworking your site content to create flat, indexable HTML pages for each product, category and style.

Natalie, without having access to your Webmaster Tools account, I can’t really give you more advice at this point. However, if you’d like to invest in a full web site audit, I can certainly take a much closer look. Just contact me to get started.

Hope this helps!


Spread the joy!

Q and A: How Do I Prepare for a Large Site Migration?

QuestionHi Kalena

I work for a medium sized hospitality chain (in the marketing dept) and our leading chain of hotels is about to undergo a brand change. I’ve just found out that management has approved a full domain name change for each of these hotels and scheduled it with our IT department to happen next month. My General Manager bought the domain name without consulting IT or marketing.

I’m freaking out a little because I’ve been given the task of making sure the change goes smoothly and doesn’t impact our Google rankings or traffic, which I’ve spent years building up. There are 3 different regional hotel properties that will be affected and the content will be transferred over to a single domain! What should I expect? Is there anything I can do to make the transition go smoothly?



Hi Belinda

Oh boy, I don’t envy you. Yes, you are right to be freaking out – at least a little. Site migrations are a royal pain in the you know where and can result in masses of lost traffic and lost search engine rankings.

By the sound of things, your site migration will be complicated by the fact that there are multiple domains shifting to a single domain. Now before you start hyperventilating, there are some things you can do in preperation:

1) Read this terrific presentation about site migration by Aidan Beanland of Yahoo and then read it again. Create a plan for your own migration situation.

2) Go spend some time with the IT department. Hopefully you get on well with them because you’ll be spending a lot of time talking to them over the next few months. Provide them with a copy of Aidan’s guide so they know what to expect. You’ll need to find out their strategy for the roll-out, including pre-switch benchmarking, 301 redirect integration and testing, specific dates for content transfer, the big switch and final DNS propagation.

3) Consider shifting the content of each individual hotel into distinct region-based sub-domains on the new site e.g. Dallas.HotelBrand.com, Austin.HotelBrand, Houston.HotelBrand rather than trying to combine all content into a single site. This way, you can optimize the sub-domains as distinctive sites and retain the location-related Google rankings you have spent so long building up. If you can prove large traffic losses will occur if you don’t do this (and they will!), it should be easy to get IT and management onside.

4) Take an active role in the pre-migration benchmarking process, particularly in relation to site analytics, most popular content and search engine rankings. Ensure your company keep ownership of the old domains and keep all sites live until the new domain has fully propagated.

5) Be prepared with other online/offline marketing activities to promote the hotels in case of sudden traffic loss.

6) Make sure your manager and stakeholders know what is within/beyond your control! Make it very clear what can go wrong during the move and protect yourself by warning them ahead of time of the potential negative outcomes.

Good Luck!


Struggling to get your site higher in Google? Download my Free SEO Lesson. No catch!


Spread the joy!