Q and A: Why doesn’t Google index my entire sitemap?

QuestionHello Kalena

I’ve submitted my sitemap to Google several times, and it doesn’t spider more than 57 pages even when I add more pages. I can’t figure out why and would really appreciate your help!

My website is [URL withheld]. The sitemap I submit to google is called sitemap.xml. I’m working on the site currently, and I want google to find the changes and new pages.

Thanks!
Greg

————————————–

Hi Greg

I’ve had a look at your sitemap and your site and I’ve worked out the problem. I think you’re going to laugh :-)

Yes, you have created a XML sitemap containing all your site URLs. Yes, you have uploaded it via your Webmaster Tools account. However, the robots.txt file on your site contains disallow rules that contradict your sitemap.

There are over 30 URLs in your robots.txt with a disallow instruction for Googlebot.  Essentially, you are giving Google a list of your pages and then instructing the search giant not to go near them! Have you re-designed your site lately? Maybe your site programmers made the change during a large site edit or testing phase and forgot to remove the URLs after completion?

All you need to do is edit your robots.txt file to remove the URLs being disallowed and then resubmit your XML sitemap.

All the best.

——————————————————————–

Need to learn SEO but not sure where to start? Access your Free SEO Lessons. No catch!

 

Share this post with others

Fast Five in Search – Week 28, 2014

fast-five

 

It’s very chilly today here in the best little capital city in the world, Wellington, New Zealand. I’m rugged up against the cold and sporting fingerless gloves as my fingers traverse the keyboard, hunting down some Fast Five deliciousness for you all.

It’s another mixed assortment this week, featuring a little bit of Apple, a lot of Google and a pinch of blogging. Enjoy!

Here’s this week’s Fast Five:

1) Panda Pummels Press Release Web Sites: The Road to Recovery by Russ Jones. Even high-trafficked, Google trusted sites aren’t immune to Google algorithm updates. In this post, Russ explains how the latest iteration of Google Panda has taken a chunk out of traffic from media darlings Search Engine Land and Seer Interactive and what lessons we can apply to our own sites as a result.

2) How Apple and Google are Disrupting Education and Changing the World by TopDegreesOnline. You know I’m a sucker for a good infographic and this one is a cracker. It shows the evolution of education with the advent of technology and the two distinct approaches from tech giants Apple and Google as they endeavour to forever change how we learn.

3) The EU’s Right to Be Forgotten is a Mess & How Google’s Making it Worse by Danny Sullivan. While we’re still on the subject of Google, the company’s recent legal loss in Europe has led to one hot fuss. In case you’re unfamiliar with the case, in May this year, the European Union Court of Justice ruled that Google could be compelled to remove information about individuals from search results as part of a new, EU-specific “right to be forgotten.” The PDF factsheet on the subject will bring you up to speed. Apparently, confused interpretation of the ruling and Google’s attempts to collaborate with it are triggering Internet censorship concerns the world over.

4) Is Your Blog a Lead Generation Machine, if Not Here is Why by Bryan Eisenberg. This bookmark-worthy post sees Bryan share his most successful techniques for converting blog readers into customers / subscribers. Rather than a long-winded blog post, Bryan has embedded his recent SlideShare presentation on the subject, featuring no less than 73 slides of conversion magic. Grab a coffee before viewing this one!

and finally…

5) The Ultimate Guide to Creating a Media Kit by Stacey Roberts. So ProBlogger has been running a theme week about Partnering With Brands. As part of the theme week, this post by Stacey Roberts is a step-by-step guide to creating a media kit for your blog or site. A fantastic resource, the post discusses what a media kit is, why it is useful, what it should include and how often it should be updated.

Happy reading!

*Image courtesy of Threadless.

——————————————————————–

Need to learn SEO but not sure where to start? Download your Free SEO Lesson. No catch!

 

Share this post with others

Fast Five in Search – Week 27, 2014

fast-five

 

Greetings and salutations! It’s been a pretty cruisy week in the office, which has given me more time to research some juicy Fast Five links for you. It’s another mixed bag, a couple of Google stories, a couple of Facebook stories and a curve ball article about Infographics.

Here’s this week’s Fast Five:

1) Why Google Places is Now Google My Business by Warren Knight. This month, Google has quietly re-branded Google Places, by combining it with Google Local and calling it the very uninspired Google My Business. In this article, Warren looks at the upgrade in more detail and gives a big thumbs up to the new streamlined version.

2) Google+ At 3 Years Old: Not a Ghost Town, But a Social Referral Graveyard by Martin Beck. There’s an old joke amongst us online marketers that goes:“Google Plus has millions of users! They all just happen to be Google employees”. But all jokes aside, despite having millions of users, Google’s own social platform is suffering from a chasm of referral traffic. Martin Beck takes a look at the depressing stats and possible reasons for the lack-lustre performance.

3) The Best Infographics of 2014 by Lindsay Kolowich. Bit of a sucker for a good infographic? Yeah, me too. Lucky for us, one of the clever crew over at HubSpot has compiled a list of this year’s most interesting and useful infographics. I feel some serious yak shaving coming on via that link.

4) Facebook and the Ethics of User Manipulation by Alex Wilhelm. So my reader has been lighting up for the past week with stories about *that* Facebook experiment, where staff at the social mammoth supposedly manipulated our newsfeeds to test our psychological reactions. With the true nature of the testing exposed, industry reaction has been overwhelmingly negative, with Huffington Post hysterically comparing the experiment to lab rat testing. This TechCrunch article from Alex Wilhelm is one of the more measured and thoughtful pieces about the entire incident.

and finally…

5) Facebook Responds to Negative Reactions to Its Experiment on Users by Adario Strange. And because it is such a hot topic, here’s another article on the whole Facebook Experiment debacle. This one by Adario Strange of Mashable includes feedback from Facebook both about the experiment and the vitriolic reaction it received from the public.

Happy reading!

*Image courtesy of Threadless.

——————————————————————–

Need to learn SEO but not sure where to start? Download your Free SEO Lesson. No catch!

 

Share this post with others

Google to Offer Domain Registration

domainsThe domain registrar industry is about to crack wide open, with new evidence that Google is moving into the domain registration market.

The move confirms rumors that Google is serious about selling domains – rumors that began when the Internet giant made DNS changes to GoogleDomains.com in late March, after having owned the domain for several years. Domaining is a thriving industry, growing every year, as evidenced by GoDaddy’s latest IPO announcement.

Google Domains is currently in invitation-only BETA release, but on appearance, will be a fully-fledged domain registration service on public launch, with all the customizable domain features of large registrars.

I’ve requested my invitation to participate and will review the service here on the blog as soon as I can. Watch this space!

Share this post with others

Fast Five in Search – Week 15, 2014

fast-five

 

Google’s head of spam fighting – Matt Cutts – posts regular videos on the Google Webmasters YouTube channel. His posts can make waves in the SEO industry like no other because they often preview upcoming changes to the Google algorithm. So this week’s Fast Five is a collection of Matt Cutt’s most popular webmaster videos of all time.

Here’s this week’s Fast Five:

1) How Does Google Search Work? – In this video, Matt explains how Google’s ranking and website evaluation process works from the crawling and analysis of a site, crawling time-lines, frequencies, priorities and filtering processes within the databases. With over 380,000 views, this is Matt’s most popular video in the Google Webmaster channel.

2) Does Google Use the Keyword Meta Tag? – After years of webmaster confusion over whether Google does or doesn’t index the content of the Meta Keywords tag, Matt put the rumor firmly to rest with this post by confirming that Google does NOT index the tag.

3) What Should We Expect in the Next Few Months in Terms of SEO for Google? – After months of Panda and Penguin algorithm tweaks, Matt thought it was time to set a few things straight and reveal a little more about what we might expect from future algorithm updates.

4) Canonical Link Element – When Google launched support for the Canonical Link Element, Matt took to video to introduce the element and the way it should be used for SEO benefit.

and finally…

5) What Are Some Effective Techniques for Building Links? – The fifth most popular Matt Cutts webmaster video involves the consistently difficult subject of link building. In this video post, Matt discusses several effective ways of building organic links that many webmasters overlook.

Happy reading!

*Image courtesy of Threadless.

Share this post with others