Q and A: Why doesn’t Google index my entire sitemap?

QuestionHello Kalena

I’ve submitted my sitemap to Google several times, and it doesn’t spider more than 57 pages even when I add more pages. I can’t figure out why and would really appreciate your help!

My website is [URL withheld]. The sitemap I submit to google is called sitemap.xml. I’m working on the site currently, and I want google to find the changes and new pages.

Thanks!
Greg

————————————–

Hi Greg

I’ve had a look at your sitemap and your site and I’ve worked out the problem. I think you’re going to laugh :-)

Yes, you have created a XML sitemap containing all your site URLs. Yes, you have uploaded it via your Webmaster Tools account. However, the robots.txt file on your site contains disallow rules that contradict your sitemap.

There are over 30 URLs in your robots.txt with a disallow instruction for Googlebot.  Essentially, you are giving Google a list of your pages and then instructing the search giant not to go near them! Have you re-designed your site lately? Maybe your site programmers made the change during a large site edit or testing phase and forgot to remove the URLs after completion?

All you need to do is edit your robots.txt file to remove the URLs being disallowed and then resubmit your XML sitemap.

All the best.

——————————————————————–

Need to learn SEO but not sure where to start? Access your Free SEO Lessons. No catch!

 

Share this post with others

Search Industry Job of the Week – Marketing Manager SEO

Job Title: Marketing Manager Search Engine Optimization
Job Reference: MAC01914
Position Type: full time
Name of employer: Macy’s Inc
Location: San Francisco
Date Posted: 21 September 2013
Position description:

Macy’s are looking for a hands-on natural search optimization (SEO) professional, with excellent problem solving skills and a basic understanding of SEO issues involving site architecture; keyword generation; search friendly content; link building strategies; and metrics-driven SEO. Strong communication skills are essential in this role, as the Manager will be integral in socializing SEO across the organization.

Essential Functions:

  • Work with the Director of SEO to test, plan and successfully execute on ROI positive SEO initiatives, including but not limited to keyword generation, content creation, URL selection for XML sitemap submissions, internal linking, and external linking.
  • Work closely with engineering, creative, merchants, and product teams to ensure consistency in SEO strategy and implementation across multiple properties.
  • Measure and report on the effectiveness of SEO strategies in generating increased web traffic and organic revenue.
  • Develop, share and implement SEO best practices across multiple families of businesses.
  • Work with external agencies and in-house teams to build quality external links.
  • Stay up to date with industry trends
  • Support merchant team with the development and creation of proposals to secure incremental vendor co-operative marketing dollars.
  • Mine macys.com’s onsite search logs and organic search referrals on a monthly basis to grow keyword portfolio.
  • Provide a weekly analysis of natural search term performance based on traffic, conversion, and sales.
  • Team up with Marketing Analyst to identify trends in behavior of customers referred by natural search.
  • Co-ordinate the design, testing, and production of new reports, as required, to improve conversion and sales from natural search.
  • Publish reports and findings to the organization on a regularly scheduled basis.
  • Regular, dependable attendance and punctuality.

Qualifications:

  • BA/BS in Marketing with a strong understanding of technical SEO or BA/BS in Engineering or similar technical discipline or relevant work experience.
  • 3+ years of experience in online marketing with a successful track record, either in-house or at an agency/consultancy.
  • Understanding of basic HTML, CSS and code structure as it relates to SEO.
  • Knowledge of XML sitemap submissions, internal linking and keyword generation.
  • Track record of successfully implementing SEO strategies with a goal of increasing traffic and revenue.
  • Familiarity with social networking and bookmarking sites.
  • Attention to detail with a strong focus on analytics.
  • Strong project management and inter-departmental coordination skills.
  • Ability to execute on multiple projects while closely measuring the impact of each project and changing course when needed.

Company Profile:

As the fastest growing part of Macy’s Inc. business, macys.com is achieving record sales and broadening their workforce. With offices in New York and San Francisco, macys.com is the best of all worlds. The entrepreneurial thinking of a Web business complements the stability and support of a national brand. Creativity and ingenuity partner with business acumen and tech savvy to build a unique business poised for continued growth. Employees at macys.com have long term opportunities and are encouraged to utilize their Supervisors and Human Resources for cross-functional movement to further their careers. At macys.com they are committed to giving back to the community by partnering with local charitable organizations. By skillfully combining the power of the Internet with the best in retailing, macys.com is reaching new heights.

Salary range: Unknown
Closing date: Unknown
More info from: Macy’s Careers
Contact: Send resumes via online form to: Macy’s Careers

For more search industry jobs, or to post a vacancy, visit Search Engine College Jobs Board.

Share this post with others

Q and A: Does Ask.com Accept XML Sitemaps?

QuestionHi Kalena

I have uploaded my XML sitemap to Google, Yahoo and more recently Bing, thanks to your blog post about the Bing Webmaster Center.

However, I’m wondering if Ask.com accept XML sitemaps and if so, how do I upload mine to Ask?

thanks
Georgia

————————————–

Hello Georgia

Yes, Ask.com DO support XML Sitemap submissions. Here’s a blurb about it from their Webmaster Help area:

“Yes, Ask.com supports the open-format Sitemaps protocol. Once you have prepared a sitemap for your site, add the sitemap auto-discovery directive to robots.txt, or submit the sitemap file directly to us via the ping URL”

The ping URL is as follows:

http://submissions.ask.com/ping?sitemap=http%3A//www.yoursite.com/sitemap.xml

To add your sitemap to your robots.txt file, simply include this line:

Sitemap: http://www.yoursite.com/sitemap.xml

Actually it’s not just Ask that supports the addition of sitemaps in robots.txt. Did you know that both Google and Yahoo also support that method of sitemap delivery?

You can either submit your sitemap via the search engine’s appropriate submission interface (e.g. Google Webmaster Tools, Yahoo Site Explorer, Bing Webmaster Center) or specify your sitemap location in your robots.txt file as per the above instructions.

Share this post with others

Q and A: Why doesn’t Google index my entire site?

Question

Dear Kalena…

I have been on the internet since 2006, I re-designed my site and for the past year it still has only indexed 16 pages out of 132.

Why doesn’t google index the entire site? I use a XML site map. I also wanted to know if leaving my old product pages up will harm my ratings. I have the site map setup to only index the new stuff and leave the old alone. I have also got the robots.txt file doing this as well. What should I do?

Jason

Hi Jason

I’ve taken a look at your site and I see a number of red flags:

  • Google hasn’t stored a cache of your home page. That’s weird. But maybe not so weird if you’ve stopped Google indexing your *old* pages.
  • I can’t find your robots.txt file. The location it should be in leads to a 404 page that contains WAY too many links to your product pages. The sheer number of links on that page and the excessive keyword repetition may have tripped a Googlebot filter. Google will be looking for your robots.txt file in the same location that I did.
  • Your XML sitemap doesn’t seem to contain links to all your pages. It should.
  • Your HTML code contains duplicate title tags. Not necessarily a problem for Google, but it’s still extraneous code.

Apart from those things, your comments above worry me. What do you mean by “old product pages”? Is the content still relevant? Do you still sell those products? If the answer is no to both, then remove them or 301 redirect them to replacement pages.

Why have you only set up your sitemap and robots.txt to index your new pages? No wonder Google hasn’t indexed your whole site. Googlebot was probably following links from your older pages and now it can’t. Your old pages contain links to your new ones right? So why would you deliberately sabotage the ability to have your new pages indexed? Assuming I’m understanding your actions correctly, any rankings and traffic you built up with your old pages have likely gone also.

Some general advice to fix the issues:

  • Run your site through the Spider Test to see how search engines index it.
  • Remove indexing restrictions in your robots-txt file and move it to the most logical place.
  • Add all your pages to your XML sitemap and change all the priority tags from 1  (sheesh!).
  • Open a Google Webmaster Tools account and verify your site. You’ll be able to see exactly how many pages of your site Google has indexed and when Googlebot last visited. If Google is having trouble indexing the site, you’ll learn about it and be given advice for how to fix it.
  • You’ve got a serious case of code bloat on your home page. The more code you have, the more potential indexing problems you risk. Shift all that excess layout code to a CSS file for Pete’s sake.
  • The number of outgoing links on your home page is extraordinary. Even Google says don’t put more than 100 links on a single page. You might want to heed that advice.
Share this post with others

Q and A: What salary should I expect as an SEO?

Question

Dear Kalena…

What salary would i have if i were to get hired as an SEO or SEM? on average hourly and annual

Cruiz

Hi Cruiz,

A great question – and one often asked by people just entering (or considering joining) the SEO community.  As you’ve probably anticipated, it’s not really possible to provide a definitive answer to this question, as the salary rates you could expect,  depend on a number of variable.  I’ve outlined below some of the most significant factors that are likely to influence SEO or SEM salaries :

  1. Location – you’ve not identified which part of the world you are from, but this can have a significant impact on Salary levels.  Salaries in the US and UK, are typically higher than those in Australia, which are usually higher again than those in India (which has a massive and thriving SEO industry by the way).  Hot Spots within a particular country are also likely to offer higher salaries that are based on the usual factors – such as cost of living, lifestyle, and competition.
  2. Organisation – whether you are working In House,  within a specialist SEO Agency or as a private Consultant , will also influence you salary.
    For In House SEOs, the size of the business, and their awareness/acceptance of the importance of SEO will influence what they are prepared to pay.  Some SMBs are not able (or willing) to justify  a full time SEO role, so Search Engine Optimisation might be seen as something that is done by the Web Developer or Marketer in their spare time.
    The salary for In House SEOs in large organisations (with SEO teams) is broadly comparable to that of the salary for an equivalent role within a specialist SEO Agency (although the Agency SEO is likely to have the opportunity to deal with a broader range of clients and experiences) .
    Salaries for private Consultants can vary dramatically – from the highest salaries for recognised SEO Gurus to the (probably) lowest hourly rates for relatively inexperienced start-up SEOs.

    [Editor Note: You might also want to review the salaries and jobs categories in this blog to get a good idea of the type of salaries that SEO/SEM staff can command. My article 11 Reasons Why You Should Consider a Job in Search Engine Marketing also lists some common salary ranges. Cheers, Kalena]

  3. Role – there  are many different types of roles and activities within the SEO Industry, some people focus on one particular role, others undertake the complete range of activities.  Typically the more experienced you get in a particular area, the more specialised you become, and the higher salary you can expect.  Types of roles include – Strategist, Consultant, Analyst, Researcher, Writer.
  4. Experience – I say experience here rather than qualification, because there is not currently an internationally  recognised  SEO qualification (although given the increasing awareness of the SEO industry – this may change in the future).  SEO Course’s such as those offered by Search Engine College are a fabulous way to gain an understanding of this field, and provide a valuable insight into SEO techniques, strategies and tips.  However, experience – dealing with customers in real world situations is probably the single best way to justify a higher salary.  Being able to demonstrate real success with high profile clients in competitive industries, proves your experience and abilities.
  5. Profile – the better you are at raising your profile in the industry, the higher salary you can expect.   A high profile is usually (but not always) a natural result of experience and confidence.  If you are outspoken in the industry – through blogging, involvement in forums, attendance and presentation at industry events, etc. your reputation will develop. If it is clear that you understand the industry and know what you are talking about; if you offer useful advice and innovative strategies; and if you can demonstrate your ability to achieve real results for your clients, you may be on your way to “SEO Guru” status gathering followers (and an increased salary) along the way.
  6. Supply and Demand – as in all things, supply and demand will influence the level of salary you can expect.  If you have few competitors for a particular role you are likely to be able to demand a higher salary – providing you have suitable experience.  Supply and demand changes from time to time and is influenced by many things including geographic location , unemployment rates, and the financial climate.
    In these days of financial uncertainty, with many businesses tightening up their budgets,  you might speculate that the demand for SEOs would decrease.  However, the reverse seems to be true.  Many SEOs are in fact  experiencing an increase in work levels, as business owners realise that they need to get smarter about how to develop their businesses and spend their marketing budget.

Rand Fishkin of SEOMoz wrote an excellent post on this topic ( see : SEO Salaries – How Much Should You Make) – however this was written in 2006 – and now, 3 years on, the annual salary figures are almost certainly higher.  (how about an update Rand?)

Search Engine Optimisation is a role requiring specialist knowledge and experience, and as such you should expect to achieve a higher salary than a more traditional web or marketing role.  Some of the factors outlined above are outside your control (unless you are willing to move to another part of the world for example), but one factor that you are able to influence is experience.  Getting some good basic SEO Training and undertaking some Search Engine Marketing Courses (through Search Engine College of course) , doing some Research, and gaining Experience (even if it is only on your own/friends websites initially) is the best way for you to improve you salary prospects.

Andy Henderson
WebConsulting Web Optimisation

Share this post with others