Q and A: Why is Google having trouble indexing our site?

QuestionHello Kalena

One of the sites we manage has a problem.

The homepage at [URL removed] is not getting indexed anymore by Google. The site was made using Sitefinity 3.7 and the hosting is provided by Rackspace. Something similar already happened two times in the past which we resolved using the option “index this page” on the page generated by Sitefinity (1st time) and by re-creating the XML sitemap and linking it directly to Google Webmaster tools (2nd time).

This time we can’t seem to find the reason. We checked if the end-user that works as the back-end has made any changes or if there was any notification from Google Webmaster Tools reports but nothing came up. Here are some more technical details:

1) The site homepage is [URL removed]. But the site root is [URL removed] which is an empty page with a redirect to the home page using a 301 redirect.

2) In Google Webmaster Tools we set up 2 Sitemaps:

  • The first at [URL removed] is indexing the Top pages of the Home page (static)
  • The second [URL removed] gets populated with the pages content generated by Sitefinity (dynamic)

3) Also, from the back-end options, a metatag ROBOTS was set at page level for the top pages, as Google suggests.

4) Google reports 5 blocked URLs when crawling our robots.txt with the message: “Google tried to crawl these URLs in the last 90 days, but was blocked by robots.txt”. This seems suspicious, because I can’t seem to understand what could be blocking it, the robot is pretty simple and not restrictive.

Could you give us an hand? I’ve left a generous donation for your coffee fund.

Thanks!
Jim

————————————–

Hi Jim

First up, thanks for the caffeine donation :-)

As for your problem, oh boy. You’ve got a few different issues going on, so let me address each of them separately:

1) Your XML sitemaps are missing contextual data specified by the Sitemaps protocol. In particular, your < loc > child entries per URL are messed up. I’m surprised this hasn’t generated an error in Webmaster Tools, but I’m pretty sure it would be confusing Googlebot. Go check your sitemaps against the protocol and re-generate them if necessary. Maybe use one of the XML generator tools recommended by Google. Personally, I like XML Sitemaps (yes that’s my affiliate link).

Also, why 2 separate sitemaps for HTML pages? I can understand having separate ones for RSS feeds or structured data stuff, but your standard site pages should all be listed in the one file so you can better manage the content and keep track of indexing history in Webmaster Tools.

2) Your robots.txt file is blocking a number of pages that you have listed in your XML sitemap. So on the one hand you’re telling Google to index pages within a certain directory, but on the other, you’re telling Google they are not allowed to access that directory. This is what the error message is about. You’ve also got conflicting instructions on some of your pages in terms of robots meta tags vs. robots.txt.

3) The 301 redirect on your root directory is your major problem. In fact, that empty landing page is your major problem. Why do you need it? You don’t use Flash and it doesn’t appear to have an IP sniffer for geo-location purposes so I can’t understand why you wouldn’t just put your home page content at the root level and let search engines index it as expected.

The way you have it set up right now is essentially telling Google that you have moved all your content to a new location, when you really haven’t. It’s adding another step to the indexing process and you are also shooting yourself in the foot as every 301 contributes to some lost PageRank. Google clearly doesn’t like the set up or isn’t processing it for some reason. There also appear to be several hundred 301s in place for other pages, so I’m not sure what that’s about. I don’t have access to your .htaccess file, but I can imagine it reads like a book!

4) Unless you specifically need a robots meta tag for a particular page scenario, I would avoid using them on every page. You can achieve the same results with your robots.txt file and it’s easier to manage robot instructions in one location rather than having to edit page by page – avoiding conflicting issues as you have now.

Apart from the obvious issues mentioned above – have you considered switching away from Sitefinity and over to WordPress? I’ve struggled optimizing Sitefinity sites for years – it’s a powerful CMS but it was never built with search engines in mind and always requires clunky hacks to get content optimized. Plus that’s a really out-dated version of Sitefinity.

Given the other issues, it might be time for a total site rebuild?

Best of luck

——————————————————————–

Like to learn SEO with a view to starting your own business? Access your Free SEO Lessons. No catch!

 

Spread the joy!

The 2015 Periodic Table of SEO Success Factors

Table of SEO Success FactorsEarlier this month, the team over at Search Engine Land updated their brilliant Periodic Table of SEO Success Factors.

Now in it’s 3rd edition, the table is a fantastic SEO resource and one of the few items on my Ubuntu desktop that gets regular eyeball attention. Content is divided between on-page and off-page factors and clearly color-coded to make it visually intuitive, with relevancy weight ranging from -3 to +3.

The new edition references new factors of SEO importance including vertical search, Direct Answers and HTTPS, with mobile friendliness and structured data acquiring a relevancy weight increase in line with recent Google updates.

The idea behind the table is to highlight tasks within the SEO process and to act as a visual reminder about what is most important and what areas to focus on for clients.

Danny Sullivan describes the goal and philosophy of the table:

“Our goal with the Periodic Table Of SEO is to help publishers focus on the fundamentals needed to achieve success with search engine optimization. This means it’s not about trying to list all 200 Google ranking factors or detail Google’s 10,000 sub-factors. It’s not about trying to advise if keywords you want to rank for should go at the beginning of an HTML title tag or the end. It’s not about whether or not Facebook Likes are counted for ranking boosts.

Instead, the table is designed to broadly guide those new to or experienced with SEO into general areas of importance. Title tags are generally important. Think about making sure they’re descriptive. Social sharing is often generally seen as good for SEO. Aim for social shares, without worrying about the specific network.”

While not exactly a cheat-sheet, my SEO students at Search Engine College tell me it is their favorite resource for assignment preparation, so that’s a pretty good endorsement.

The Table can be downloaded as a PDF in large or condensed format, or you can grab the code to embed the infographic directly into your web site.

Spread the joy!

Revised link building course now available at Search Engine College

SEC-laptop-2014-colourAfter several months of revision, I am pleased to announce that we have just re-launched our Link Building 101 course at Search Engine College.

The course content has been completely updated with new material and videos to reflect Google’s revised stance on acceptable link building tactics. This 10 lesson course also takes into account the impact of recent tweaks to Google’s Panda, Penguin and Hummingbird algorithms.

The course is now available to all students with a paid subscription. We have also added new assessment items including review quizzes, 4 tutor-graded assignments and a final exam available exclusively to subscribers who want to upgrade their subscription to Certification.

Just as a reminder, Certification is for current subscribers who want to benefit from tutor supervision, complete set assessments to reach our knowledge benchmark and receive formal, industry-recognized certification following completion. You can upgrade your subscription to Certification at any point for any course.

Hope to see some of you in our new link building class.

Spread the joy!

Q and A: How should I manage the client expectations of my SEO efforts?

QuestionHello Kalena

I saw your recent post about how SEO freelancers can generate new business for themselves and I wondered if you could answer a related question please.

I am currently the in-house SEO for a real estate chain, but I am quite keen to start my own consulting business part time. The major reason I haven’t done so already is that I wouldn’t know what to tell clients to expect when it comes to the projected results of my optimization efforts. I mean I know how long it took me to optimize these sites for my boss and it was many weeks before they even started to show up in the first few pages of Google.

But isn’t every site different? Doesn’t it depend entirely on what Google thinks of each site and the indexed content? How can I get potential SEO clients interested in my services if I don’t know what type of results they can expect and when? How do you handle this type of situation and what advice can you give me?

Thanks so much Kalena

Kelly

————————————–

Hi Kelly

Thanks for reaching out – and your question is a good one. Search engine optimization is a unique service in that you rarely see the results of your efforts right away. It can take days, weeks or even months for Google and the other search engines to index and re-rank your newly optimized pages. It’s vital that you explain this to your clients up front. Most of your clients will be expecting results quickly and it’s your job to extinguish that myth.

The key to managing client expectations is to be as transparent as possible about your process and educate them in the process. [My Udemy course – How to Earn a Full Time Income as a Part Time SEO Consultant – can give you more tips about this]

It always astounds me when I hear about SEO companies who don’t actually explain to clients what they do. Some SEOs don’t even inform the clients what changes they are going to make to their web sites! Sometimes I’ll take on a new client who has worked with another SEO in the past and when I ask them “what changes did the SEO make to your site? What SEO tasks did they perform?” they will have NO CLUE. Or they’ve been told that the process is “secret” or “proprietory”. What a load of bollocks. How can a client possibly understand the value of what you do if you don’t explain to them what you actually do?

You want to know why some SEOs refuse to reveal their process to clients? Because they don’t actually know how to optimize a web site. It’s true. Many of the large so called *SEO* firms you see advertising and cold calling these days claim to be selling SEO services, but they DON’T PERFORM ANY SEO WHATSOEVER. What they are actually selling is Google AdWords. They make grand claims to help customers get their sites ranking high in Google using *proprietory SEO methods*, when what they’re actually doing is buying up masses of cheap keywords on AdWords via bulk accounts and displaying ads pointing to their customer sites. Yes, the ads might appear above the organic search results from time to time and deliver traffic, but the click costs usually increase month on month. As soon as that customer stops paying, the traffic stops coming. And what are the clients left with? The same unoptimized site they started with, no more traffic and the opinion that SEO simply doesn’t work.

No wonder SEO has such a bad name!

Some SEOs I’ve talked to are afraid that the client will take that knowledge and use it to perform their own SEO or to train staff in-house to take over the SEO process, putting them out of a job. You know what I say to that? Fantastic. The best SEOs work themselves out of a role, in my opinion.

If you can educate your client to the point where they understand the importance of SEO and the value to their business, you have done the very best job possible and I guarantee you they will be singing your praises for life. Because you will have turned them from a customer into a fan, you will probably get more business from an ex-client you’ve educated than you would have from that same client if you had kept them as a long term customer. How? Via referrals and word of mouth. A passionate testimonial from a happy customer can win many, many clients. You can use that testimonial on your web site, business cards and marketing material. You can use that client as a verbal referee if future clients want to talk to a previous customer.

So don’t be afraid to educate your clients during every single step of the SEO process. Make sure they track their own progress via Google Analytics. Show them what you’re doing and manage their expectations by explaining to them that you don’t know exactly when their site rankings and traffic will start to climb and you may have to tweak things along the way.

Simply be honest with them and they will ultimately respect you more and thank you for it.

——————————————————————–

Like to learn how to earn a full time income as a part time SEO consultant? Check out my Udemy course on the subject.

 

Spread the joy!

Q and A: What are some ideas for SEO lead generation?

QuestionHello Kalena

I’ve recently struck out on my own as a search engine optimization freelancer and I’m finding it really difficult to get new leads.

I can’t afford to pay per click the amounts that the big players are paying on Google AdWords and I’m not confident enough with Bing Ads to try that. I did experiment with some Facebook advertising, but the response wasn’t great so I don’t think it’s really where my market is.

I know you recommended in the past to try some freelancing sites, but I was on Elance for a month and was out-bid by a lot of SEO crews based in South East Asia, who seem to low-ball everybody and win most of the projects. I know my work is higher quality than what they are offering, but it doesn’t seem to matter to the client.

Any creative ideas for how I can generate new leads and find genuine SEO clients who are willing to pay fairly for quality?

Thanks
Anthony

————————————–

Hi Anthony

I feel your pain, I truly do. Having my own SEO consultancy, I am fully aware just how challenging it can be to secure a steady flow of new business and compete with rivals after the same clients.

Here are my tips for gaining new SEO leads:

1) Install MySiteAuditor on Your Main Sales PageMy Site Auditor is a custom-built lead generation tool for SEO professionals and freelancers. You simply install the code somewhere prominent on your site and it allows visitors to generate a one minute SEO audit for a URL of their choice, for a particular keyword or phrase. Visitors need to enter their email address in order to generate the audit, creating automatic leads to follow up.

You can see it in action over at Site Audit NZ. There is a 10 day free trial and two low cost monthly subscription plans, depending on whether you want to embed the tool, white-label the audits and have leads emailed to you.  We use it and it’s a great way to increase SEO sales and leads by simply embedding a useful tool on your site.

2) Utilize Keyword Alert and Social Monitoring Tools – Services like Google Alerts and the fairly new Talkwalker Alerts allow you to track mentions of your target keywords or search phrases across the web without lifting a finger. You simply choose the search terms you want to be alerted about and then the service will email you whenever it finds a new mention of those terms in Google search results (for Google Alerts) or in news sites, blogs and discussion boards across the entire web (for Talkwalker). Klout, Social Mention and HootSuite all offer a similar service to track real time mentions on social sites. Raven Tools used to offer the same, but have sadly discontinued it.

What this means is that you can pin-point potential leads by what they are searching for / talking about. So, for example, we are always looking for new students to join Search Engine College and a great way to find potentials is to see who is discussing topics like “I want to learn SEO” or “Teach myself SEO”.  We can then (tactfully) approach those persons directly on the sites where the discussion was found to see if our courses might be a good fit for them.

You can do the same thing by creating alerts for phrases like “Need help with SEO” or “SEO my site”. Just make sure you follow up fast though, because discussions can become stale quickly, especially on social networks. Also be super careful that you take a helpful approach rather than dive in with a sales pitch, or you’re likely to frighten off any genuine potentials.

3) Give Bing Ads a Whirl – You can teach yourself how to use it fairly quickly and the CPC is much, much cheaper than Google AdWords in my experience. If you want to contact me directly, I can even send you a USD100 voucher for ad credit (as I’m a Bing Partner).

4) Give Stuff Away – Got an ebook or a white-paper lying around? Turn it into a free give-away on your site, in exchange for a visitor’s email address. Companies like HubSpot do this type of incentive-based lead generation exceptionally well. With a little forethought, any client case study or interesting SEO research can be turned into a downloadable incentive to part with an email address. Just make sure you have their permission to contact these visitors with your marketing messages later on.

These tactics may not work for you, but they consistently work for me, so I’m confident you will get some good results.

Best of luck :-)

——————————————————————–

Like to learn SEO with a view to starting your own business? Access your Free SEO Lessons. No catch!

 

Spread the joy!