Q and A: Why is Google having trouble indexing our site?

QuestionHello Kalena

One of the sites we manage has a problem.

The homepage at [URL removed] is not getting indexed anymore by Google. The site was made using Sitefinity 3.7 and the hosting is provided by Rackspace. Something similar already happened two times in the past which we resolved using the option “index this page” on the page generated by Sitefinity (1st time) and by re-creating the XML sitemap and linking it directly to Google Webmaster tools (2nd time).

This time we can’t seem to find the reason. We checked if the end-user that works as the back-end has made any changes or if there was any notification from Google Webmaster Tools reports but nothing came up. Here are some more technical details:

1) The site homepage is [URL removed]. But the site root is [URL removed] which is an empty page with a redirect to the home page using a 301 redirect.

2) In Google Webmaster Tools we set up 2 Sitemaps:

  • The first at [URL removed] is indexing the Top pages of the Home page (static)
  • The second [URL removed] gets populated with the pages content generated by Sitefinity (dynamic)

3) Also, from the back-end options, a metatag ROBOTS was set at page level for the top pages, as Google suggests.

4) Google reports 5 blocked URLs when crawling our robots.txt with the message: “Google tried to crawl these URLs in the last 90 days, but was blocked by robots.txt”. This seems suspicious, because I can’t seem to understand what could be blocking it, the robot is pretty simple and not restrictive.

Could you give us an hand? I’ve left a generous donation for your coffee fund.

Thanks!
Jim

————————————–

Hi Jim

First up, thanks for the caffeine donation :-)

As for your problem, oh boy. You’ve got a few different issues going on, so let me address each of them separately:

1) Your XML sitemaps are missing contextual data specified by the Sitemaps protocol. In particular, your < loc > child entries per URL are messed up. I’m surprised this hasn’t generated an error in Webmaster Tools, but I’m pretty sure it would be confusing Googlebot. Go check your sitemaps against the protocol and re-generate them if necessary. Maybe use one of the XML generator tools recommended by Google. Personally, I like XML Sitemaps (yes that’s my affiliate link).

Also, why 2 separate sitemaps for HTML pages? I can understand having separate ones for RSS feeds or structured data stuff, but your standard site pages should all be listed in the one file so you can better manage the content and keep track of indexing history in Webmaster Tools.

2) Your robots.txt file is blocking a number of pages that you have listed in your XML sitemap. So on the one hand you’re telling Google to index pages within a certain directory, but on the other, you’re telling Google they are not allowed to access that directory. This is what the error message is about. You’ve also got conflicting instructions on some of your pages in terms of robots meta tags vs. robots.txt.

3) The 301 redirect on your root directory is your major problem. In fact, that empty landing page is your major problem. Why do you need it? You don’t use Flash and it doesn’t appear to have an IP sniffer for geo-location purposes so I can’t understand why you wouldn’t just put your home page content at the root level and let search engines index it as expected.

The way you have it set up right now is essentially telling Google that you have moved all your content to a new location, when you really haven’t. It’s adding another step to the indexing process and you are also shooting yourself in the foot as every 301 contributes to some lost PageRank. Google clearly doesn’t like the set up or isn’t processing it for some reason. There also appear to be several hundred 301s in place for other pages, so I’m not sure what that’s about. I don’t have access to your .htaccess file, but I can imagine it reads like a book!

4) Unless you specifically need a robots meta tag for a particular page scenario, I would avoid using them on every page. You can achieve the same results with your robots.txt file and it’s easier to manage robot instructions in one location rather than having to edit page by page – avoiding conflicting issues as you have now.

Apart from the obvious issues mentioned above – have you considered switching away from Sitefinity and over to WordPress? I’ve struggled optimizing Sitefinity sites for years – it’s a powerful CMS but it was never built with search engines in mind and always requires clunky hacks to get content optimized. Plus that’s a really out-dated version of Sitefinity.

Given the other issues, it might be time for a total site rebuild?

Best of luck

——————————————————————–

Like to learn SEO with a view to starting your own business? Access your Free SEO Lessons. No catch!

 

Spread the joy!

Q and A: How should I manage the client expectations of my SEO efforts?

QuestionHello Kalena

I saw your recent post about how SEO freelancers can generate new business for themselves and I wondered if you could answer a related question please.

I am currently the in-house SEO for a real estate chain, but I am quite keen to start my own consulting business part time. The major reason I haven’t done so already is that I wouldn’t know what to tell clients to expect when it comes to the projected results of my optimization efforts. I mean I know how long it took me to optimize these sites for my boss and it was many weeks before they even started to show up in the first few pages of Google.

But isn’t every site different? Doesn’t it depend entirely on what Google thinks of each site and the indexed content? How can I get potential SEO clients interested in my services if I don’t know what type of results they can expect and when? How do you handle this type of situation and what advice can you give me?

Thanks so much Kalena

Kelly

————————————–

Hi Kelly

Thanks for reaching out – and your question is a good one. Search engine optimization is a unique service in that you rarely see the results of your efforts right away. It can take days, weeks or even months for Google and the other search engines to index and re-rank your newly optimized pages. It’s vital that you explain this to your clients up front. Most of your clients will be expecting results quickly and it’s your job to extinguish that myth.

The key to managing client expectations is to be as transparent as possible about your process and educate them in the process. [My Udemy course – How to Earn a Full Time Income as a Part Time SEO Consultant – can give you more tips about this]

It always astounds me when I hear about SEO companies who don’t actually explain to clients what they do. Some SEOs don’t even inform the clients what changes they are going to make to their web sites! Sometimes I’ll take on a new client who has worked with another SEO in the past and when I ask them “what changes did the SEO make to your site? What SEO tasks did they perform?” they will have NO CLUE. Or they’ve been told that the process is “secret” or “proprietory”. What a load of bollocks. How can a client possibly understand the value of what you do if you don’t explain to them what you actually do?

You want to know why some SEOs refuse to reveal their process to clients? Because they don’t actually know how to optimize a web site. It’s true. Many of the large so called *SEO* firms you see advertising and cold calling these days claim to be selling SEO services, but they DON’T PERFORM ANY SEO WHATSOEVER. What they are actually selling is Google AdWords. They make grand claims to help customers get their sites ranking high in Google using *proprietory SEO methods*, when what they’re actually doing is buying up masses of cheap keywords on AdWords via bulk accounts and displaying ads pointing to their customer sites. Yes, the ads might appear above the organic search results from time to time and deliver traffic, but the click costs usually increase month on month. As soon as that customer stops paying, the traffic stops coming. And what are the clients left with? The same unoptimized site they started with, no more traffic and the opinion that SEO simply doesn’t work.

No wonder SEO has such a bad name!

Some SEOs I’ve talked to are afraid that the client will take that knowledge and use it to perform their own SEO or to train staff in-house to take over the SEO process, putting them out of a job. You know what I say to that? Fantastic. The best SEOs work themselves out of a role, in my opinion.

If you can educate your client to the point where they understand the importance of SEO and the value to their business, you have done the very best job possible and I guarantee you they will be singing your praises for life. Because you will have turned them from a customer into a fan, you will probably get more business from an ex-client you’ve educated than you would have from that same client if you had kept them as a long term customer. How? Via referrals and word of mouth. A passionate testimonial from a happy customer can win many, many clients. You can use that testimonial on your web site, business cards and marketing material. You can use that client as a verbal referee if future clients want to talk to a previous customer.

So don’t be afraid to educate your clients during every single step of the SEO process. Make sure they track their own progress via Google Analytics. Show them what you’re doing and manage their expectations by explaining to them that you don’t know exactly when their site rankings and traffic will start to climb and you may have to tweak things along the way.

Simply be honest with them and they will ultimately respect you more and thank you for it.

——————————————————————–

Like to learn how to earn a full time income as a part time SEO consultant? Check out my Udemy course on the subject.

 

Spread the joy!

Q and A: What are some ideas for SEO lead generation?

QuestionHello Kalena

I’ve recently struck out on my own as a search engine optimization freelancer and I’m finding it really difficult to get new leads.

I can’t afford to pay per click the amounts that the big players are paying on Google AdWords and I’m not confident enough with Bing Ads to try that. I did experiment with some Facebook advertising, but the response wasn’t great so I don’t think it’s really where my market is.

I know you recommended in the past to try some freelancing sites, but I was on Elance for a month and was out-bid by a lot of SEO crews based in South East Asia, who seem to low-ball everybody and win most of the projects. I know my work is higher quality than what they are offering, but it doesn’t seem to matter to the client.

Any creative ideas for how I can generate new leads and find genuine SEO clients who are willing to pay fairly for quality?

Thanks
Anthony

————————————–

Hi Anthony

I feel your pain, I truly do. Having my own SEO consultancy, I am fully aware just how challenging it can be to secure a steady flow of new business and compete with rivals after the same clients.

Here are my tips for gaining new SEO leads:

1) Install MySiteAuditor on Your Main Sales PageMy Site Auditor is a custom-built lead generation tool for SEO professionals and freelancers. You simply install the code somewhere prominent on your site and it allows visitors to generate a one minute SEO audit for a URL of their choice, for a particular keyword or phrase. Visitors need to enter their email address in order to generate the audit, creating automatic leads to follow up.

You can see it in action over at Site Audit NZ. There is a 10 day free trial and two low cost monthly subscription plans, depending on whether you want to embed the tool, white-label the audits and have leads emailed to you.  We use it and it’s a great way to increase SEO sales and leads by simply embedding a useful tool on your site.

2) Utilize Keyword Alert and Social Monitoring Tools – Services like Google Alerts and the fairly new Talkwalker Alerts allow you to track mentions of your target keywords or search phrases across the web without lifting a finger. You simply choose the search terms you want to be alerted about and then the service will email you whenever it finds a new mention of those terms in Google search results (for Google Alerts) or in news sites, blogs and discussion boards across the entire web (for Talkwalker). Klout, Social Mention and HootSuite all offer a similar service to track real time mentions on social sites. Raven Tools used to offer the same, but have sadly discontinued it.

What this means is that you can pin-point potential leads by what they are searching for / talking about. So, for example, we are always looking for new students to join Search Engine College and a great way to find potentials is to see who is discussing topics like “I want to learn SEO” or “Teach myself SEO”.  We can then (tactfully) approach those persons directly on the sites where the discussion was found to see if our courses might be a good fit for them.

You can do the same thing by creating alerts for phrases like “Need help with SEO” or “SEO my site”. Just make sure you follow up fast though, because discussions can become stale quickly, especially on social networks. Also be super careful that you take a helpful approach rather than dive in with a sales pitch, or you’re likely to frighten off any genuine potentials.

3) Give Bing Ads a Whirl – You can teach yourself how to use it fairly quickly and the CPC is much, much cheaper than Google AdWords in my experience. If you want to contact me directly, I can even send you a USD100 voucher for ad credit (as I’m a Bing Partner).

4) Give Stuff Away – Got an ebook or a white-paper lying around? Turn it into a free give-away on your site, in exchange for a visitor’s email address. Companies like HubSpot do this type of incentive-based lead generation exceptionally well. With a little forethought, any client case study or interesting SEO research can be turned into a downloadable incentive to part with an email address. Just make sure you have their permission to contact these visitors with your marketing messages later on.

These tactics may not work for you, but they consistently work for me, so I’m confident you will get some good results.

Best of luck :-)

——————————————————————–

Like to learn SEO with a view to starting your own business? Access your Free SEO Lessons. No catch!

 

Spread the joy!

Q and A: Is it ok to target very specific keywords for SEO purposes?

QuestionHello Kalena

Just doing keyword research on a site [URL removed] and I have two questions:

1.  There is no option to add any text on the homepage.  The site is built on bigcommerce platform, which I can imagine quite a few sites are on this type of platform now.  To optimise it, do I just use the product pages that do allow text?

2.  The keywords I chose are all searched less than 10 times per month, but I cannot see how I can use any other keyword.  The product is obviously not a very highly searched for product, but is so specific, that I cannot use general keywords like “baby changing table”.   What do you suggest here?

thanks
Steph

————————————–

Hi Steph

You don’t make it clear whether this is a client site of yours or your own site.

1) I’m not familiar with the ecommerce platform you mention, but if this was my client, I’d be encouraging them to ditch it and start again with a more SEO-friendly ecommerce platform, or even a WordPress site with an ecommerce plugin.

Any CMS (Content Management System) that doesn’t allow you to add or edit text to pages, especially the home page, is going to restrict the ROI of a business.

If your client is unwilling to embrace recommendations to start over, you can only work with what you’ve got. So use our free SEO lessons and the advice in this blog to optimise what you can (product pages, meta tags, title tags, alt img tags, filenames etc.)

2) When it comes to keyword research, having a niche market is actually a positive thing. You’re generally not competing with a huge number of other sites, so when you target more specific keyword phrases, you have less competition from other websites trying to optimise for the same keywords. Don’t be afraid of targeting very specific keywords and phrases, provided they accurately describe the product. Sure, you may get less traffic by optimising for specific keywords, but the traffic you DO get will be highly qualified and more likely to convert.

Also think about whether you can optimise for location-based qualifying keywords. For example, does the company export their products outside New Zealand? If not, then it might make sense to target phrases like “baby change tables New Zealand”. Is their product of higher quality than other products? Then use *high quality* in your target phrases and web site text.

Ask your colleagues, family and friends how they would search for the products and add these phrases to your keyword seed list for targeting purposes. Run a short term AdWords campaign and look closely at the number of impressions each keyword you bid on receives. This will give you a more accurate estimate of how many times the keyword is searched in your target market and help you narrow down your selection.

Hope this helps!

——————————————————————–

Like to get geeky and teach yourself SEO? Access your Free SEO Lessons

 

Spread the joy!

Q and A: Should I source backlinks from a link merchant?

QuestionHi Kalena

I’m working through the link building course at Search Engine College, but I’m a tad unsure where to source links for my website. I know I can get them from industry related websites, but think this might be a slow process. Is it therefore deemed appropriate for me to source links from a service such as linksmanagement.com? If so, can you please answer the following questions:

1. How many links should I acquire on a weekly/monthly basis?

2. Can I focus on 1 page of my website at a time when building links, or should I spread them evenly on various pages of my website say 3-4 pages at a time?

3. Should link building be an ongoing process, or can I stop when I’ve achieved the ranking I desire….and we all know what position that is! :-)

If linksmanagement.com is not a source you would recommend me to use, can you recommend another please?

Kind Regards
Alistair

————————————–

Hi Alistair

Links should be acquired organically – at a moderate pace. I recommend adding no more than five new links a week to a site. Sites that acquire large blocks of links in a short space of time are more likely to attract attention from Google’s anti-spam team.

Whatever you do, DO NOT use a link selling site such as LinksManagement to buy backlinks. Links must be earned, not bought. The selling or purchasing of backlinks is in direct violation of the Google Webmaster Guidelines and could earn your site a ranking penalty or removal from Google’s index altogether.

Instead, I recommend that you use Raven Tools or another all-in-one SEO tool-kit with which to manage your link building efforts. These suites of tools generally enable you to research, find, contact and track link partners all in the one location. Raven’s link research tools in particular are brilliant for finding potential link opportunities and keeping track of who has linked to you.

We are still editing the remainder of the Link Building course, so you’ll probably find that (when published) the remaining lessons will answer your questions. But in the meantime, can I suggest that you review the Link Building lessons within the SEO101 and SEO201 courses? Also look at the recommended reading and resources for those lessons. They contain a wealth of information about link building.

Finally, take a look through my previous blog posts about link building as they should give you some inspiration about where to find new link partners.

Hope this helps!

——————————————————————–

Like to learn more about SEO and link building methods? Access your Free SEO Lessons. No catch!

 

Spread the joy!