When the guys at Search Marketing Standard magazine emailed me asking for my feedback on a new venture they were about to launch, I was intrigued. Would I review the brand new site and provide my honest feedback? “Sure”, I said, “What’s the concept?”

“Well”, they said, “SEMCompare is a ratings and review site covering SEM agencies. The idea behind the site is to encourage clients to submit reviews on any of the SEM agencies they’ve worked with – good and bad.”

“Uh Oh”, I thought and the skepticism started to flow. After a few minutes of browsing the site, my skepticism turned to concern and my opinion of the site was confirmed: Trouble. With a capital T.

You see, back in 2001, Marketing Sherpa published the first issue of their Buyer’s Guide to SEO Firms. They’ve since changed the model, but the launch issue used a rating system where they gave the SEO companies (including my own) a score rating based on client reviews, search rankings achieved and their internal measure of SEO tactics used by the companies. This prompted some legal action by one or more firms reviewed within the Guide. To me, the SEMCompare site sounded like a very similar concept that could have the same disastrous outcome.

The problem with the concept of rating SEM firms is that there are no published search industry standards that SEM agencies are required to meet and no formal accreditation body imposing such standards. There are certainly de facto or unspoken industry standards that are widely accepted as a benchmark, but no governing body yet exists to regulate the implementation of such standards. As such, a site or service that imposes a rating system is by default claiming a standards yardstick by which all SEM firms should be measured – dangerous territory indeed!

Giving consumers the power to anonymously rate SEM firms opens up another can of worms: how to keep the system free of corruption by SEM firms and disgruntled consumers alike? I couldn’t believe my friends at SMS would open themselves up to so much potential grief, so I drafted some questions about the site and sent them off for a response.

Here are my questions and the answers I received from Matt Alland, Business Development Director for SEMCompare:

1) What prompted you to launch such a site?

Thus far, choosing an SEM company has been a confusing process, especially for those who are completely new to search engine marketing. There really hasn’t been a clear way for consumers to accurately review companies. While there have been some basic independent reviews of select SEM firms, there hasn’t been a more widely accepted industry standard. Having talked to a lot of people looking for a better way to find and compare SEM companies, we decided to create SEMCompare, and offer a new much needed service to this industry.

2) How is the site any different to the Better Business Bureau or Consumer Advocacy groups?

In many ways, we function in a similar fashion, but not precisely. While users submitting reviews are able to rank companies, we’re less concerned about the precise numbers, and put more emphasis on the reviews themselves – why a company did or did not work for a certain business or individual. Our goal is to be a launching pad for people starting their search for SEM companies, and a point of research. I don’t think our focus is to try and become a “scam alert” type site – we encourage users to do additional research on their own and look for those warning signs. Rather we hope people will use SEMCompare to find companies that work for their specific needs.

3) How do you prevent conflict of interest issues and manipulation of the system by SEM agencies posing as consumers?

We do a variety of things internally. I can’t mention everything, but we definitely do IP tracking, cookie tracking, and keep close tabs on email addresses, names, and trends in reviews. If we see suspicious spikes or unusual activity we make note of it and may deny posting those highly questionable reviews as a result. We will sometimes contact the reviewer themselves, and we always read over every review before it goes live to the site.

4) How do you protect SEM agencies from consumers who may have an axe to grind with a particular SEM staff member?

Once again, that goes into a lot of the review tracking techniques we use. If certain trends seem suspicious or unusual we take that into account. These are the same kinds of challenges any review site must go through, so luckily there are a lot of resources at our disposal at this point in time to combat deceptive or faulty reviews. And as a final note, we always have a flagging opportunity so people are able to dispute specific reviews deemed unfair. However, the site does not protect companies from bad reviews and it is not our job to take those down just because a company disagrees with it. Our main job is to prevent companies from posting reviews themselves and from reckless individuals looking to disrupt competition with floods of bad reviews.

5) Marketing Sherpa published a Buyer’s Guide to SEM Firms that rated SEM firms and this prompted some legal action by one or more of the firms within the Guide. How do you resolve to avoid similar legal problems?

We don’t review companies at SEMCompare, but rather provide the platform for people to do so. While we try to weed out as many suspicious reviews as we can, at the end of the day we can’t really endorse or condone reviews on the site. We would expect that people will use their own sensibilities, look into the reviews and follow up with companies themselves.

Despite a number of challenges and imperfections that are bound to exist for this type of site, we’re excited to be the first to really take on this concept fully. Ultimately, the benefits of having an industry accepted review site outweigh the challenges that exist. Review sites are pivotal in so many industries now: automobiles, cameras, consumer electronics, as well as industries like web hosting and web design. SEM is certainly a unique industry, but the same core elements of a review site remain the same, and people’s concerns are no different than with any review site that has existed before us.

Ultimately, we’re excited about the opportunity SEMCompare will be able to offer to consumers, giving people a new means to share their experiences and learn more about the differences between companies instead of randomly selecting providers. This is something the industry thus far has not had, and a service like this offers a chance to simplify and standardize the process of comparing the hundreds of companies out there.

Something from the above exchange triggered a huge red flag for me: “At the end of the day we can’t really endorse or condone reviews on the site.” What? So you aren’t endorsing your own site content? Then why bother creating a site around such content at all? Unless the site’s primary goal is to sell magazine advertising to SEM firms with reputations tarnished by bad reviews? Things that make you go hmmmm.

SEMCompare claim to have already received hundreds of reviews with over 150 SEM companies evaluated. Some of the companies reviewed have an already litigious history, which is alarming. Even more concerning is that a few of the negative reviews are for companies that I’m very familiar with and that have excellent and ethical reputations in the industry, making the true motivations of reviewers seem highly suspicious. I can’t help thinking that many of the reviews I read would make legal departments trigger happy.

Here are some sample negative reviews taken from the site yesterday:

“Paid over $19,000 over 9 months and our natural traffic declined by 30%. They setup duplicate pages (static versions of our dynamic content) and we were penalized by Google. “

“They never (not once) delivered on expectations, and most of their deliverables were sub par. I came to find out late in our relationship that they outsourced the majority of work that our company outsourced to them…thus they were nothing more than a parasitic middleman.”

“The only thing they were good at was stealing my money… I think they even got me banned from Google for three months for using black hat techniques.”

“Their sales team was absolutely relentless in trying to get more money from us. Even when I was complaining that deadlines were being missed, they would try to sell me something. There is no way on earth that I’d recommend this company to anyone.”

“In summary, [company name removed] misrepresented their expertise, their technology, and showed no integrity as they breached their contract, refunding nothing to us. I strongly warn anyone and everyone to avoid spending any resources with this organization. Take your money and burn it. You’ll get more satisfaction from that experience than with working with [company name removed] .”

All I can say is OUCH. Maybe the guys will prove me wrong, but it’s my prediction that this time next year the headaches caused by SEMCompare will see it self-implode.