Webstock 09: A Mashup Case Study

Live blogging a Mashup Case Study presentation at Webstock 09 by EveryBlock.com founder Adrian Holovaty:

A mini Google Maps app was actually developed before Google did it by a NZ developer using the Google API. Got Adrian thinking and he developed ChicagoCrime.org which then morphed into EveryBlock.com.

ChicagoCrime.org was a mashup of crime data put together from the Chicago Police Dept’s web site and published together with Google Maps. Searchable via route, area, block or neighbourhood locations (e.g. barber shops, restaurants), even via mobster activity. Google’s rollout of Street View added even more data.

They then added Flickr data, user material etc. This eventually became EveryBlock.com, a news feed for your block. You can go to the site, select a city and choose your neighbourhood to see all news relating to your neighbourhood that has been written about in local media. The goal is to find the news that is very specific to you that may be buried on the major news sites by national or international media stories.

The data includes restaurant reviews, product launches, feel-good stories, lost and found stories etc. You can search via trends e.g. all restaurants with particular food safety violations e.g. vermin. You can search via RSS feeds in your neighbourhood, real estate listings, press releases etc.

How Does it Work?

Complex technology is used to pull all the data together. The site uses natural language processing including datafeeds from portals, search engines, media agencies, APIs like Flickr’s, geocoding API, Craiglist, Yelp, Valpak, Trulia. Government resources.

How to Best Make a Successful Mashup:

Lesson 1: Take advantage of existing data

Lesson 2: Sort by date - this adds value

Lesson 3: The more effort it takes, the higher the reward. Unlike Google. Google likes convenient, centralized, USA building-permits databases. Anything else is too much work. Be the anything else.

Lesson 4: Governments are opening up. Be nice and appeal to civil servants (and librarians!). Plot cities/agencies against each other. Governments shouldn’t build interfaces into their data, they should offer an API so others can pull out RAW data.

Lesson 5: Avoid using maps as a crutch. Google Maps is not the app. Use it as a tool to create an awesome app - push data, don’t pull.

Lesson 6: Permalink everything. Anything that can be a link should be a link. Used IMDB.com as example. There’s a special place in hell for session cookies.  EveryBlock gets a LOT of link love and rankings for long tail search queries from Google because of permalinks. Granular Permalinks future proofs a site.

Lesson 7: The simplest way to granulize URLs: Use attributed URLs E.g. Date / crime/dates Type: /crime/types/ Address: /crime/addresses/  Location: /crime/locations/  Then drill down further e.g. /crime/locations/hotels/  Flickr and Delicious use that. Use the Django databrowse to look at database schema and create a directory structure to suit!

Lesson 8: Move Beyond Points - web apps that use geo-spatial info should respect usability. Points aren’t everything. Bubbles aren’t everything. Use lines to make the maps more usable. Use polygons, use transparencies. Don’t overlay data on data. Use GeoRSS to geographically encode objects for RSS feeds.

Lesson 9: Roll your own maps - Google maps is great, BUT use it intelligently. Use colors, fonts, text size, behaviour at zoom levels, road widths, languages etc.

Lesson 10: Make a mashup!

Spread the joy!

Off to Webstock 09

Just a quickie to let you all know that I’m in Wellington for the next 3 days, attending Webstock 09 so won’t have much time to blog here.

But never fear, I’ll be making some live blog posts at SiteProNews about the event so stay tuned. Also hoping to see the Monet exhibition at Te Papa and do a spot of geocaching while I’m here.

Spread the joy!

Web 2.Overwhelming: 22 Ways to Frustrate Your Site Visitors

Damian ConwayDamian Conway is known as the “Mad Scientist of Perl” and he was my favorite speaker at Webstock 2008. He holds a Ph.D. in Computer Science and until recently was an honorary Associate Professor with the School of Computer Science and Software Engineering at Monash University Australia.

A popular speaker and trainer, he is a former columnist for The Perl Journal and author of two books about Perl. He also runs an international IT training company which provides programmer training throughout Europe, North America, and Australasia.

Damian kicked off his presentation by revealing that his wife is responsible for him being at Webstock. A few months ago, she was trying to buy a DVD on the Internet and was yelling expletives. He went to help her and after several minutes of frustration he finally gave up. Her response was “Are they deliberately trying to make it impossible for non-geeks?” His reply was “Yes”. Damian’s impression is that mankind has evolved into two distinct species - typical web users / typical web designers.

Damian then showed the Irony Ahead symbol for the Americans in the audience. The sad truth, he says, is that the web designers are losing the battle to the masses. There are now normal humans who can almost use web sites on a daily basis! His sacred promise is to protect your web sites from infiltration by the terrible general public.

In reverse order, here are Damien’s top 22 web design ideas to fend off the non-geeks and prevent Web 2.0 from taking hold:

22) Use Zen: Confuse them with anime and odd artistic blobs that are a complete mystery. Is it a web site? Is it art? Is it impossible to enter?

21) Use yellow or black and yellow: This signals danger as in wasps and Star Trek uniforms.

20) Use xenophobia: Try geo-location as an instrument of torture. Show only products not available in their country or illegal in their country e.g. “show me products I can’t buy with a credit card issued in my country”.

19) Get a site that requires the “www” to work: This is known as a canonical URL to us normal people. Make the www necessary and confuse the heck out of them when they type in the domain name without the w’s and get shown an error page.

18) Use variable navigational layout (VNL): Use the previous and next links as people hate them. Let’s replicate it for the entire web!

17) Throw usability out the window: Navigability is the pre-requisite for usability. So let’s create navigation buttons that don’t go anywhere. The non-geeks will be occupied for at least half an hour. Use Javascript to turn the navigation into confusing shadowy arrows. Javascript your scrollbars so they don’t look anything like regular browser scrollbars. Use back buttons that embed links that take you up a level rather than actually taking you back. Your visitors will get lost in the hierarchy. Inconsistency is important.

Typography doesn't matter16) Terrorize them with typography: Most non-geeks don’t care about typography. They don’t even have a favorite typeface! If they do, it’s comic sans. Or Impact! All they care about is whether they can read it. Therefore, typography DOES matter. Go with something unreadable! Go with Abduction 2 font or something just as annoying. Fonts are not toys people, fonts are weapons!

15) Make shipping a last minute surprise: Shipping is a powerful tool to dissuade purchasers. It delays their instant gratification. Don’t let them calculate the shipping cost in the cart otherwise they can go and comparison check on other sites. Instead, use the W3-recommended 34 step method and make all these fields compulsory:

  • product selection
  • shopping cart
  • checkout
  • purchaser address
  • phone number
  • fax number
  • email address
  • social security number
  • payment method
  • billing address
  • shipping address
  • shipping method
  • shipping costs
  • income bracket
  • referral source
  • etc.

With any luck, they’ll abandon the cart in total frustration.

14) Make them register and login before they purchase: They’ll be naturally terrified and run off. Even better, make them register before they can even view the web site!

13) Reduce the quality of site search results: How can you minimize the quality? Don’t provide site search facilities at all! Or make the options highly improbable. Don’t let them search for the product. Make them search by date of manufacture, or the name of the manufacturer. Or, make them search for the type of person that they are. Or what type of person YOU think they are. Pure genius.

12) Add pages ad infinitum: Don’t return more than 10 results for a search at one time, even if you have to list 250 pages of search results. God put the fold there for a reason. If you don’t follow this rule, it can result in scrolling! Protect the kids from scrolling!I'm a programmer - you have to guess what I mean

13) Delay their gratification or their dis-gratification: Show items that aren’t in stock, services you used to provide, options that won’t work for them. Only tell them a product is not available AFTER they click through to the shopping cart total. This builds up a sense of hope so you can dash it immediately.

14) Don’t allow them to sort search results: This non-sortability of results preserves the natural social order. Sort things in random order. For example, don’t let them sort by product type, or price. Provide an alphabetical sorting option only. Or sort according to the web designer’s favorite items.

9) Use background music and lots of it: It’s a sure way to irritate your users. Your music choice probably sucks if you choose it carefully enough. Don’t provide a stop button. Make it restart again on every new page. They’ll soon leave.

8 ) The little things count: Like tiny little font. It’s the most effective deterrent for anyone over the age of 20. Damian finds size 4 or 3 point is pretty good. Government and news sites use it all the time to great effect. Some browsers have the ability to change text size. Thankfully, most web users Damian surveyed didn’t know this until it was pointed out to them. But it’s ok! Because 2 weeks later, they’d forgotten again. Tiny text is the web designer’s ally.

People don't care7) Use Cute Kitten Aversion Therapy: There are some web sites that you don’t want your kids to see, Damian says. One of these is the W3 HTML Validator. AAARGGGGHH! It means that solutions for non-valid HTML code could be discovered by anyone and you don’t want that. So spread the message, every time you validate, someone kills a kitten!

6) Use J-version therapy: The non-geeks have a strong aversion to the letter J and things like Jscript, Javascript etc. These J languages create fear in the non-geek. Online security companies have scared them into avoiding sites with Javascript or other items starting with J because hackers use them to distribute viruses. If you’re lucky, they’re so convinced by these fears that they’ve turned off Javascript in their browsers. This means that if your site uses Javascript menus, they can’t be navigated! Brilliant.

5) You can never use too many images: Encode your important data and text in an image so it can’t be cut and pasted and make the images huge and dark so that they can’t be printed out. Or they can be printed but they use up masses amounts of printer toner. It’s a great way to scare off even the most persistent of non-geeks.

4) Play hide and seek with your site visitors: They don’t like to wait, so make them. Information that is impossible to find is safe. Don’t use a sitemap and make sure there is no rational hierarchy to your site. Hide your most important data on a page that has no links pointing to it!

3) Use gray: It’s the new black and it goes so well with black or darker gray. When using gray, make it impossible to read. Use nano-text in gray or even gray text on white. The site visitors run away! Even better, use gray on darker gray - it’s the low contrast approach. This is even more effective for site visitors with a color impairment. If all else fails, use intestinal beige. It’s apparently the new gray.

2) Flash is very important in our defense against web-mortals: “Clocksucking Flash” they call it. Some non-geeks even have their Flash facilities turned off so make your site entirely in Flash. Once visitors arrive at your Flash page, they see - “loading 1%”. This is delayed gratification at it’s best! Also, don’t provide a “skip intro” button. Or if you do use it, make the link move away from the mouse. Over and over again. If the visitor persists, then make it disappear entirely. If the visitor without Flash is determined to view your Flash site, provide the “You Need Flash” link. Then make them download an enormous file that maxes out their bandwidth limits. Even better, use a Flash-based installer that requires them to have Flash installed first.

1) Combine all of the above for optimum effect: This is Damian’s number #1 best way to scare away non-geek visitors. His favorite example of this in action is the World Glaucoma Association. Scroll down and place your mouse over the eye for the full effect. [Editor note: My own personal favorite is Fred Frap and Friends where the pink text on the purple background asks you to click on the non-existent image to enter. Nice!]

Irony endsBut seriously folks, Damian says, the non-geek level is the SAME LEVEL as the MAJORITY OF YOUR WEB SITE USERS. Remember this. The typical experience for web-mortals is bad. He’s here to plead with us all to build software for how people really ARE. The non-geek users.

The Grandma Usability Metric

The single biggest mistake that web designers make is not doing accessibility testing on their grandparents. Your grandparents are the typical Internet users. Use the Grandma usability metric. It’s not about what your client wants. It’s about what your client’s customers want. It’s not about clever, it’s about comprehension. It’s not about style, it’s about usability. It’s not about searching. It’s about finding. It’s not about ambience it’s about the outcomes for people visiting your site.

Web 2.0 is Web 2.0verwhelming for most people. So make their experience measurably superior.

Spread the joy!

How Google Applies Science to Search

Craig Nevill-ManningSiteProNews have now published my 2 part article based on the Webstock 2008 presentation by Google’s Senior Research Scientist, Dr. Craig Nevill-Manning.

Here’s Part 1 and Part 2.

In his presentation, Craig, who is New Zealand born and bred, explained how Google uses science to develop more precise search techniques. I found his talk absolutely riveting and typed frantically during the whole thing in my hurry to blog it.

Here are a couple of classic excerpts:

Google used to do a terrible job of defining terms. Craig noticed people were searching for “definition of…”, or “what is a….” etc so he wanted the search engine to provide better results for these searches. He found lots of web pages that contained glossaries and definitions, so he hacked up a Perl script to get the glossary formats.

The first recall results were only 50 percent accurate. He wanted to improve this rate, so he did some experiments with the data. But he could never reach an accuracy level he was happy with. It was later he realized that most of the questions people actually needed answers to could be answered with his crappy little Perl script. He concluded that 100 percent accuracy is not important, that scale is much more important.

Craig says that once a week, a person at each data center has a list of all the failed hard disks and walks around the datacenter with a pile of hard drives, replacing them one at a time. Velcro is Google’s secret weapon! All Google’s hard disks are velcroed in. This allows super quick service and replacement time. So curiously, there is no downside to hardware failures at Google, because they are expected and managed via scale.

Fascinating stuff!

Spread the joy!

Meeting Jill Whalen

Jill and KalWell after nearly 8 years of communicating online, I’ve finally met Jill Whalen in person.

We’ve always talked about meeting up at one of the search engine conferences in the U.S. but I can never quite get my act (or budget!) together to make the long trip. When I heard that Jill was speaking at Webstock 2008 here in New Zealand, I figured it was my best chance to catch up with her so I registered and booked my flight from Christchurch to Wellington.

Originally I was planning on surprising Jill by just turning up to her session and tapping her on the shoulder, but then I thought that if I didn’t tell her I was coming, her schedule would be booked out and we wouldn’t have the chance for a major catch-up. So I emailed her to let her know I was coming. Turns out that she didn’t have any plans apart from the speaker’s dinner and was glad to find a playmate!

The day I arrived we had a brief chat over a drink at her hotel (The Museum Hotel) and then on Thursday night we had dinner at a Chinese restaurant. We originally planned to go to Restaurant 88, a well-known Vietnamese place but being Valentine’s Day, the place was booked solid and turned us away. So we went next door to a little Chinese restaurant instead. The photo above shows us at the restaurant Thursday night. As you can see by the empty chairs behind us, we stayed way longer than any other patrons and I think they were about to kick us out! Apologies for the grainy photo but my digital camera was playing up the whole trip.

It was great to chat to Jill and gossip about this crazy industry we work in. We talked quite a bit about the personalities in search and about the generation gap that is developing between SEO experts of our era and the new generation of SEO young guns climbing the charts. We also talked about the opportunities that conferences and events can provide for people in this industry and how it can make the difference between an SEO and what I call an SEO Rockstar. While talking to Jill, I realized that I had probably let quite a lot of opportunities for Search Engine College and my consulting business slide in the past because I haven’t really embraced the conference scene.

Anyway, it was just fantastic to meet Jill and be able to give her a real hug instead of a virtual one. She’s always been an inspiration and it was great to pick her brain and have a good old gossip. Maybe the next time we are both on the same continent we can meet up again eh Jill?

Spread the joy!