As a technologist who’s focused on marketing, I love stats (short for statistics) because they help me tell a story. But as a former researcher, I’m very familiar with the famous quote by humorist Mark Twain:

“Facts are stubborn, but statistics are more pliable.”

At last night’s AiMA event on search engine strategies, the speakers referenced a study where users showed no significant preference to Bing or Google. After a short web search (via Google), I found the research paper by the Catalyst Group (see below). In the study, users reported that they wouldn’t switch from their current search engines even though Bing possessed some favorable improvements to Google.

While the findings attempted to explain how Bing will never catch up to Google, I was surprised that the speakers chose to quote the study as fact. There were two glaring issues that I immediately noticed once I located the study:

• The participants used Google as their main search engine.
• The study involved only 12 participants.

While some may argue that the opinion of the participants was tainted (since none used Microsoft Search as their primary search engine), I was even more surprised that no one considered the small participants pool. While I subscribe to the notion that we’re creatures of habit and users won’t change when they can’t find a significant value in the new “shinny object,” the bottom line is that the study’s findings are statistically invalid. While I’m fairly certain that the speakers were not aware of flaw in the study, it demonstrates how easy it is to rely on and propagate bad statistics and how careful marketers need to be when they quote a study.

NOTE: I recently wrote a blog post about Twitter data and Rapleaf based on an NPR radio bit. Rapleaf reached out and explained the issue with the misinformation that I had referenced. It goes to show that we’re all human; we make mistakes.

Catalyst Group Bing V. Google Usability Study

Online retailer Amazon.com recently blew out revenue estimates by posting a 28% rise in quarterly revenue to $5.45 B for Q3 2009. According to the conference call transcript (source), the improvements were due to a few factors, including:

• Continued focus on Amazon’s value proposition: low prices, expansion of selection, convenience, and good in-stock levels.
• Availability of Amazon Prime program, which allows customer to enjoy “all-you-can-eat” fast shipping on eligible purchases for an annual membership fee of $79.

Amazon has leveraged their low prices to attract buyers. While pricing is important to these visitors, the online customer experience helps convinces them to complete the purchase vs. going to a rival retailer. For example, Amazon monitors when a customer drills down on to a specific product and uses email to send them special pricing for that item if they don’t purchase it. Amazon has been able to expand their selection of products by offering better tools to sellers, which is another area that Amazon invests heavily to keep their brick-and-mortar competition in check. In addition to these, there are many other feature that Amazon has refined over the years to improve the online experience (view analysis of Amazon’s online experience).

According to a 2008 study by ServiceXRG (source), customers are 3X more likely to to buy a product or service and 4X times more likely to recommend the company or renew the relationship if they have a positive experience. The recent quarter at Amazon is a testament that you can own the market place by implementing only a few simple, fundamental changes.