Smart Web Blog

Why Local Businesses Lag in Benefitting from Online Feedback?

Last week, I came across an interesting post on Greg Sterling's Screenwerk blog about how online feedback is influencing consumer purchasing decisions. The survey and analysis conducted by Opinion Research Corporation concluded that  84 percent of Americans used online customer evaluations in making decision to purchase a product or service. Here's the breakdown on the types of products and services that were surveyed and reported.

While it is clear that travel and consumer electronic goods have reached some level of saturation, the most interesting aspect is that all "local services"  continue to trend up and still lag behind. Contrast this with the fact that, in the pre-online reviews era, local services were the most influenced category through word-of-mouth. I have often believed that information about local services are significantly skewed since they are hard to track. I also believe that numbers would be significantly lower if all the local categories were tallied. However, the following 3 aspects could explain the growth and adoption pattern in local services:

Fragmented Recommendations - There are more than 20 million small businesses and only a small percentage of them have online reviews. It is likely that people would look for different level of trust and authority on recommendations depending on the type of service. One may want a recommendation from a close friend/confidant for a wedding planner while they could easily take a recommendation from an anonymous online reviewer for a dining occasion. Also, if you are traveling for business/leisure and need a good local business recommendation there, you may not get a recommendation from a trusted source. Twitter, Facebook and LinkedIn are definitely helping in eliminating the geography-based limitations but coverage-gap still remains a huge issue. The following picture illustrates the coverage vs. trust conflict in fragmented local business domain.

Review sites such as Yelp, Citysearch have played a huge role in generating reviews that could expand coverage but still their content generation prowess has been limited to major metropolitan areas and is very time & resource consuming. The slow but increasing growth in Opinion Research Corporation's survey may be directly a result of the penetration of such review sources. Review & recommendation aggregators may also be contributing the growth.

Convergence of Opinion (or lack thereof) - Reviews about consumer electronics tend to converge very quickly. For many gadgets, it comes down to product specs and price/performance comparison with others in the category. The early reviewers influence other buyers and ratings quickly reach a settling state. On the other hand, local products & services are very "touchy/feely" and experience tends to differ from person to person. Even a business with 50 reviews may not offer sufficient insight and trust for a prospective buyer to make a decision. This adds to the inability to make a purchasing decision about a local product/service.

Buried Reviews (sort of) - Despite being highly fragmented, local products are services are still the most discussed category amongst consumers. Whether in newsgroups, informal conversations, tweets or blogs, they still dominate the discussion in our daily lives. Many small businesses are rarely online. They may have a website, but between their operational challenges and limited budget, they find it extremely hard to build their real online presence.  The online reputation of a small business is essential to influencing and attracting new customers.  Small businesses need to work on moving the informal discussions in newsgroups, tweets, blogs and other online forums to formal review mechanisms, so they can be tracked and used as influencers in a consumer decision process.

Evaluating Restaurant Search Sites

Last weekend, I had a chance to review the results of the analysis conducted by graduate business students of Santa Clara University. Many thanks to Prof. Kirthi Kalyanam for sharing some of the details on study related to search for "restaurants" on Google, Yelp and BooRah. The analysis included a few questions on business models etc., but I thought I'd share some interesting aspects on customer experience, relevance and how they rated the sites. The graduate business students all evaluated responses based on searching for restaurants in palo alto.

Making Sense of Search Results

"Google uses a combination of keywords, site popularity, and a proprietary cataloging routing to determine the order to present results. Yelp, on the other hand, first presents it sponsored restaurants and then the remainder of the restaurants. It is unclear how it determines the order of the non-sponsored restaurants - although it appears to be come combination of ratings and popularity. Boorah presents the list of restaurants simply by the quantity of "rahs" for each restaurant"

Most of the students inferred that Google did not provide a lot of supporting detail and seemed to perform well only when consumers were looking for maps, directions or a quick phone number. However, this is not surprising since Google is the just the starting point for a more comprehensive search in most cases.

"Boorah/Yelp: Better Dining Experience = Informed Reviews + Restaurant Information + Dining Expectations"

"Google: Quick Location of Restaurant = Map Location + Directions"

Search Results Relevance

The ranking for restaurants vary based on algorithms used by various sites but the trust and credibility in those rankings comes from supporting information. Overall, the analysis concluded that general purpose search engines like Google provided limited but valuable starting point for restaurant search.

Below are some of the typical criteria that were used to determine which site to use in making a dining decision:

    * Site Navigation
    * Number of Search Results
    * Accuracy / Credibility of Ratings
    * Response Time
    * Reviews
    * Relevant information such as reservations, discounts, menus

Google was highly rated them for easy navigation, response time and comprehensiveness of search results. Yelp scored highly for good reviews, good graphical view of user ratings, less clutter and usefulness of information. BooRah scored well when they found search results to be most relevant amongst the three, comprehensiveness of information such as online reservations, menus,discounts and ability to search for those.

Reliability of Reviews & Ratings

Google did not feature much in this discussion but some users clearly identified the two different approaches to building credibility amongst consumers. Yelp's approach is firmly planted in identity ownership and personal information that describes person's reviewing tactics and whether you agree or disagree with their distributions.  The same analysis describes BooRah's approach as a model of aggregation and semantic processing of vast quantity of information, achieving relevance through volume and processing.

Takeaways

Google results did not seem to offer any ability to make a credible dining decision. Except for phone and maps information, the students relied on sites such as BooRah & Yelp to help with recommendations.  A striking commonality across the analysis is how search has matured over the past year.  Almost all of the analysis viewed Google as simply a starting point for certain types of search and a source for basic information.  Not too long ago, people would be somewhat confused about what a ratings and reviews site is and how to use/judge one. Now it seems that there is very little confusion about the first aspect.  Perhaps 2009 will lead to consumers forming clear preferences on what they consider to be important features in a ratings and reviews site.

At BooRah, we firmly believe that a semantic approach to user reviews provides a unique and valuable consumer experience and at the same time caters to a consistent experience on the phone and web. I'm pleasantly surprised by the analysis that endorsed our approach and feel that we provide a highly relevant search for consumer needs. Our reach and breadth are a bit limited for now, but we are growing at a rapid rate and expanding to newer horizons.

Cool real estate search on PaloAltoOnline's beta Site

PaloAltoOnline.com  has launched a new cool beta search site for real estate. It has some really cool features that I like

• Integrated Map and Search options - You can add multiple options to filter the results such as number of bedrooms, neighborhood, min and max price range and see an integrated view of the content. It some ways I like this better than Zillow. Further, the visual indicators for map icons for new, open and prior sales are very intuitive
• You can also go back and forth between the table view and the map view  that allows easy filtering with up/down sort options
• I think video ads for real estate are extremely helpful and are invaluable for consumers

I think some additional information about the agents including their most recent transaction history would be valuable. The ability to mash-up valuable information and enhance user experience is what makes web2.0 fun and exciting.

Restaurant Ratings: What beats Zagat?

Every year, this time around, our favorite movies, Hollywood productions and personalities vie for coveted awards at Golden Globes, People Choice Awards and Oscars. As I see it, the Academy Awards have had significantly larger clout in the industry but the people’s choice awards (PCA) are more indicative of the interests and choices of mainstream American audience.  Similarly, while Professional food critics have given us their recommendations and ratings for restaurants for a long time, there may now be a method for trusting our friends and peers in making our next dining decision.

Zagat has been the pre-eminent publication and authority on restaurant ratings in major cities for a long time. However, with increased user engagement in restaurant review websites such as citysearch, yahoo, yelp, insider pages, there are lot of “self-proclaimed food experts” on the internet. Collectively they may represent our tastes and distastes better than a select few.

My company, BooRah (`boo`s and hur`rah`s), provides an indepth ratings system similar to Zagat's but represents the voice of the reviewers, critics and bloggers in a unified format. In fact, one of our advisors, Kirthi Kalyanam, likes to call BooRah the equivalent of People’s Choice Awards for Restaurant Ratings.  While BooRah leverages reviews written on the Internet, it’s completely different from Yahoo, CitySearch, Yelp and others in how it generates ratings for restaurants. Yahoo, CitySearch and Yelp collect star ratings as input from their users and roll-up the ratings to compute the overall rating for the restaurant. BooRah, on the other hand, only relies on actual review content to eliminate rating bias while at the same time automatically generating comprehensive scores for food, service and ambiance (which are not available on any of the above mentioned sites).

The idea behind this analysis is to demonstrate that a Natural Language Processing(NLP) based system can be as effective or better in analyzing user generated content and computing ratings. So why is this valuable?

In the greater San Francisco Bay Area, there are more than 15000 restaurants and Zagat carries ratings for about 10% of the restaurants. About half of the restaurants covered are in the City of San Francisco. So, if you are someone who lives in the sub-urban areas, it’s unlikely that Zagat has any ratings for restaurants in your neighborhood. On the other hand, BooRah has ratings on food/service/ambiance for 5000 or more restaurants. We can do this because we do not employ any editors. A scalable system that covers the breadth of local search and can generate the quality of ratings similar to Zagat would make the consumer experience a lot more desirable.

So, how do BooRah ratings compare to Zagat? 
Read on for my analysis and conclusions...

Background 

BooRah, uses a patent-pending Natural Language Processing(NLP) system to analyze various user comments and rates the sentiments expressed by users in plain-English text.
It’s different from Zagat since the system directly infers user sentiment from what people have actually written in their “reviews” and does not bias the result with any editorial perspective. It’s similar to Zagat in being able to extract specific sentiments related to Food, Service and Ambiance for various restaurants and rate the restaurant for each of those attributes.

Data for Analysis

I gave my wife a copy of the Zagat 2008 Guide for San Francisco Bay Area restaurants and asked her to pick 100 random restaurants in San Francisco. We noted their Zagat scores for food, service and ambiance in a spreadsheet.  I then wrote a program that calculated the BooRah scores on a normalized scale(note: Zagat ratings are out of 30).  Here I’d like to illustrate a simple example to differentiate how the ratings on our website are different from this analysis. Here’s a sample review:

“the ambience was outstanding and the Wait Staff (other than our waiter) was very attentive. On the negative side our waiter was unprofessional and pompous, the food was presented well but did not have complex flavors “

This review snippet was analyzed by BooRah's Natural Language system and broken down into following scored sentiments:

“ambience was outstanding” – Ambiance – 9
“wait staff was very attentive” – Service – 8
“waiter was unprofessional and pompous” – Service – 5
“food was presented well” – Food – 7 


As you can see, 3 of the 4 scored sentences were positive, there by giving us a value of 75% Rah (which is similar to what you’d see on BooRah's website) where as the actual scores that drive this inference is what is used to compare our ratings with Zagat. 

Correlation Approach

Of the 100 Samples that were picked, 73 samples were valid and met the threshold requirements to proceed further. The first step was to compute an aggregate score for each attribute such as food, service and ambiance. BooRah's proprietary algorithm generated scores for food, service and ambiance for each restaurant. Since our scores are computed out of 10, I multiplied the results by 3 to normalize them to Zagat scores.

To compute the correlation between the scores generated by BooRah and Zagat scores, I used Spearman’s Rank Correlation Coefficient, which does not assume any linear relationship and correlates purely on the rank order of the restaurants. A correlation coefficient greater than 0.5 is considered a strong, and values between 0.0-0.5 are considered to weak to moderate.

Results

Here is a spreadsheet(Download zbr.xls) of values for restaurants that had relevant data for all aspects such food, service and ambiance. Based on the correlation algorithm, following values for rank correlation coefficient were computed:

Food Rank Correlation : .57
Ambiance Rank Correlation : .35
Service Rank Correlation: .30

These values indicate a very strong correlation between BooRah Ranking and Zagat Ranking for Food and average-good correlation for Service and Ambiance.

 

In each of these cases a minimum threshold of number of sentiments/reviews (atleast 20 reviews) was chosen. Further, the higher the threshold, the better the correlation. E.g, if threshold for sentiments was raised to 50 (min 35 review required), the correlation was closer to .7

This trend is generally indicative of any natural language application, including news aggregation systems, which do a much better job given larger data sets.

I then went on to understand the anomalies in the rank correlation. The first discrepancy was for “La Folie in San Francisco”, which is Zagat’s top food restaurant in this 100 list. It scored 27(F), 23(S), 26(A) while BooRah rated it 23(F),23(S),22(A). So I started checking out user reviews & ratings on various websites such as Insider Pages, Yelp and CitySearch and concluded that BooRah ratings were more representative of the user comments than Zagat’s. Similarly, I picked Joe’s cable car restaurant in San Francisco which Zagat rated 24(F),14(S),16(A). I then read reviews from various websites(yelp, insider pages) and concluded that our rating of 22(F),14(S),21(A) was more appropriate for food than Zagat. However, we could have done a better job with Ambiance. While our system was able to capture a lot of sentiments about food ( > 200 Boo’s & Rah’s), Service and Ambiance only had 10-15 sentiments, so a larger data set could have resulted in better result and correlation.

Conclusions

• In a local search use case like “restaurants” where top lists are highly relevant, a ranking system like BooRah is comparable to Zagat and can exceed relevance for an average consumer searching for recommendations on the Internet.

• We’ve all come to believe in the wisdom of crowd (i.e, user generated content in the current era).  The ability to extract and summarize finer detail from such user generated content will be a big step in the future of the semantic web.

It was  very encouraging to see such a good correlation with a gold-standard like Zagat with sometimes the opportunity to exceed their rankings and ratings while exceeding their coverage 5-times over.

Do you agree that an automated natural-language based approach to summarizing user generated content is the best approach? Where do you see summaries and detailed rating on "local search" sites, how do they compare to BooRah or Zagat scores?

Testing Limits for Google

Happy New Year - 2008

I always thought Google was system as big as universe. I've always been more impressed with their response time and keyword indexing than the pagerank itself ( not denying that pagerank was a revolution). Lately I've been experimenting with some large Google adwords campaigns for BooRah partners. Today, I came across this "Alert" on my dashboard - Am I reaching google's adwords limits or am I doing something really stupid?

Is there a Long Tail in "Local Search Queries"

Is there a long tail in local search? In the context of this discussion, 3 things to consider are:

1. "Local" by definition is fragmented, there are a lot of small businesses and the local patrons who utilize their services and products
2. Content about these local businesses - created by professional critics, user reviews, blogs, pictures and videos
3. Searches corresponding to these businesses - ie how to customers find these businesses on the Internet.

With the number of blogs and user review sites that put out valuable content, the long tail seems to be building up for various local businesses, albeit, on a category by category basis. Restaurants, local events, travel related services, salons and bunch of other categories have a lot of meaningful long tail data.  So, the one that interests me the most now is #3.

To be precise in how I formulate the question, let's assume you're looking for a restaurant in Palo Alto. Here are possibilities for search on Google ( and various combinations for each of these queries)

• Restaurants In Palo Alto
• Italian Restaurants in Palo Alto
• Osteria  Palo Alto

Looking at our traffic on BooRah  from Google ( which accounts for a big chunk), we do seem to get a lot of very specific queries from Google ( such as Osteria Palo Alto). The disproportionate amount of traffic to specific searches could mean two things - either we were not well indexed for  generic searches or  there were a lot of long tail searches that consumers performed on the Internet.

Just started experimenting on Google Adwords to get directed traffic against each of those types of keywords and identify any patterns.

Any other thoughts or ideas on how to explore this further?

Destination Palo Alto - Power of Community

Palo Alto, CA has launched Destination Palo Alto, in collaboration with City of Palo Alto, Stanford, Palo Alto Online, Chamber of Commerce and other local organizations. As the world gets smaller and smaller, we become increasingly focused on our local community and lives surrounding us ( stole this from recent article I read, can't remember where). The key to small and niche market success is to cater to the community and engage them in an online dialog. Large social media sites donot necessarily capture that dialog. It's new opportunity that companies like 'backfence' have gone after but could not execute in a standalone fashion. A close tie with incumbents such as community newspapers could work wonders economically. My company BooRah   is powering the restaurants section for this local site. Other local business verticals that BooRah will launch will provide  great opportunity for us and the local advertisers to tap into the trend of bigger online spend in the local market.

Local Search Puzzle

Today's announcement  on Insider Pages acquisition by IAC is another interesting development in the local search market space. This article by Jeremy over at Lightspeed analyzes market dynamics pretty well. Here are some things to consider,

• The average Joe running a small business is a swiss army knife, all skills packed in one. From marketing to operations to  promoting the business, it's pretty tough job. Most of them end up working long hours, weekends and probably stressed out. Even for a Internet savvy small business owner, there is little time to research and actively engage in the online aspects of their business
• The traditional ways of promoting business still seems to work for them, or at least they think so. The people they interact on a regular basis still are the biggest influencers. Their customers, distributors, phone company, local newspapers are their circle of influence. The interesting trend of user generated content, blogs and other forms of social content is certainly intriguing to them (atleast in major metros), but when it comes to getting them to spend money on these online channels, it is still work in progress. Further, Google and Yahoo haven't exactly made it easy to address their needs.
• Therefore a successful approach to cracking the local search puzzle should leverage the ecosystem that has been integral to these small businesses, at least for a while . The relationships that have been established though years must be availed. Companies that try to crack the entire nut in one go will more likely fail than not.

From that perspective, today's announcement is more interesting in how traditional players are trying to aggregate all the social pieces of the puzzle ( well I guess the price was right on this one!)