What Have We Learned About Amazon Book Reviews?

April 16th, 2012

Following is a ‘reader’s digest’ summation of some of the scholarly research published in the last five years that analyses out how Amazon’s book review system functions.

I’ve pulled this together for the sole purpose of stimulating thought and discussion on the topic. It’s research: not gospel truth. But the ideas are provocative. Perhaps most provocative is what underlies this posting: that there is an effective way to game Amazon’s book review system without being caught and banned.

For the researchers who I paraphrase here, let me apologize in advance if my summations fail to do justice to your work. That is why I’ve included the bibliographic references and links: so that readers can turn to the source and make their own assessment (some are behind firewalls – you may need to set foot in a library to read them).

I’ve collected 22 papers thus far, so will augment this post when time permits.

1. The Bandwagon Effect of Collaborative Filtering Technology

S. Shyam Sundar, Anne Oeldorf-Hirsch, Qian Xu
CHI 2008, Florence, Italy, ACM 978-1-60558-012-8/08/04

The paper analyses “The Bandwagon Effect” – “if others think that something is good, then I should, too,” using three variables:

  1. Star ratings
  2. Number of customer reviews
  3. Sales rank

The results were unsurprising: these factors make an impact on perceived value.

What I found interesting was a discussion of a paper that compared how readers of a news websites felt about stories that they selected versus those selected for them by a news editor, a computer, or other users. They found that “other users” beat both the computer algorithm and the ‘expert’ news editor.

2. The Effect of Word Of Mouth on Sales: Online Book Reviews

Judith A. Chevalier, Dina Mayzlin, Yale School of Management, September 2005

Our econometric analysis is designed to answer the question: If a cranky consumer posts a negative review of a book on BN.com but not on Amazon.com, would the sales of that book at BN.com fall relative to the sales of that book at Amazon.com?

We analyze reviewing practices at Amazon and BN.com. We find that customer reviews tend to be very positive at both sites and that they are more detailed at Amazon. We find that an improvement in a book’s reviews leads to an increase in relative sales at that site, and the (negative) impact of 1-star reviews is greater than the (positive) impact of 5-star reviews.

3. All Reviews are Not Created Equal: The Disaggregate Impact of Reviews and Reviewers at Amazon.com

Pei-yu Chen, Samita Dhanasobhon and Michael D. Smith, May 2008

Ratings that provide a simple 5-star (or 1-star) review, while having equal weight in the overall average star rating listed on Amazon’s site, do not have as much influence on consumer response as more detailed reviews that have been rated as “helpful” by other members of the community. Moreover, these reviews have a stronger impact on less popular books than on more popular books. We also find evidence that “Spotlight” reviews have a stronger effect on book sales than overall reviews do — which is consistent with the hypothesis that consumers may economize on costly search by focusing their attention on a few highlighted reviews. However, contrary to our expectations, we find no evidence that the reputation of reviewers (i.e. top reviewers) is an important factor in consumers’ purchase decisions.

4. Can Brand Reputation Improve the Odds of Being Reviewed On-Line?

Naveen Amblee and Tung Bui
International Journal of Electronic Commerce / Spring 2008

I found this study a little muddled: based on an analysis of 395 ebooks sold by Amazon over a period of six months, the authors found that goods that “start with a highly rated brand” are more likely to have additional reviews posted than goods with “an initial poorly rated brand”. You’ll have to read the study to understand how they bring “brand” into the mix.

5. Designing Ranking Systems for Consumer Reviews:

The Impact of Review Subjectivity on Product Sales and Review Quality

Anindya Ghose, Panagiotis G. Ipeirotis
Department of Information, Operations, and Management Sciences, Stern School of Business, New York University

This paper is unique in looking at how sentiment in text of a review affects product sales and the extent to which these reviews are informative as gauged by the affect of sentiments on helpfulness of these reviews. We find that reviews which tend to include a mixture of subjective and objective elements are considered more informative (or helpful) by the users.

In terms of subjectivity and effect on helpfulness, we observe that for feature-based goods, such as electronics, users prefer reviews to contain mainly objective information with a few subjective sentences. In other words, the users want the reviews to confirm the validity of the product description, giving a small number of comments (not giving comments decreases the usefulness of the review). For experience goods, such as movies, users prefer a brief description of the “objective” elements of the good (e.g., the plot) and then the users expect to see a personalized, highly sentimental positioning, describing aspects of the good that are not captured by the product description.

The authors (with Nikolay Archak) take their work to the next level with their 2007 paper:

Show me the Money! Deriving the Pricing Power of Product Features by Mining Consumer Reviews

This paper focuses purely on electronics to offer examples “where opinions that would be interpreted as positive by existing opinion mining systems are actually negative within the context of electronic markets.”

Anindya Ghose and Panagiotis G. Ipeirotis are on a roll! Their 2011 paper, Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics (IEEE Transactions On Knowledge and Data Engineering, October 2011) builds on their previous work “by expanding our data to include multiple product categories and multiple textual features such as different readability metrics, information about the reviewer history, different features of reviewer disclosure, and so on. The present paper is unique in looking at how subjectivity levels, readability, and spelling errors in the text of reviews affect product sales and the perceived helpfulness of these reviews.”

They pin down their earlier hypothesis and state clearly: “Reviews that rate products negatively can be associated with increased product sales when the review text is informative and detailed. This is likely to occur when the reviewer clearly outlines the pros and cons of the product, thereby providing sufficient information to the consumer to make a purchase.” (Italics theirs.)

Gem #2: “For experience goods such as DVDs… users prefer to see a brief description of the “objective” elements of the good (e.g., the plot), (and then) a personalized, highly sentimental positioning, describing aspects of the movie that are not captured by the product description provided by the producers.”

Gem #3: “An increase in the readability of reviews has a positive and statistical impact on review helpfulness while an increase in the proportion of spelling errors has a negative and statistically significant impact on review helpfulness for audio-video products and DVDs.”

There’s more, sufficient that the authors can state with confidence that they can identify the most effective reviews.

6. Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets

Chris Forman, Anindya Ghose and Batia Wiesenfeld, 2008

“Our findings suggest that in the world of online consumer reviews and sales, we may need a more nuanced understanding of the old adage that “there is no such thing as bad publicity…. Our findings suggest that negative reviews may not necessarily be bad for sales. More likely to have a negative impact on sales is anonymous reviews – those failing to disclose real name or location.”