19 Comments

Summary:

For many online shoppers, Amazon (NSDQ: AMZN) is their starting point for researching products they want to buy — and Amazon’s customer rev…

amazon reviewers

For many online shoppers, Amazon (NSDQ: AMZN) is their starting point for researching products they want to buy — and Amazon’s customer reviews, in particular, play a key role in those purchasing decisions.

But there is some new evidence suggesting that Amazon’s customer reviewers–particularly the top 1,000 reviewers–do not always make independent decisions about which books and other products they write about. According to a new Cornell study that we previewed last week, the reviewers in many cases acknowledge that in order to maintain their high rankings and continue to receive free products (one of the perks of being a top reviewer), they have to make surprisingly calculated decisions about what to review and what to say about those products.

The author of the study, Cornell professor Trevor Pinch, says the fundamental problem is that people reading the reviews probably naturally assume that the Amazon reviewers are regular shoppers just like them–when, in fact, their relationship to the products they review can be a little more complicated. “The issue of the ‘customers’ not really being customers needs to be addressed,” says Pinch, who surveyed 166 of Amazon’s top 1,000 reviewers for his study.

We reached out to Amazon to talk about its product-review system, and to get the company’s response to some of Pinch’s claims. But the company didn’t respond.

History

In Amazon’s early days, in the late 1990s, the “Editorial Reviews” that appeared on book pages were written by Amazon employees–especially editors, but “anyone who worked for the company, including warehouse staff, were asked to write as many as 10 reviews a week.” Amazon later made deals with book review publications like Booklist, Publishers Weekly, Kirkus, The Library Journal and the New York Times Book Review, to copy their reviews of newly published books. Over time, the literary editors hired to write reviews in those early years have either left or moved to other positions in the company, and customers themselves have become the main source of reviews on the site–though the Editorial Reviews remain.

For the most part, though, Amazon has outsourced the job of writing reviews to thousands of unpaid citizen reviewers. Seventy percent of the top reviewers are male, their median age is 51-60, and more than half hold a graduate degree. About 14 percent of those reviewers are professional writers. Why do they write the reviews for free? Respondents to Pinch’s survey overwhelmingly mentioned “self-expression” and “enjoyment” as their motivations. Many respondents also cited altruistic reasons for reviewing–“hope to help others decide whether to buy,” “wanting to share what I have liked with others,” etc. And some said they write reviews to help them keep track of the books they’ve read and the movies they’ve watched.

But in the interviews with Pinch, respondents talk about some other motivations that might interest readers of these reviews. The study, for example, found that 85 percent of respondents had received free products from publishers, agents, authors, and others. Why is that an issue? Professional critics–at a publication like the New York Times–also receive free books to review, of course. But those critics are paid by the publications they write for, and their job is to review these books objectively. For Amazon’s unpaid customer reviewers, the only tangible benefit of their “job”–and the study indeed found that for top reviewers reviewing is akin to a second “career,” a “crossover occupation”–is any free books and products they receive. The way to keep those freebies flowing is to pump out glowing book reviews. (Amazon explicitly tells reviewers to “please clearly and conspicuously disclose that that you received the product free of charge,” but Pinch says reviewers don’t always adhere to that directive.)

Some 88 percent of respondents reported that most or all of the reviews they wrote were positive. “I don’t want to make waves, and I don’t want to offend the author,” one said. “I’m in the midst of writing a book myself, and I’m thinking it might be prudent not to be TOO overly critical of books that go through the traditional publishing process.”

The Ranking System

One reason reviewers care about being “prudent” is the way that Amazon’s reviewer ranking system works. In 2008, Amazon made broad changes to the system, causing many longtime reviewers to lose their top rankings. These changes have been a subject of great debate on Amazon forums. While Amazon has been secretive about the algorithms for its reviewer rankings, it says that under the new ranking system rank is determined by the “overall helpfulness” of all of reviewer’s reviews (as rated by customer votes), the number of reviews the person has written, and the recentness of the review. Recent reviews get more weight.

Sixty-seven percent of the respondents to Pinch’s survey disliked the new ranking system: “It rewards the newbies at the expense of long-term reviewers who have worked for years on the site,” one top reviewer wrote. Confusingly, Amazon still includes both reviewer ranking systems on its website, listing reviewers by both New Reviewer Rank and Classic Reviewer Rank. The top-10 lists are different, and nobody who is a top-10 reviewer also holds a top-10 spot in the classic rankings.

“Not Helpful”

Some reviewers told Pinch that they steer clear of books on controversial topics like politics and religion — because reviewing those books can increase their number of “not helpful” ratings. One said: “A positive review of a conservative politics title is sure to attract a great number of ‘not helpful’ votes by those who don’t like the author’s politics.” Wrote another: “Since some people mark reviews as ‘unhelpful’ simply because they disagree with them, this means a top reviewer is most likely to be someone who only gives the ‘correct’ review of a book, rather than a more nuanced and balanced review, or critical one. The new system discriminates against minority opinions and seeks homogeneous reviews and fans of those reviews.” And a third respondent said, “A reviewer can either be willing to address a controversy OR simply go for a higher Amazon ranking. He cannot do both as Amazon has made them mutually exclusive!”

“Helpful” votes lead not only to higher rankings, but also to more free books. Pinch says it appears that publishing companies and agents start to offer free review copies to Amazon reviewers when they hit the top 1,000. Once they made it into the top 100 or top 50 reviewers, they got many more offers. Some respondents mentioned that if they didn’t like a book they received, they would give its sender the choice of whether or not they should post the review. Not surprisingly, the answer was “invariably” no.

To be sure, Amazon isn’t the only site that has critics who question the soundness of its reviews. Yelp, one of the top review sites by traffic, has its share of detractors. In May, a group of small-business owners filed a lawsuit against the site, accusing it of offering to bury bad reviews if the business bought ads. Yelp has rejected the claims, and that case is currently in court.

Who’s to blame for apparent flaws with Amazon’s reviews? There don’t appear to be any obvious villains here. There’s no evidence that Amazon is secretly pulling the strings behind the scenes to keep all the reviews upbeat. And it certainly doesn’t seem as if the citizen reviewers have some innate desire to avoid important but politically charged topics.

That said, reading Pinch’s interviews with reviewers, you get a sense of how hard it is maintain the integrity of a process that is dependent on a virtual army of unpaid but still presumably capitalist-minded laborers. If they’re not paid, they are going to find other incentives and motivations–which may in some cases work at cross purposes with their primary mandate, to produce honest and independent-minded reviews.

  1. I’m not sure of the point of this story. Are you implying that anyone honestly believes that the reviews on Amazon comprise credible journalistic posts? Or that those reviews should demonstrate the same sort of fairness and impartiality that mainstream journalism is supposed to possess?

    If so, that’s silly. No one who writes a review on Amazon agrees to abide by any sort of ethical code (nor, I think, do many journalists; but I digress). At least Amazon features a badge that lets readers know whether the review’s author purchased the product on Amazon. As far as reviewers’ motives — whether those are to maintain a ranked position, score points with a manufacturer, or feed an ego by building post count — again, who cares?

    If you’re reading reviews on Amazon without any critical thinking occurring between your ears, that’s your fault, not the site’s.  If your key criterion for the value of a product review is that it’s “honest and independent-minded,” I suggest visiting Consumer Reports or a review site with a reputation for editorial credibility. Amazon’s reviews are anecdotal and should be treated as such.

    Sorry, but why anyone would wonder “who’s to blame for apparent flaws with Amazon’s reviews?” is beyond me. If you don’t trust them, don’t read them. Simple as that.

    Share
    1. I have heard numerous people claim they 100% trust theAmazon reviews because they do beleive the reviews are posted by people with no bias or ulterior motive, people like themselves.  Really, many just count the number of favorable reviews and use that to make purchase decisions without doing any more research in either the product or the quality of the reviewer. And I fear that’s close to a majority of buyers.  When I  mentioned the possibility of bias people just looked at me.

      Share
  2. As the article clearly stated, the problem is that most people assume that the top reviewers are customers like themselves, who post reviews to express their opinion.  That is not the case: it turns out that top reviewers are receiving kickbacks in the form of free books, and there is an inherent incentive not to be overly critical of a book.  It’s not that people weren’t already reading these reviews critically, it is just that now they have to account even more forms of bias.  These are not simply enthusiastic readers of the books.

    Share
  3. Amazon Top 1000 Reviewer Saturday, June 25, 2011

    As one of those top Amazon reviewers who gets “kickbacks” as some people seem to think they are, I will say this study is so far off the mark it’s disappointing. Even the demographics of the reviewers is hugely slanted. I’m a top 1000 reviewer, and I was never asked to be a part of any study. I have been buying and reviewing at Amazon.com for years, and for every one or two dozen reviews I do on products I buy, I will get a chance to review (for free) an item like Post It sticky notes, or a book from some unknown author that no one will buy until someone else has read and reviewed it to share some form of an opinion.  Also, top reviewers give plenty of unfavorable reviews on items. If we didn’t everyone who purchases the items after reading our reviews would come back and vote negatively on all of our reviews, then we would lose our rank and not be a top reviewer. We get to be top ranked by how the public votes on our reviews. If we aren’t accurate, we lose our title as a top reviewer. It’s our fellow customers who decide who’s the top reviewers, not Amazon. Amazon doesn’t read our reviews and decided who does the best. It’s all on how people reading the reviews vote on how helpful the reviews are. How sad that the one time a great study about this could have been created, they totally miss the mark by a mile.

    Share
  4. My personal experience with my first book, Asatru For Beginners, is that real customers who have bought my book consistently give it higher ratings and better reviews than book bloggers who received a free copy.

    You can see that by reading the reviews here: http://www.amazon.com/Asatru-Beginners-Erin-Lale/product-reviews/1448961491/ref=pr_all_summary_cm_cr_acr_txt?ie=UTF8&showViewpoints=1

    Share
  5. I’m in the 300s in the Amazon ranking. I have no issue with being critical.  I
    get some freebies through Amazon and some through a review site, but
    most items I purchase. None of the freebies are dependent on a positive
    review. The items through Amazon Vine are automatically posted under a
    banner explaining they were given, and with the items I receive
    elsewhere I post that info in the body of the review. And then I tell
    people what I really think. I know a lot of reviewers who won’t rock the
    boat, but in my experience that’s a personality trait based on not
    wanting to be a green meanie and nothing to do with swag.

    I’m pretty sure there’s a fairly long list of writers who think I’m a witch.

    Share
    1. Are you? A witch, that is? If not, that’s a poor choice of words. :)

      Share
      1. Am I a witch? You’d have to ask my husband.

        I think I’m a reader/reviewer with reasonable standards. I get moderately bummed out when I see an article like this calling my ethics into question when life would be a lot easier if I could lower the aforementioned standards — be an author booster instead of a reader advocate.

        Share
  6. What is missed in your article is the new trend in Amazon reviews. The ‘free chapter’ review. This is where a so called reviewer writes a short review based on reading either the ‘Look Inside’ feature, or a Kindle sample only. 

    These are becoming more frequent and once posted are fed to a number of Twitter accounts and Facebook pages. It is now very common to see the exact same review posted on Twitter by as many as 20 or more accounts. They are also used in ‘click’ marketing campaigns. 

    Most of them are inoffensive and glowing of course, but I have seen some emerging recently that are very negative and could well be assumed to be aimed at reducing a competitive author’s book ranking.

    While as an author I am very pleased with the Amazon reviews I have received and have no complaint, there will always be those who push the rules as far as they can. It is and was no different with traditional publishers who used advance reader copies of a new book to garner reviews before release. So nothing is new really.

    Share
    1. Hey Derek, I hadn’t known about the free chapter reviews…interesting. Thanks for commenting!

      Share
  7. David Holberlt Monday, June 27, 2011

    I am a consultant who touches upon marketing content from time to time. I don’t write reviews and don’t know any reviewers. The tone in the article of something having been uncovered is mystifying.

    Is there something wrong with a higher count of positive reviews? These are free reviews for generally obscure material. These are not prominent critics weighing in on already well publicized material. The whole point is to engage people with content they don’t know about. It would be quite natural to refrain from reviewing if the impression were negative. Selection is exercised by a consumer initiating a search (not an opinion of a reviewer). Then they refine by reviews (and other means as well). They can jut as well do this with a range of positive reviews (or ignore them altogether as a comment above observes). And who’s judging what’s a “positive review” anyway? I imagine many a book is damned by faint praise. And what’s up with all those glowing quotes every book ever published received from professional reviewers? Was there some scandal about that? 

    Is there something wrong with forwarding content to a reviewer in hopes of getting some publicity? What, someone’s going to compromise their integrity to get more free books they didn’t want in the first place? How do you suppose, in this brave new world, an unknown artist or author would get started? Amazon reviewers are not the replacement for the New York Times Book Review. They are vetting process whereby new content is recommended (by degrees). If they replace anything, they are replacing book jackets, posters, and ads.

    Share
    1. Hi David, I think the thing that most people who are shopping on Amazon don’t realize is that all this is going on behind the scenes–i.e. that the people writing the consumer reviews are not, in all cases, consumers. I do agree with you that it is up to the people *reading* the reviews to be conscientious about the buying decisions they make, but it’s important that they realize that there is more going on than someone simply buying a book and recommending it (or not). And if Amazon reviews are becoming something like free advertising “replacing book jackets, posters, and ads,” that seems very problematic.

      Share
  8. I am an Amazon reviewer. I receive up to 2 products (in my case books) a month from the Vine programme, in return for which I review the book. Amazon’s instructions insist that reviews are honest, that is, critical or positive. Also, Amazon automatically inserts a line to the effect I’ve received the book free as part of Vine. 
    I also write book reviews for publications, websites and my own blog, and often archive them on Amazon. As I am keen on translated fiction, I hope other potential readers find the reviews useful as these books are often not reviewed in the mainstream newspapers that tend to focus on “bestsellers”. 
    I read other people’s reviews on Amazon if I’m not sure about a book but do not rely exclusively on them – I’ve almost always read external reviews of a book (or have read the author before) before I buy on Amazon. I find some of the top 1000 reviews useful and some less so, in my view anyone who purchases books and is a keen reader is perfectly capable of judging for him or herself the value to him/her of a review.

    Now, on the other hand, self-published rubbish on Amazon with fake publisher names, fake rankings and fake reviews (or ecstatic reviews from little circles of fellow writers doing each other good turns) — that really drives me mad! I wish Amazon would do something about that.

    Share
  9. How to make money, the Amazon way:
    1. “For the most part, though, Amazon has outsourced the job of writing reviews to thousands of unpaid citizen reviewers.” 2. Use bricks and mortar bookstores as your free storefronts.

    Share
  10. Here’s a new topic of conversation, guys, something I was thinking about…do you think that reviews for books are less reliable than books for other stuff on Amazon (like electronics)? What have you found in your experience?

    Share
  11. Oh please. What a crock, I’ve been in the top 500 reviewers for ages and
    do no “calculations” or “decisions” – I just review what I feel like.
    The new ranking system finally eliminates “review mills” – many reviewers in the the top 200 did not write real reviews – they use software to generate them and just add in the publisher’s comments.  Total waste of time and BS.

    Share
  12. I am a top 500 reviewer.  The implication that review quality is compromised bceause a book is free is somewhat ridiculous, or that it is some sort of worthwile professional enterprise for the reviewer is somewhat ambitious.  Firstly, a $15 book requires a considerable investment of time merely to write a review. 

    I always tell people I accept books on a no obligation basis, and do not guarantee a review or that it will be positive.  Usually, free books are not bestsellers but authors being published often for the first time and in need of reviews to boost sales.

    I see no correlation between the idea that someone being paid professionally to write a review, is somehow more likely to be honest than someone who is not to be somehow flawed. As if money makes you more honest.  Is someone writing for a magazine not subject to an editorspproval?  Likewise, the idea that the review is based on approval is also suspect. 

    Although top reviewers do get offered free stuff, the significance of this is small.  As the top 1000 represents only 1% of the top 1% of the 8 million reviewers on Amazon, and products advanced free account for only a small amount of this.  The chances that you will ever read a review based on a free product is pretty remote unless you are reading reviews posted on the Vine Program.

    Share
    1. Lp542 — Basic conflict of interest. I always turn down the offer of a free book from a publisher who wants me to review it. Always.
      If it is a subject I care about, I will buy the book, and do not need the publisher freebie.

      Share
  13. There is a definite catch 22 here for new authors.  In order to break out of the “friends and family” circles and get a wider following, one must enlist the aid of bloggers/reviewers and that may include offering a free copy.  I would encourage them to write an honest review, after all, what will I do, show up on their doorstep and demand the book back?  Of course not.  As an honest author, I want an honest review, and–as we all know–if there is conflict or even a stinker review, that may only heighten interest.

    So far, I am reasonably pleased with Amazon’s performance and reliability with my first published book and am even considering cutting to the chase with my next and publishing in e-format only.  We’ll see.  It seems the best way to reach a broader readership.
     

    Share

Comments have been disabled for this post