NYTimes misuses Kindle Review stats to attack the Kindle

The NYTimes’ Bits Blog has an article using Kindle Review statistics to attack the Kindle

It’s a perfect example of why people are losing faith in newspapers.

Here’s what the ‘Is Amazon Working Backward’ article by Nick Bilton does -

  1. It draws up pie-charts of the review ratings of the Kindle 1, the Kindle 2, and the Kindle DX.
  2. It takes the number of 1 star Kindle reviews for each Kindle and compares them.
  3. It points out the fact that Kindle DX has a lesser percentage of 5 star reviews.

It draws the conclusion that customer satisfaction has gone down.

There’s one small problem with this piece of non-journalism – Mr. Bilton never really checked whether the 1 star reviews were legitimate.

Most of the 1 star reviews are not from Kindle owners

If you’re going to claim that user satisfaction of the Kindle is going down you at least should read through the 1 star reviews.

This is the entirety of Mr. Bilton’s due diligence -

A quick perusal of the comments shows customers repeatedly griping about poor screen quality, unattractive device design and the constraints of digital rights management software on books and newspapers.

A quick perusal?

You’re writing an article in the NY Times attacking the Kindle (and making fun of Amazon’s CEO saying they work backwards from the customer) and you based it on ‘a quick perusal’.

Most 1 star reviews are from non-owners

Well, let’s consider these posts that are not based on a quick perusal -

  1. January 2008 – Kindle 1 was getting biased reviews – Only 16 out of the 130 most helpful Kindle 1 star reviews were from actual Kindle owners. 
  2. March 2009 – Kindle 2 was getting biased reviews – Only 24 out of the 79 1 star reviews were from actual Kindle owners.  
  3. August 2009 – 96 out of the first 500 Kindle DX reviews were 1 star reviews. Only 30 out of those 96 reviews were from Kindle owners. 

There’s a clear pattern of Kindle-hating from people who don’t own a Kindle.

It’s rather amusing that Mr. Bilton would highlight 1 star reviews when the majority of 1 star reviews aren’t from Kindle owners.

How could satisfaction of Kindle owners be going down when the 1 star reviews are from non-customers?

The more appropriate theme would be – More people hate the Kindle now, than before.

There are more 1 star reviews for the Kindle 2 because it’s a bigger threat

The real truth about why there are more negative reviews (1 star reviews) for the Kindle 2 is that as the Kindle became successful more people had an axe to grind.

Here are the some of the reasons for which the Kindle 2 has been attacked (from Kindle reviews at Amazon) -

  1. Having DRM. 
  2. After the 1984 incident several ‘Open’ groups had campaigns to go on Amazon.com and add 1 star reviews – People who had no intention of ever buying an eBook Reader were going out of their way to add 1 star reviews.
  3. Lack of PDF support – which has now been added. 
  4. The fact that some Publishers turned off Text to Speech in their books.
  5. Blind Groups upset that the Kindle isn’t accessible (being addressed with a mid 2010 update).
  6. It’s not a real book.

A significant number of the attacks are ideological attacks by people who have never touched a Kindle.

These people are not customers and it’s irresponsible of a NYTimes non-journalist to use 1 star reviews from non-owners as the basis of a claim that customer satisfaction has gone down.

Now look at the reasons the Kindle 1 had been attacked -

  1. DRM. 
  2. It’s not a real book. 
  3. Too Pricey at $399. 
  4. No PDF support. 
  5. Coverage.
  6. You don’t own the book.
  7. 10 cent charge is bad. 
  8. Will not work outside the US. 

A lot of the grounds on which Kindle 1 has been attacked are addressed in Kindle 2 (price, PDF support, coverage, international support). The Kindle product line (let’s leave aside the Kindle DX for the moment) is getting better and better.

The number of non-owners attacking the Kindle 2 is higher because the Kindle 2 is seen as more of a threat.

  1. They no longer have strong reasons to attack the Kindle 2 – it’s no longer $399, it has international support, it has PDF support.
  2. That means we see more and more ideological attacks.

The Kindle DX situation – it’s a different product line

TeleRead pick up the NY Times Blog article and they also point out the lower satisfaction rating of the Kindle DX.

TeleRead picking up the article is why I’m writing about it at all. NY Times’ reputation means people believe what they write is well-researched.

Well, we’ve already seen the article isn’t well researched.

The second problem is – You can’t really compare Kindle and Kindle DX customer reviews and ratings.

They are separate product lines – it’s like comparing a shirt with a sweater.

The Kindle DX is a completely separate product line -

  1. It’s $489 which creates higher expectations.
  2. It’s aimed at a different set of users.
  3. It’s aimed at a different set of uses i.e. textbooks, newspapers, so forth.

We all know the Kindle DX isn’t selling as much as the Kindle 2 and that the NFB (blind associaton) closed down some of the University trials.  

It’s true that, as of August 16th, 2009,  a lower percentage of Kindle DX owners are happy -

  1. 74% of actual Kindle DX owners giving it 5 stars or 4 stars. 
  2. 82% of actual Kindle 2 owners giving it 5 stars or 4 stars.

However, the difference isn’t as stark as Mr. Bilton makes it seem. Not only is he comparing two different product lines he’s only using figures that support his argument.  

It’s misleading to write ‘only 45% of Kindle DX owners gave it 5 stars’ – You’re ignoring 4 star reviews. If we’re tallking about happy customers we should compare 4 and 5 star review totals.

Using Kindle Reviews to attack the Kindle is rather inelegant

Perhaps my biggest gripe with the article is that it’s terrible strategy.

If you use an amorphous, hard to put into words concept you are home safe -

  1. Use Adobe DRM and attack Kindle for not being open. 
  2. Talk about how Amazon is creating unsustainable prices even though you won’t reveal actual balance sheets. 
  3. Talk about how you can share books even though only some Publishers allow it and only one single time per ebook.

You could also attack actual flaws. An attack on either front (made-up weaknesses or actual weaknesses) is understandable.

However, Mr. Bilton’s attack is comical.

By attacking customer satisfaction and doing it via Kindle reviews Mr. Bilton has attacked Amazon on its advantages.

Kindle Reviews from owners are a major strength for Amazon.

Amazon has every Kindle Review and all the statistics out in the open -

  1. 82% of Kindle owners and 74% of Kindle DX owners are giving 4 or 5 stars.
  2. Amazon don’t even remove 1 star reviews from anti-DRM people.
  3. Contrast with the Nook that doesn’t even have customer reviews.
  4. Amazon users can vote on reviews so you can check the most helpful positive and negative reviews.

Mr. Bilton couldn’t have picked a worse set of statistics to attack.

Amazon’s Customer Service is a huge strength, perhaps it’s biggest strength

Lots of people are buying the Kindle based primarily on Amazon and its customer service -

  1. In both the US and UK, Amazon regularly gets voted #1 or #2 for customer service.  
  2. Check out any forum and there are a lot of people very happy with customer service.
  3. Read the negative Kindle reviews and the ones from actual Kindle owners sometimes reflect the level of service people expect i.e. free replacements for broken kindles and so forth.

Most Amazon customers are going to laugh at the suggestion that Amazon sucks at customer service.

Closing Thought – Amazon’s enemies need to read Art of War or something

Consider some of the mis-steps -

  1. Nook announcing its features 6-7 weeks ahead of time.  
  2. Nook and Sony Reader not knowing actual market demand (thanks to bad projections and Kindle secrecy ;) ) and not having enough stock this year.
  3. Newspapers attacking the Kindle on issues people either don’t understand or don’t care about.
  4. This article from NY Times attacking Amazon on two core strengths i.e. open reviews (oh my god – Amazon is open, it’s terrible) and focus on customer service.  

Trust NY Times to attack Kindle and put up pie charts that show 82% of Kindle owners are giving it 4 stars or 5 stars – even using kindle reviews from non-owners they can’t mount a decent attack.

The utter incompetence of this ‘Kindle attack’ article overwhelms me.

The Press are the Kindle’s biggest enemy and a huge threat. It’s good for Amazon that anti-Kindle attacks tend to be based on ‘a quick perusal’ of kindle reviews and such rather than actual research.

24 Responses

  1. I agree it’s unusually sloppy and I was one of the many commenters to his strange article. Mine was:

    ===
    “I imagine you did not know about the 1-star review for Amazon Kindle campaign by “Defective by Design Org” to protest DRM and anything else Amazon is doing wrong — the idea was to give 1-star reviews to the Kindle and explain to prospective customers why they should not buy a Kindle.

    Here’s an update they wrote after two such campaigns:

    http://www.defectivebydesign.org/blog/1249

    The headline there is “Update: hundreds of 1-star reviews added to Kindle page!
    Posted On: Wed, 2009-07-22 12:37 by holmesworcester”
    ===

    You’d think he would have pointed out what you did — that Amazon allows all the anti-DRM posts by non-Kindle owners to go up and be counted along with actual reviews.
    Maybe that’s to prevent charges of ‘censorship’ in a product-reviews area.

  2. [...] statistics the Time’s article was taken, has a rebuttal to the Bits Blog conclusions and The Review says: It’s a perfect example of why people are losing faith in newspapers.  Paul [...]

  3. I think it’s a mistake to say people are losing “faith” in newspapers. That’s a conflation. People love newspapers, they’re losing the willingness to pay for them when most of the content is syndicated and free. I see zero evidence that people think newspapers are losing credibility, and I think you’re making an outrageous leap here, precisely the thing you’re accusing the NYT of.

    • Well, after this article my faith in the New York Times goes from 90 out of 100 to 75 out of 100.
      That is losing faith.

      Paying money is directly related to trust.

      Newspapers WILL lose credibility with articles like this. Check on Techmeme for the Seth Godin article where he points out that the initial graphs showed 7% and 11% shares as more than what they would actually be on a pie-chart.
      That’s either dishonesty or sheer incompetence.

      • in my mind it is not so much that the article was written, you can write bad articles. But that the actual statistics as presented were inaccurate. This is exactly why I am losing faith in newspapers. I think we need journalists, but those that work for newspapers are ignoring their competencies and strengths and instead trying to compete with the freewheeling (and sensationalistic) Cable News on the one hand and focused and indepth work of (at least some) bloggers on the other. I don’t buy newspapers and I rarely read their websites unless linked by bloggers.

  4. Why can’t somebody who does not own a Kindle review it? Most consumer electronics are reviewed buy reviewers who don’t own one. They could have tried it.

    Can’t one base their review on why they would not choose to buy one? Would you review a car you test drove and say it’s too small? Probably.

    DRM is a serious problem. Why can’t I read my Kindle books on a better reader if one comes out?

    I own a Kindle, by the way, and I’d probably give it a 3-star rating. It’s a Kindle 2. The screen is like reading a soggy newspaper (to quote Nicholson Baker), and it’s too hard to get my own stuff on it. I enjoy reading my Kindle books on my iPhone more than the Kindle.

    • It’s more about what you consider sufficient experience to review a product.

      A reviewer reading 2-3 books on a kindle and then reviewing it is fine.

      There’s a difference between a review based on using a product and an attack based on philosophical differences.

  5. while I agree the article sucks, there’s some inaccuracies which need corrected.

    1. Sharing: Sharing books is allowed with kindle – up to 5 different kindles can read the same book. What is NOT clear to the general public is how that share works. You have to be on the same account to share a book. I can’t just say “here you can have this” and it automatically goes. Up to 5 kindles can be registered on the same account. That means, if I want you to share a book, I have to register you as my kindle, let you get the book and trust you not to use my credit card to order alot more. Obviously, only very close friends or family will EVER get an opportunity to share. Just as my husband reads my books and vice versa now, should he have his own kindle, he could still do so with ebooks.

    2. Customer service- My kindle broke inside. Not sure why, not sure how, but Amazon replaced it within 3 days! Awesome, awesome awesome customer service. I’d only had it about 3 months at the time.

    3. The Nook being sold out due to demand is not the detriment you mention, nor was it the first ereader to have this problem. When the Kindle was first released (i think it was 2007) it also had a problem with demand. It was sold out for six months straight, if I remember right. And this was just for those who followed the news – most people didn’t even know the Kindle existed yet. Then,last year, Oprah talked about it on the air, and it sold out again.

    As for loosing faith with newspapers, I’ve lost faith with the media in general. I’ve watched the tv news lie by omission in some cases, news I’ve already read online with details which explain the situation better. Reporters, both paper and television, seem to have the sensationalistic, fatalistic excitement down to a science, but in the process, forgot their purpose – Informing the public.

    That being said, I give my kindle 4.5 stars. :D

  6. It’s too bad that Amazon doesn’t indicate if the reviewer actually purchased the product or not from Amazon.

  7. I agree with much of the criticism of this specific article and I’d like to make two assertions, but won’t back them. (Color me too busy/lazy.)

    One: NYT is really trying to use new data sources for journalism. This is challenging, and they will succeed and fail as they learn more about this. NYT has been able to capture more collective attention from more people that I [come into contact with|notice mentioning via the web], than other journalistic endeavors in this regard, so far.

    Two: Expert domain specific knowledge is lacking from “general knowledge” journalistic content, as a rule, not as an exception. Nuance and “the finer points” are being edited out of articles to make them compete better with more ever crowded attention economic marketplace.

    Whenever someone like yourself encounters an article like this, it surfaces these aspects of the mass market journalism businesses.

    Embracing the pattern of “linking to the experts” is the only solution for this that I can see at this point in time.

  8. I own a Kindle DX and truly love it. It delivers what it advertises. My first love are paper books, but the Kindle definitely fits a need well in my reading desires.

    The NYT article does not surprise me in it’s lack of accuracy. Journalistic integrity seems to continue to erode at this newspaper. Is it due to a lack of intellectual critical analysis, laziness in research, lack of passion for accuracy? I don’t know, but it would seem to be an indicator of why the paper’s sales continue to decline and employees continue to be let go.

  9. The Kindle is a joke. It is black and white. It is awfuly expensive for a limited device that ONLY serves as an E-reader. It’s like all those gadgets you get from Sharper Image that ONLY play chess, or ONLY give you the local time and weather.

    • I love that it’s only an E-Reader. I don’t need all that other stuff, and why waste hard drive space with something I don’t need? E-Readers are for the power reader. Power readers are people who buy books by the dozens, and read them fast. I’ve been known to read three books in one day. Why would I want something else? I don’t want to have the small screens of the IPhone. I want something dedicated only to my books. Feel free to use the kindle for IPhone or whatever it’s called if you don’t want the Kindle. As they say, “There’s an app for that.” Enjoy it.

      I once had a cell phone provider tell me I needed a better phone, that I was too smart for the one I was holding. I told him I was too smart to waste my money on something I didn’t need, so don’t worry about it. And it’s true. My cell phone is an emergency use only thing. I’ve never gone over my minutes in the 7 years I’ve been using one.

      And with the Ereader (kindle), I can do what I want, when I want. I even purchased a nice purse that it fits in perfectly. (I rarely carry a purse, except when I want to take the kindle with me).

      I don’t need to play chess or get the local time or weather. I don’t need to use my kindle to look up local restaurants or browse the web. “There’s an app for that”, should I really decide that’s imperative for my life.

      Books and audio books are all I need it for, and it does a super job of both!

  10. [...] Amazon Working Backward? (NY Times) Learning from bad graphs and weak analysis (Seth’s Blog) NYTimes misuses Kindle Review stats to attack the Kindle (Kindle [...]

  11. This is the same thing that the Mac and iPhone has gone through with some blogs and even some major newspaper morons.

    These days, all the NY Times does is give their biased opinion as facts and only cover stories that they think make its point. And they are wondering why less and less people buy their rag!

  12. Wait, so am I not a customer if I don’t actually own a product? That’s not right. A customer is someone who is shopping in a store, whether they buy something or not. I agree that t’s impossible for someone to review a book or movie that they haven’t read yet, but they are completely entitled to review a product that doesn’t meet the standards they hope for, for example, a Kindle with bad design, or DRM problems.

    • There’s a difference between potential customer and actual customer.

      • No there isn’t. How can an actual customer buy something that doesn’t exist yet. Your argument doesn’t make sense. If ‘potential customers’ are complaining about the pre-purchase of the Kindle, then Amazon isn’t listening to their ‘potential customers’. If Amazon is only thinking about ‘actual customers’ their not going to grow and sell more products. So an actual customer and potential customer are the same exact thing. It’s called Economics 101.

      • You need actual customers to make money and potential customers to grow. Don’t know what’s more important. Didn’t study Economics which explains why your point eludes me.

  13. [...] NYTimes misuses Kindle Review stats to attack the Kindle The NYTimes’ Bits Blog has an article using Kindle Review statistics to attack the Kindle.  It’s a [...] [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 9,804 other followers