Test comparing Kindle, Book, iPad reading speeds loses the plot

Jakob Nielsen runs a Kindle vs iPad vs Physical Book reading speed test that totally loses the plot. The results of the test are that the Kindle is 10.7% slower than print and iPad is 6.2% slower than print.

Users at the official Kindle forum are up in arms and quite rightly point out that reading on the Kindle is actually faster. Let’s explore why Mr. Nielsen’s test is biased against the Kindle and see what interesting observations we can make.

Decades of Experience with Books Vs 17.33 minutes with the Kindle

Let’s start with the obvious problems -

  1. It’s 24 people reading stories that took an average of 17.33 minutes to read. That by itself is a good indicator of the unsuitability of this ‘reading speed comparison’. How can you use 17.33 minutes of reading to contrast reading speed on books which usually take 3-6 hours to read?
  2. The people recruited liked reading and frequently read books. There’s no information on whether they’d used the Kindle, the iPad, or iPhones before. Chances are high that these are people who had their first experience with the Kindle – You’re literally comparing reading speed during first 17 minutes of using a device with reading speed on physical books which they’ve been using for decades.
  3. They compared customer satisfaction levels – After people read 17 minute short stories. Perhaps that’s what Amazon’s Kindle return period should be – You get 17 minutes to read a short story when your Kindle first arrives. After that you have to instantly decide – You either keep it for life or the UPS guy will take it back with him.
  4. Another really important factor missing is lighting conditions – Was there bright lighting? Did they read in sunshine? Did they read in the dark with a reading light?
  5. They say that the difference between the 6.2% lower reading speed of iPad (as compared to the print book) and 10.7% lower reading speed of Kindle (again, it’s 10.7% lower than reading speed on the print book) is not statistically significant. If a 4.5% difference isn’t statistically significant then how much more significant is a 6.2% difference? What’s the exact margin of error? 

They conclude -

 tablets still haven’t beaten the printed book: the difference between Kindle and the book was significant at the p<.01 level, and the difference between iPad and the book was marginally significant at p=.06.

Really?

Wouldn’t the more accurate statement be - Tablets like the iPad and eReaders like the Kindle don’t beat physical books if the user has used them for a grand total of 17 minutes.

A comment from Robin at the kindle forum is very apt -

One thing that popped out at me is that they were testing people who were evidently unfamiliar with either the iPad or Kindle and who normally read paper books.

Even though they taught the test subjects how to operate the devices before running the actual testing, I think it’s a bit absurd to think that a reader would adjust that quickly to using something new.

It’s very unlike Jakob Nielsen to do such a remarkably poor usability study – There’s absolutely no way you can review reading speed on the Kindle and user satisfaction for a user when it’s the user’s first 17 minutes with the Kindle.   

Kindle Specific things Users can’t be expected to grasp in 17 minutes

Just to highlight how absurd this comparison is here are a few things a user is not going to be able to figure out in the first few minutes of using the Kindle -

  1. How best to work with Page Turns. Most of us know just when to press the Page Turn button to catch the last line just as the page refreshes – This, in fact, makes it much faster to turn pages on the Kindle than on a physical book.
  2. What they feel about eInk and what lighting condition goes best i.e. the more the better. If they compared the iPad and the Kindle in similar lighting conditions one of the devices probably got read in non-optimal lighting conditions.
  3. What’s the most comfortable Font Size for them. They probably didn’t even realize that they could set line spacing and the number of words per line.
  4. How best to hold the Kindle and where to place hands. If they were moving their entire hand every time they had to flip a page that’s adding a second per page.

There are additional reasons this study is biased against the Kindle, especially when it comes to user satisfaction, including - 

  1. No tiredness comes into play. In 17 minutes hardly anyone is going to realize that books and the Kindle are much better for the eyes – unless of course you’re LCD compatible.
  2. You don’t get to see a lot of Kindle benefits like the in-built dictionary, saving of last page read, searching the book, searching Wikipedia or the Internet.
  3. Users don’t know that they can continue their reading on their Android phone or iPhone or Blackberry and on their PC and then continue it on the Kindle.
  4. Users won’t know that their brother or mother or husband can read the same book at the same time on another Kindle or device.
  5. You don’t get a chance to switch back and forth between devices and compare. Ideally you’d want people to read 1 or 2 books on each of the devices – PC, physical book, iPad, and Kindle.

Interestingly enough this 24 person, 17 minute reading speed comparison is being picked up by the Press as a sign that reading on the Kindle is slow or in some way inferior to reading a physical book. The Press just love to write negative things about the Kindle.

Looking beyond Reading Speed on the Kindle

There are a few things much bigger than an imaginary 10% difference in reading speed to consider -

  • Lots of people need the Kindle’s larger font sizes and text to speech feature. This includes blind people, low vision people, those not comfortable with the size in printed books, dyslexic children, and even most young children as they prefer size 24 and size 28 font.
  • People are now reading more as they can get books easily and quickly and aren’t distracted.
  • The Kindle probably leads to longer stretches of reading. It’s much lighter than a hardcover, it can be read using a single hand, you can use the text to speech, and it’s easy on the eyes.

Here’s Jakob Nielsen’s study summary -

 Summary:
A study of people reading long-form text on tablets finds higher reading speeds than in the past, but they’re still slower than reading print.

Here’s my Summary of the study -

24 people tried out the Kindle and iPad for 17 minutes each and then an expert made the assumption that their observations are the holy grail and that ‘reading speed on the Kindle is slower than on physical books’.

Various Blogs and newspapers read the above summary in 17 seconds, spent another 43 seconds to quickly scan for a paragraph they could quote and then wrote about how physical books are better than the Kindle.

It’s 2.5 years since the Kindle was launched and the Press and various ‘Experts’ thought Amazon would be lucky to sell 10,000 Kindles. We might have had millions of Kindles sold but as far as the Press and Experts are concerned – not much has changed.

9 Responses

  1. As we all know, Amazon’s claim that the object disappears in your hand so that you can dive into the author’s world is true. However, when encountering something new, as well as the functional stumbling, we will also be distracted by thoughts such as ‘Wow! This is pretty cool. I could get used to this. I wonder what it smells like?’

    Our mind is likely to wander from the task.

    I agree… this test (as reported) is hardly rigorous and probably useless.

  2. It’s not reading, it’s a joke, until it’s comparing reading 1,000 pages + on the one device, and/or/after the other.

    If you don’t read 1,000 pages or more a month, don’t bother with the Kindle, you’re a skimmer or a dilettante, not a reader.,

  3. I think you make lots of valid points, but the tone here seems a bit outraged.

    I love my Kindle, but when I read the test, the immediate thing for me was how small the sample size was. 32 people reduced to 24 isn’t really enough to find anything important, even if the test had been constructed well.

    But, in essence, the test is flawed by choosing short stories — people who love reading books don’t necessarily love reading short stories.

    I sure don’t!

    And I think there must be a way to make a test that would include people choosing their own books and reading for a few hours in various conditions. I’m not a test designer, but something like that with a larger test group would probably give us more interesting data and more authoritative conclusions.

    Of course, I’m not sure where the money for a good test would come from that would seem neutral.

    • It’s just that Jakob Nielsen is THE usability guru in my opinion. Have linked to his posts so often. Then he goes and does a test so half-heartedly. Mortified that it’s on TechMeme and now everyone will think I hate him when it’s the opposite – I respect his expertise so it’s really surprising to see this pointless comparison.

  4. I own an iPad and a Kindle 2 and there is no explicable reason for the Kindle to be slower – except that, as noted, we who are experienced readers know when to hit the page turn button and new users are going to make mistakes experienced users won’t. Remember hitting the page back button instead of forward? How about accidentally holding the forward button or the home or menu button? Having both a larger display and turning pages faster, it is easy to see why the iPad was faster.

    But still – why slower on the iPad? Newness explains some of it but. Is that all there was? I wonder if the dictionary came into play. How often have you been reading and came across an unfamiliar word where context let you probably guess the meaning. Did you look it up when you had to grab a dictionary? How about with the Kindle or an iPad reader?

  5. [...] Test comparing Kindle, Book, iPad reading speeds loses the plot Jakob Nielsen runs a Kindle vs iPad vs Physical Book reading speed test that totally loses the plot. The results of the [...] [...]

  6. To me, it was a VERY interesting study. Would it be nicer to see 100 people? Sure, but who is footing the bill? Also, while 17 minutes is arguably on the short side, it’s on the short side of 30 minutes or 45 minutes… that a book may take 4 to 8 hours to read is not relevant. Very few get the chance to sit down and do that. I probably read 30-45 minutes at night, and in smaller amounts throughout the day.

    I also think a 10% swing in reading speed is… meaningless. I’m a former Kindle 2 user (current iPad), and in both cases, the absolute convenience outweighs any difference in how fast I can read. That said, there is a sure delay on page turning in eInk screens, and it’s disingenuous to factor in hitting the button a line in advance – it’s still a disadvantage… not one that can’t be mitigated, but it’s a drawback. Besides, many a time have I done that, and had to go back, having not quite finished on schedule that sentence.

    To be honest, there really isn’t a huge eye fatigue factor for most people. Yes, at the beach, there’s a real disadvantage with the iPad, but it’s no different than at night when the glare from a booklight is a disadvantage with the Kindle… except I read at night more than I’m reading outside.

    I read your site regularly, but to be honest, it’s hard sometimes, since your opinions are often rendered as fact, and your bias towards the Kindle is quite obvious. It’s just not the perfect device, nor is it perfect for everybody.

    To the commenter who said people should choose their own books – that wouldn’t work, as reading speed can vary by the book material itself. Reading a science heavy research book is going to be slower reading than the latest John Grisham book. Rereading something I’ve already read will have me reading faster, arguably at the expense of comprehension.

    I’d like to see about 25-30 people reading an hour at a time, for three sessions. The sessions can split by material, maybe a light fiction novel, a light biography, and something rather heavy.

  7. [...] Nielsen finds that participants read more slowly on a Kindle or an iPad than on a traditional book. Abhi raises some reasonable questions about the findings. 2. A 2006 Amazon patent comes to light and may [...]

  8. [...] looked at Jakob Nielsen’s 24 people, 17 minutes study and how it’s pretty much useless. However, there have been thousands of articles using it to [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 5,228 other followers