Jakob Nielsen runs a Kindle vs iPad vs Physical Book reading speed test that totally loses the plot. The results of the test are that the Kindle is 10.7% slower than print and iPad is 6.2% slower than print.
Users at the official Kindle forum are up in arms and quite rightly point out that reading on the Kindle is actually faster. Let’s explore why Mr. Nielsen’s test is biased against the Kindle and see what interesting observations we can make.
Decades of Experience with Books Vs 17.33 minutes with the Kindle
Let’s start with the obvious problems -
- It’s 24 people reading stories that took an average of 17.33 minutes to read. That by itself is a good indicator of the unsuitability of this ‘reading speed comparison’. How can you use 17.33 minutes of reading to contrast reading speed on books which usually take 3-6 hours to read?
- The people recruited liked reading and frequently read books. There’s no information on whether they’d used the Kindle, the iPad, or iPhones before. Chances are high that these are people who had their first experience with the Kindle – You’re literally comparing reading speed during first 17 minutes of using a device with reading speed on physical books which they’ve been using for decades.
- They compared customer satisfaction levels – After people read 17 minute short stories. Perhaps that’s what Amazon’s Kindle return period should be – You get 17 minutes to read a short story when your Kindle first arrives. After that you have to instantly decide – You either keep it for life or the UPS guy will take it back with him.
- Another really important factor missing is lighting conditions – Was there bright lighting? Did they read in sunshine? Did they read in the dark with a reading light?
- They say that the difference between the 6.2% lower reading speed of iPad (as compared to the print book) and 10.7% lower reading speed of Kindle (again, it’s 10.7% lower than reading speed on the print book) is not statistically significant. If a 4.5% difference isn’t statistically significant then how much more significant is a 6.2% difference? What’s the exact margin of error?
They conclude -
tablets still haven’t beaten the printed book: the difference between Kindle and the book was significant at the p<.01 level, and the difference between iPad and the book was marginally significant at p=.06.
Wouldn’t the more accurate statement be – Tablets like the iPad and eReaders like the Kindle don’t beat physical books if the user has used them for a grand total of 17 minutes.
A comment from Robin at the kindle forum is very apt -
One thing that popped out at me is that they were testing people who were evidently unfamiliar with either the iPad or Kindle and who normally read paper books.
Even though they taught the test subjects how to operate the devices before running the actual testing, I think it’s a bit absurd to think that a reader would adjust that quickly to using something new.
It’s very unlike Jakob Nielsen to do such a remarkably poor usability study – There’s absolutely no way you can review reading speed on the Kindle and user satisfaction for a user when it’s the user’s first 17 minutes with the Kindle.
Kindle Specific things Users can’t be expected to grasp in 17 minutes
Just to highlight how absurd this comparison is here are a few things a user is not going to be able to figure out in the first few minutes of using the Kindle -
- How best to work with Page Turns. Most of us know just when to press the Page Turn button to catch the last line just as the page refreshes – This, in fact, makes it much faster to turn pages on the Kindle than on a physical book.
- What they feel about eInk and what lighting condition goes best i.e. the more the better. If they compared the iPad and the Kindle in similar lighting conditions one of the devices probably got read in non-optimal lighting conditions.
- What’s the most comfortable Font Size for them. They probably didn’t even realize that they could set line spacing and the number of words per line.
- How best to hold the Kindle and where to place hands. If they were moving their entire hand every time they had to flip a page that’s adding a second per page.
There are additional reasons this study is biased against the Kindle, especially when it comes to user satisfaction, including –
- No tiredness comes into play. In 17 minutes hardly anyone is going to realize that books and the Kindle are much better for the eyes – unless of course you’re LCD compatible.
- You don’t get to see a lot of Kindle benefits like the in-built dictionary, saving of last page read, searching the book, searching Wikipedia or the Internet.
- Users don’t know that they can continue their reading on their Android phone or iPhone or Blackberry and on their PC and then continue it on the Kindle.
- Users won’t know that their brother or mother or husband can read the same book at the same time on another Kindle or device.
- You don’t get a chance to switch back and forth between devices and compare. Ideally you’d want people to read 1 or 2 books on each of the devices – PC, physical book, iPad, and Kindle.
Interestingly enough this 24 person, 17 minute reading speed comparison is being picked up by the Press as a sign that reading on the Kindle is slow or in some way inferior to reading a physical book. The Press just love to write negative things about the Kindle.
Looking beyond Reading Speed on the Kindle
There are a few things much bigger than an imaginary 10% difference in reading speed to consider -
- Lots of people need the Kindle’s larger font sizes and text to speech feature. This includes blind people, low vision people, those not comfortable with the size in printed books, dyslexic children, and even most young children as they prefer size 24 and size 28 font.
- People are now reading more as they can get books easily and quickly and aren’t distracted.
- The Kindle probably leads to longer stretches of reading. It’s much lighter than a hardcover, it can be read using a single hand, you can use the text to speech, and it’s easy on the eyes.
Here’s Jakob Nielsen’s study summary -
A study of people reading long-form text on tablets finds higher reading speeds than in the past, but they’re still slower than reading print.
Here’s my Summary of the study -
24 people tried out the Kindle and iPad for 17 minutes each and then an expert made the assumption that their observations are the holy grail and that ‘reading speed on the Kindle is slower than on physical books’.
Various Blogs and newspapers read the above summary in 17 seconds, spent another 43 seconds to quickly scan for a paragraph they could quote and then wrote about how physical books are better than the Kindle.
It’s 2.5 years since the Kindle was launched and the Press and various ‘Experts’ thought Amazon would be lucky to sell 10,000 Kindles. We might have had millions of Kindles sold but as far as the Press and Experts are concerned – not much has changed.