Kindle Review – Kindle Fire Review, Kindle 4 Review

Update: After further inspection this new J. Bezos patent sounds like the basis of a completely new gesture based Kindle OS that skips the need for touch completely. Probably meant for Kindle 4 or Kindle 5. Read the details and it’s hard not to think that it’s cooler than touch. Do hope there’s still the keyboard for longer text entry.

TechFlash write about a new Amazon gesture recognition patent that would let users make purchases with a nod or a raised eyebrow. This is fascinating as it’s technology independent of eInk that could be used to advance the Kindle. It was filed in December 10th, 2008 so it’s unlikely but not impossible that it makes it into the Kindle 3. There’s a pretty good chance it does make it into the Kindle 4 (perhaps along with TouchCo’s multi-touch technology).

Kindle 4 gesture recognition – What is this gesture recognition patent?

The patent is titled ‘Movement Recognition as Input Mechanism’ and that’s what it literally is – Using your facial gestures as the new keyboard and buttons. Hope they still leave the keyboard in for writing notes and such.

1. It uses both relative motion and relative orientation (between user and device) to control aspects of the device.
2. Input can be provided by tilting the device, user moving their head, facial expressions, and similar actions.  
3. To be used to supplement or even replace keyboards and touchscreens.  
4. It was filed December 10th, 2008 and the inventor is Jeffrey P. Bezos. That pretty much guarantees it’s Amazon’s patent.
5. It discusses replacing pretty much every aspect of using the Kindle with gesture recognition.

First, let’s confirm this really is for the Kindle.

How do we know this is for the Kindle? 

Well,

• It mentions electronic reader device multiple times as an example of where the technology could be used.
• It desribes buying things from an eletronic store that sounds awfully like the Kindle Store.
• It talks about being a way to interact with the graphical user interface on a portable computing device and the Kindle is the only one Amazon has.
• It talks about navigating file systems.
• It talks about navigating an electronic catalog.
• It talks about selecting an alphanumeric character – exactly what the SYM button currently does.
• It talks about limiting cursor movement in exactly the way it’s limited when you jump into Kindle Menus and the Aa Font Options Menu.

While the patent covers all devices (even PCs and TVs) it does seem primarily focused on the Kindle.

Consider these two snippets -

… a user might utilize a device such as an electronic book reader or a portable media device to read a book, newspaper, or other such electronic content. When viewing such material, the user might desire to navigate to or “turn” to the next page of an electronic book (e-book), and might perform an action, such as tilting the device or turning the user’s head to the right …

There also can be one or more physical buttons, selection wheels, touch interfaces, and the like on the device that can be used in combination with the relative movements for input. For example, a user might tilt the device to the right to cause the electronic book reader to start turning pages of a book, and a user can press a button to indicate that the desired page is being displayed, signaling the device to stop paging through the e-book.

It certainly seems like it’s custom-built for the Kindle.

Kindle Gesture Recognition – The Details

Further Details on the Kindle Gesture Recognition Patent

There are lots and lots of interesting details -

1. You can shake the device to trigger the gesture recognition.
2. The measured aspects include – dimensions of one or both of the user’s eyes, separation of features on the user’s face, relative location of a feature on the user’s face, relative orientation between features on the user’s face.
3. You can flip orientation via gestures. Guessing that a slight nod to the left would skip the screen from landscape to portrait mode (Update: Actually that would function as Previous Page).
4. It talks about moving a cursor via gestures.
5. It also talks about adjusting point of view to create new 2D image of a 3D object. That’s pretty un-Kindle like.
6. You can stack up two gestures and in some cases those get interpreted as a different gesture. 
7. Gestures can be used for navigating between items in an electronic catalog, navigating a file system, and selecting items from a rotating view.
8. It talks about selecting one out of a set of characters and then selecting a single character out of that set of characters – seems like the basis for searching through books or files.
9. You can change orientation to select an item.
10. Orientation detection is via one of – an accelerometer, a motion detector, an electronic compass, a GPS receiver, and an electronic level. 
11. There’s visual feedback to indicate what element you’re on. It will be one of - a cursor over it, color changes, shape changes, background color changes, a shape being displayed around it.
12. There are certain thresholds – so very tiny movements don’t trigger anything. Guessing this is to avoid accidental changes.
13. There might be one feature linked to each of your eyes, your nose, your mouth, and your eyebrows. This might end up being hilarious to watch.
14. There is a gesture corresponding to viewing and selecting an item.
15. User recognition via the image – Hints at biometric security.
16. There’s also a very strong hint of user profiles and being able to switch to a user’s settings when she/he is identified.
17. You can confirm actions via gestures. So you could raise your left eyebrow to delete a file and then nod to confirm the deletion.
18. For some reason get the feeling that Amazon are going to stack together multiple gesture and orientation recognition means (perhaps electronic level plus accelerometer plus two cameras) and create a ‘most likely gesture’ output. It would make sense to use two cameras to get a composite image.
19. 43 is the last and most important detail – The imagine element is a high-speed video camera. How appropriate.

This patent has gotten me all excited – If this makes its way into Kindle 4 it’ll destroy the competition.

Kindle 4 Gesture Recognition Patent – Even more details

This is a really long, detailed patent and it seems like Amazon have put in a ton of effort and covered almost every aspect of the Kindle’s functionality. It could also form the basis of a very effective mobile OS.

Some further details -

1. There’s talk of using gestures to navigate an electronic map system.
2. Talk of having a 2-D view of a 3D system comes up multiple times including alongside talk of a rotating selection view - It just screams 3D operating system.
3. There is talk of the camera being either a high-speed camera or a high-speed video camera.
4. There is a lot of talk of not using the images themselves and only the changes between images.
5. Tilt the device left or right to move the cursor across a row of letters. Tilt the device up or down to move the cursor over a column of letters. Pretty cool.
6. More talk of viewing a 3D image and talk of ‘looking into a box’ and being able to see the top, bottom, side, and interior of the box by rotating the box (by changing how you look at it).
7. Turn your head to the right to go to the next page.
8. There’s talk of combining the visual input with physical buttons which makes me happy. A gesture to start turning pages and a button to pause it when you get to the page you want.
9. The Kindle Store will be displayed as a three-dimensional grouping by category -
   

   a user is browsing items offered through an electronic marketplace, such as via a website of a Web-based retailer, the user might view a number of items corresponding to a search request response or category of items. If the items are displayed in a three-dimensional grouping by category, for example, the user can tilt or move the device to zoom or otherwise move to the level of the desired category

   if the user wishes to purchase an item, the user can perform an action such as nodding his or her head up and down to indicate that the user would like to purchase the item.

10. In the browser you can do lots of actions – tilt up to scroll up, tilt down to scroll down, bring device closer to zoom in, etc.
11. There’s lots of talk of using just the camera or of using the camera in conjunction with an accelerometer.
12. Not sure how this doesn’t run into conflict with Microsoft’s Project Natal for Xbox 360. There’s talk of Amazon wanting to use this patent for phones, PCs, laptops, TVs, PDAs, eReaders, video game systems, and portable media players.
13. The imaging element (camera or video camera) rotates to follow the user’s face and make sure it’s capturing gestures.
14. Different applications can use different interpretations for the same gestures.
15. Some apps can use just the accelerometer and leave out the camera input.
16. Camera input can be turned off.
17. Kindle 4 could include an additional input method – a push button, touch pad, touch screen, wheel, joystick, keyboard, mouse, keypad, or something else.
18. Kindle 4 might include a microphone to capture voice commands or other audio commands.
19. A predictive text algorithm to help users with text entry.
20. Talk of a third dimension that can be used to activate Shift, Alt, and Caps Lock. Also talk of infinite positions along that 3rd dimension.
21. There’s allowance for being in a moving vehicle or in lower visibility situations – It segments the screen into parts and lets the user progressively drill into more and more specific regions.
22. The Kindle store seems to be this giant 3D globe you can navigate around by moving the Kindle. There is talk of using either a button or a touchscreen or a gesture or a cursor to spin/scroll/rotate the globe.
23. The 3D globe could represent (in addition to the Kindle Store) the chapters in a book, a user’s social graph, search results, a file system hierarchy, and other potential uses.
24. Lots of talk of copying, moving, and renaming files and 3 dimensional and conventional folder structures.
25. The patent describes a rotating carousel display and a mechanism to select items from such a display.
26. There’s a good description of how gestures would work with maps including being able to see a store’s inventory available for purchase -
    

    … users can pan across maps by tilting the device (and camera) in various directions. Additionally, users can zoom in and out to reveal more map detail by moving the device closer or farther away from the user …

    … the user can zoom into an increasingly detailed view of a selected point. In some cases, the user may be able to actually view an image of the front of buildings or even peer into windows of the businesses, such as stores, to navigate within the store and view inventory available for purchase.

27. Talk of wrist flicks or tilts of the head to turn pages.
28. There are lots of degrees of freedom – moving the device laterally, moving the device vertically, tilting the device up or down (detected via an accelerometer), rotating the device left or right, and so forth. They really plan on making it very flexible and powerful - The gesture recognition patent says there will be at least six degrees of freedom.
29. They talk about using complex motions – using the movement in the six degrees of freedom with other motions, expressions or inputs. An example they give is tilting right, left, and right to bring up your shopping cart.
30. The Patent includes the provision to detect a variety of motions using just a camera.
31. There seems to be an auto-scroll function spelled out pretty clearly including controlling the speed of scrolling via the angle of tilt of the Kindle.
32. It discusses the capability to detect expressions like smiles and frowns.
33. Users would be able to hold up a number of fingers to indicate number and can draw images/alphabets with their fingers. It also allows for specific motions that indicate home or another button.
34. Users could set passwords – three nods up and down + a smile + two eyebrow raises. It seems a bit complicated.
35. There’s also a provision for eye-tracking where the movement of the pupils relative to the whites of a user’s eyes is used to figure out where the user is looking.
36. Perhaps the coolest gesture is you close your eyes for a few seconds and send the Kindle to sleep.
37. Users can look at an item in a list and blink twice to select it. Look at a map and a particular state and say Zoom to zoom into it. Lots of very cool shortcuts and gestures.
38. You can set up macros – a series of actions that represent a particular action.
39. There is provision for apps and for allowing apps written in various computer programming and scripting languages including Java, C, C#, C++, Perl, and Python. There are multiple mentions of apps.
40. There is, at the very end, talk of connecting devices to a wireless network and to various servers.

This is one amazing patent – hopefully it makes its way on to a Kindle soon.

Ramifications of a Kindle 4 with gesture recognition

Keyboard to Gesture Recognition – Skipping Touch Completely

This patent hints that Amazon might be considering skipping the touch revolution completely and moving on to the next higher level of user input.

Talk of using gestures to move the cursor certainly hints at a complete input system. As does talk of using gestures to navigate a file system. It also talks about jumping to a point on the screen based on eye positioning – which almost makes touch unnecessary.

It seems a bit awkward to look at a point on the screen and to then do a gesture to indicate you want to touch (let’s say blink your eyes). However, in practice, it’ll be much faster than moving your finger.

It’d be one heck of a way to bypass all the touchscreen mania and make it seem outdated.

A Kindle 4 with gesture recognition would let Amazon capture user emotions

No idea if they would do it – However, they could tell exactly what a user thinks of a book. The points at which you laugh, the points at which you are bored, and how you feel at pretty much every point in the book.

It’d be the sort of wealth in user review information that traditional Publishers and bookstores can’t even imagine. It would also provide the patterns of successful books i.e. what chain of emotions does a really, really good book put users through. Amazon would literally have a blueprint for how to make a book a great one or how to make it sell well or how to appeal to a particular set of users.

If Amazon can get this working it’s game over for Publishers – Well, it already is threatening to be. However, with the success patterns and emotion patterns of books that a gesture based interface allows Amazon would have an unmatchable advantage. Amazon Encore could, in the best case, construct bestsellers at will and, in the worse case, dramatically increase the probability of success.

Filed under: kindle 4 Tagged: | kindle 4, kindle gesture recognition