‘Digits’, a wristwatch-sized instrument panel capable of tracking and interpreting all hand gestures in 3D, enables the wearer to remotely control everything from a mobile phone to the TV.
A bracelet that enables you to remotely control your smartphone, television or any other electronic device – that’s basically what a team from Newcastle University and Microsoft Research (MSR) Cambridge in the UK has just come up with. The team has developed a system based on a set of sensors capable of tracking all free hand movements in three dimensions. The system then analyses and interprets the gestures in real time. Which means you will be able do away with your TV-remote and games controller and just give your instructions with a simple wave of the hand. The range of possibilities includes answering your mobile without taking it out of your pocket and adjusting the sound on your podcast without touching anything. The only thing you have to do first is programme the system to understand what action is called for by each hand gesture and tune it in to the specific device. The wristwatch-sized instrument panel, which the team has named ‘Digits’, “doesn’t rely on any external infrastructure so it’s completely mobile,” explained David Kim, an MSR-funded PhD student at Newcastle University*.
Interpreting every hand gesture
“This means users are not bound to a fixed space,” went on David Kim.“They can interact while moving from room to room or even running down the street.” In order to enable 3-D spatial interactions anywhere, the ‘Digits’ team had to overcome a number of difficulties: the device had to be light, consume little power, and be as small and comfortable as a watch. At the same time it had to be capable of high-precision gesture sensing and of ‘understanding’ the human hand, from wrist-orientation to the angle of each finger joint, so that interaction would not be limited to 3-D points in space. Basically the device needs to understand what the hand is trying to express, even while inside a pocket.
‘Digits’ approach could improve gaming systems
The current prototype, which was showcased at the 25th Association for Computing Machinery Symposium on User Interface Software and Technology on 7-10 October in Cambridge, Massachusetts, includes an infrared (IR) camera, an IR laser line generator, IR diffuse illuminator and an inertial measurement unit (IMU) track. One of the project’s main contributions is a real-time signal-processing pipeline that samples key parts of the hand, such as the tips and lower regions of each finger. Other important research achievements are two kinematic models that enable full reconstruction of hand poses from just five key points. The team all agree that the hardest challenge was “extrapolating natural-looking hand motions from a sparse sampling of the key points sensed by the camera.” They believe that one area of application for the ‘Digits’ device could well be in the video games field, as gaming systems do not currently provide very precise hand-sensing.
*The other team members are Otmar Hilliges, Shahram Izadi, Alex Butler and Jiawen Chen of MSR Cambridge; Iason Oikonomidis of Greece's Foundation for Research & Technology; and Professor Patrick Olivier of Newcastle University's Culture Lab.