You can use remote gestures to control your laptop, your video game console and your TV. So why not the device you use more than any other?

In a similar fashion to how Microsoft’s Kinect brought gesture controls to the Xbox, PointGrab, using a different technology, is trying to bring hands-free, gesture-based control to iOS games and apps. Gestures can be easily learned and perhaps even more natural than learned input mechanisms like tap to zoom or pinch to enlarge, PointGrab VP of Marketing Assaf Gad argues.

“The same way we as people interact with each other, we can allow devices to understand our body language,” he told me via Skype from PointGrab’s headquarters in Hod Hasharon, Israel last week.

For example, he said, all children know the finger-to-lips “shush” gesture that means “quiet down.” So, rather than hunting for a button or volume menu to lower the volume, why can’t our devices know that action means we want the sound muted?

Motion control can be natural and fun — hands-free Angry Birds, anyone? — but occasionally necessary too: devices that understand remote gestures would let you, for example, scroll down the recipe you have open on your iPad while your hands are still wet or covered in raw egg.

How it works

PointGrab doesn’t require additional hardware to make this happen — it uses the device’s own camera. PointGrab’s technology uses motion-detecting algorithms to identify the X and Y coordinates of a person’s hand through the camera. Gad says the software can detect a single finger up to 17 feet away. Here’s a video Laptop Mag shot at CES 2013 of a free app PointGrab created simply to demonstrate its technology on an iPhone or iPad.

You can also download that app, called CamMe, from the iOS App Store yourself. I tried it out and it works as promised although it’s occasionally a bit slow: if you want a self-portrait or group photo but don’t have Inspector Gadget arms, you can use this by setting your device on something, raising your hand to turn on the camera from a couple feet away and closing your fist when you’re ready. The app counts down three seconds before snapping the photo. The downside is you’re using the front-facing camera, which isn’t as good as the iPhone’s rear camera.

Gad emphasizes that this isn’t supposed to replace a touchscreen altogether but is for use in certain cases: “We’re extending a user interface for specific scenarios in mobile.”

A gesture-controlled future

But there’s growing momentum in this space. In the living room, it’s already moved from futuristic fantasy to Thing Normal People Use, thanks to Microsoft’s success with the Kinect. And Samsung, the world’s largest TV maker, wants to replace your remote with your hand when controlling its new smart TVs.

But the most high profile up-and-comer now trying to make gesture-based computing happen for a broader user segment is Leap Motion, whose mission is to bring 3D motion control to computers. (Its website contains the rather cheeky phrase, “Typing? Seriously? That’s fine for writing a novel.”) It recently got $30 million in venture funding to try to pull this off.

PointGrab, meanwhile, has been around since 2008. Its gesture-control software has been in Fujitsu laptops since 2010, and the company has worked with Acer and Samsung on laptops too. But PointGrab is looking past laptops now; it wisely sees the future is more mobile and is attempting to get developers interested in using the technology in their iOS apps.

The thing LeapMotion has going for it that PointGrab does not yet is active third-party developer interest — there are 12,000 developers working with LeapMotion’s SDK. But PointGrab hopes to change that soon. This week at Mobile World Congress in Barcelona, PointGrab is for the first time opening up its SDK to anyone who wants to give it a whirl.

Gad is particularly hoping for forward-thinking developers who aren’t satisfied with only touch or voice-based control of the iPhone or iPad. “I think existing applications always try to find another way for interaction or another way to suggest new features to the user. I believe our technology will ad another capability.”

PointGrab’s SDK will be free at first for those who have “good ideas” for how to apply the software to their apps.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Survey: How apps can solve photo management
• Analyzing the wearable computing market
• Carrier IQ and the continued erosion of operator trust

Even without the SDK, Microsoft’s Kinect is successful and forward thinking in its own right. When paired with the company’s Xbox 360 console, Kinect gets gamers off the couch and actually “into” the game. How so? The $150 gadget combines infrared sensors, a color camera, four microphones and its own tilting mechanism to see and hear game players in detail. This data is sent to the game console over a standard USB port where software interprets players actions in real-time and is used to control the on-screen game. Our first look video shows how this works in a two-player obstacle course game.

As fun (and as exhausting) as game play is with the Kinect, the technologies it provides for a relatively low cost, combined with the standard USB port, are what could make Kinect even more successful. Not long after the device debut last November, hackers and enthusiasts realized Microsoft wasn’t encrypting the Kinect’s data; users could access the information by connecting a computer to the device. That led to some amazing 3-D imagery from two Kinect units, homebrewed holographic television from M.I.T., and even an inexpensive robot kit that pairs Kinect with a netbook and basic Roomba device.

Bear in mind that these and many other Kinect hacks are just that: hacks. All were done without Microsoft providing any software or drivers to put the pieces together. Of course, that changed Thursday with the official SDK available for download, which includes these features:

• Raw sensor streams. Developers can see and use the raw data from Kinect’s depth sensor, color camera sensor, and microphone array. This means apps can “see” and “hear,” perhaps leading to interpretation applications for the hearing or vision-impaired.
• Skeletal tracking. Images of one or two people at a time are supported and can be used for new gesture-based user interfaces or virtual wardrobe applications, for example.
• Advanced audio capabilities. Kinect’s SDK can take advantage of both advanced noise cancellation and the Windows speech recognition engine. Natural language commands and queries for computers, appliances or web-connected devices could arise.

One sample game included with the Kinect SDK illustrates a simple game that leverages the skeletal tracking and voice recognition features.Up to two players interact with falling geometric shapes such as stars and triangles. Shouting out a color changes the color of the shapes on the screen. More impressive to me were some of the Code Camp efforts that third-party developers created in under 24 hours with the new Kinect SDK. Earlier Thursday, Microsoft live-streamed some of these and I saw a flying toy helicopter controlled via Kinect. Raising one’s hands caused the helicopter to rise, while turning one’s body controlled left and right movement.

But toy flying machines are simply an example of Kinect’s potential. The “killer app” of Kinect isn’t one specific application; it’s the potential for a limitless amount of applications that bridge the physical world with the computing world. Think beyond the decades-old mouse and keyboard model for input. Consider not just video games, but interactive video movies that allow you to be part of the show. And perhaps Kinect-powered applications can power not just robots, but sensory input and interaction for those that aren’t physically able to use computing devices.

Without a doubt, the Kinect hardware is a runaway hit, selling faster than even Apple’s iPad 2. But like the iPad, it’s the software that could revolutionize the way people think of and interact with the world of computers. Thanks to early efforts by the hacker community, Microsoft has recognized the potential of its amazing new device and unlocked the doors to a world of new applications that can help merge physicality with computing. And the company has made it easy too: Just minutes after downloading the SDK and plugging my Kinect into a computer, I’ve been “Kinnectified”!

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from connected consumer’s second quarter
• Connected Consumer Q2: Digital music meets the cloud; e-book growth explodes
• Analyzing the wearable computing market

Some of the milestones come from Patrick Baudisch, who’s spent years integrating back-of-device user interface interaction into mobile gadgets. Baudisch’s latest effort is called Nano Touch, which as you can see in the video below, allows for touch-based interface input on the rear of a small device — as opposed to on the front display. This brings two benefits: One, it expands the total amount of device real estate that the user can interact with; and two, the user’s own fingers doesn’t have to occlude the display while input is taking place.

http://www.youtube.com/v/4xfgZy2B5ro&hl=en_US&fs=1&

According to Baudisch’s research page for the Nano Touch project, this back-of-device interaction would be especially applicable for very small devices, including touch-capable electronic jewelry.

Baudisch isn’t the only researcher who sees the value of a backdoor. This weekend at the Siggraph Asia show in Yokohama, Japan, MIT researchers will demo a prototype of their hand gesture recognition technology for liquid crystal displays (LCDs). It works by integrating a layer of sensors on the reverse side of a display. The sensors take regular snapshots of hand movements made in front of the display to generate a 3-D model, which is then translated into directions by the device. A video of how this works is below:

http://c.brightcove.com/services/viewer/federated_f9/36804639001?isVid=1&publisherID=36545971001

If efforts to imbue the reverse side of devices and displays with user interface options succeed, they could have a profound impact on mobile applications. Mobile browsers, for example, could include off-screen elements and controls that could be accessed from the rear of a device without disrupting work going up front. That’s called making the most of your mobile gadget.

Image courtesy of Patrick Baudisch.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Survey: How apps can solve photo management
• Social networks will displace business processes, not socialize them