Blog Post

Envisioning Sketch Interpretation on Apple’s iPad and Other Tablets

From the Apple Newton (s aapl) to Palm’s Graffiti (s palm) to modern day Microsoft Windows Tablet PCs (s msft) and Apple’s forthcoming iPad, people have looked at digital inking as another form of input. The keyboard and mouse are far more common both for typing text and navigating around today, but maybe there’s life beyond handwriting for ink solutions. MIT is looking past the traditional uses for digital ink with their sketch-interpreting software.

[W]hile a drawing can be rich in information, it’s information that’s usually inaccessible to computers. If you draw a diagram on the screen of a tablet computer, like the new Apple iPad, the computer can of course store the drawing as an image. But it can’t tell what the image means.

The video demo shows a practical application — practical if you’re a chemist, that is. Sketching the molecular composition of a compound is easy enough on a display, but getting the computer to recognize and identify the sketch is the challenge. MIT’s software does just that however, and once the sketch is identified, accessing related data is simple. How does the recognition work? The solution combines what was drawn along with how it was drawn — it’s not difficult to determine if a stroke was made from left to right or up to down, for example. Those two data sets further break down into individual elements of the sketch and all of the information is compared to a database for recognition.

As impressive as this feat is, it also highlights the challenge that’s plagued slate tablets for years and relegated them mainly to vertical niche markets. Sketches and drawings can be interpreted into a nearly infinite number of items, while text is simply text. With standard keyboard input or even handwritten text, the input generally means one thing. Freehand drawing — while liberating — could be an image, a standard figure or interpretation of one, a molecular compound, a math equation, or an emoticon to name a few. And that output variance for ink usage is where the challenge lies. Each “object” drawn could have many different contexts or meanings. Accounting for all of those permutations and combinations through recognition software is something the personal computing industry simply isn’t equipped for just yet. Instead, there are more and more custom solutions designed to recognize specific object types, which points those solutions at specific vertical markets.

Should you expect to see an iPad application from the folks at MIT? That’s highly doubtful. Might we see some similar, generic solutions on tablets in the future, though? As this type of research blossoms out of educational halls and is adapted by software developers, it should come to market. But it’s going to take a smart set of algorithms and the hardware to power them before you can freely draw anything on a tablet and have the device recognize it for what it is.

Related research on GigaOM Pro (sub req’d):

Web Tablet Survey: Apple’s iPad Hits the Right Notes

8 Responses to “Envisioning Sketch Interpretation on Apple’s iPad and Other Tablets”

  1. Walt French

    There must be some reason why this (Flash) video, once interrupted by a glitch in my WiFi access or whatever, stalls and can’t be restarted on my MacBook Pro. Nor, it seems, can I go back to catch up on something I didn’t quite see right on first viewing. I have to refresh the page, and again it’s subject to any glitches.

    I see this crap on the Wall Street Journal website; it’s not a low-rent issue. Is this inherent to Vimeo? A problem with coding this particular Flash vid? A glitch that the Flash upgrade of a few days ago reintroduced?

  2. I think that this is more generally image search. Quite cool though! One of my favorite apps on the Tablet PC platform when it launched was Covey Tablet Planner, which would leave your input as handwriting, but interpret it in the background. I loved that app! In it’s first incarnation anyway…