As good as handwriting recognition is today, do you use it?

25 Comments

Computers_input_205660A recent post on TechChrunch by John Biggs got the attention of some Tablet PC enthusiasts:

“A laptop is an interactive tool. An ebook, even if it’s just aglorified, dual screen laptop, is a reading tool. That is why tabletPCs never took off in the mainstream: people don’t know what to do witha form factor that is clearly not a laptop yet is also clearly apowerful computer. There is no way to connect the act of “scratchingout words on a tablet” to processing worksheets in a spreadsheet. Whydoesn’t the iPhone have handwriting recognition? Because it’s ahorrible way to talk to a computer, even now.”

John was talking about the book form OLPC 2 that forgoes a physical keyboard with a dual-screen format but his comments about the Tablet PC gives one pause to consider the technology of handwriting recognition since that’s the tech behind the interface that John mentions.  I’ve given his comments a lot of thought and spent some time analyzing how I use my Tablet PCs and surprisingly I find myself agreeing with John’s comments for the most part.

To give this some adequate consideration we have to take a look at the handwriting recognition that is part of the Tablet PC today.  This means we look at Vista and the reco bits integrated in the OS so we get a feel for how good it is or isn’t.  I have been using Tablet PCs for a very long time and I have to tell you that the reco ability in Vista is outstanding.  My handwriting is, shall we say, less than good and it never ceases to amaze me how accurately the recognition engine in Vista understands what I write and converts it to text.  It is almost scary at times for me to see how well my ink has been interpreted when I can’t easily read it myself.  The advancements and improvements have been continual and impressive and the current technology is very good indeed.  So does this mean that John Biggs is wrong and that handwriting recognition is actually a good way to talk to a computer?

I am afraid not, at least in my opinion. Now before having a knee-jerk reaction to this listen to me for a bit.  The reason I say this is because I have analyzed the way I use my Tablet PCs and that paints a picture that is hard to ignore.  As good as the recognition is today I find that if I need to input more than a couple of sentences I will reach for the keyboard.  Part of this is because I type faster than I ink and part of this is because of the process itself.  When I am writing I like to input the text without thinking about the process and get on with the input.  Even with a high level of accuracy in handwriting recognition  there are still errors that have to be manually corrected before getting too far along.  Many people no doubt find this correction process to be one that derails the process, forcing one to stop the writing and fix what has already been written.  Even though recognition can have a greater than 98% accuracy, that still means that for longer passages several corrections will have to be made.  I find this to be totally obstructive to the writing process and I’m sure others would agree with that.

Now I don’t type perfectly with a keyboard either, but the errors are different than the errors you get with recognition failure.  When I mistype something it usually means I have a word that is misspelled and spell checkers can catch and fix it.  Not so with recognition errors and why you have to correct them immediately when you get them.  Reco errors result in giving you a different word than you wrote but a real word that won’t be flagged by spell checkers.  That means if you don’t correct it immediately you run the risk of having a strange word in the middle of your text, something that can make you look pretty foolish at times.  It is the primary reason that for text entry of any length I reach for the keyboard and not the pen and I don’t think I am unusual in that regard.

The Tablet PC has many other great uses that I exploit all the time, pen control and input is great under many circumstances but this article is focussed strictly on text entry as were Biggs’ comments.  I realize that those who are not good typists or who work in environments that don’t allow an arrangement for using a keyboard will feel quite differently than I and prefer pen input.  That’s the power of the Tablet PC, especially the convertible form that provides solid inking and keyboarding.  It’s the choice that makes it work so well for many.  But for the most part it is not hard to understand why Tablets have not cracked the mainstream market because most folks exchanged the pen for the keyboard long ago and that’s not going to change.  As cool as text entry is using the pen it is not practical for many for the reasons I’ve detailed and that is the root problem with Tablet acceptance.

25 Comments

GENGHIS

Sorry a late-comer to this discussion. I am a current Newton user.

If on the fly handwriting recognition is so distracting why don’t users just turn it off and process the ink into text afterwards?

That’s what I do with meeting notes on a Newton MP2100. By not activating the recognition engine and storing your notes as ink for susequent processing the distraction is totally avoided.

Handwriting is still a really useful way to get one’s thoughts down on paper. Sometimes a keyboard is a little too quick, handwriting forces one to think carefully about what you want to say before committing it to the screen. Many a half-baked thought got successfully intercepted before causing future embarrassment that way.

Brick ONeil

That article is spot-on! Pen input is great for short posts, articles and IM’s, etc., it is not applicable for longer work. When using the pen I often have to stop and backtrack to correct a mis-understood word. Also often, I’ll write a sentence and half-way through the tablet will change the words that i’ve already written, causing me to backtrack again. It really wastes more time than it saves.

Brick ONeil

That article is spot-on! Pen input is great for short posts, articles and IM’s, etc., it is not applicable for longer work. When using the pen I often have to stop and backtrack to correct a mis-understood word. Also often, I’ll write a sentence and half-way through the tablet will change the words that i’ve already written, causing me to backtrack again. It really wastes more time than it saves.

William R. Brohinsky

Handwriting recognition has never been a major issue for me (although it seems to be for everyone I’ve shown the tablet to.) As with you, James, being able to ink notes in class trumps handwriting->text every time, and as Doug Carmichael mentions, finding stuff in handwritten text is useful beyond reason. (Although I’ll also mention that I rarely use it, because my notes tend to be topic-logical if the teacher is, and so things are relatively easy to find again.)

So while I was attracted to the idea of HR, I’ve never actually used it for more than a few words at a time, mostly when in Sunday School and inputting a search for the Logos Software while mostly paying attention to the teacher. Even here, HR is a burden rather than a helpful tool, because it gets in the way of finding Habakkuk 3:16 or 1 James 1:5. These are not things that vista’s HR learning system seems to ever learn to parse.

On the other hand, I ink as fast as I type, and without recognition I can take copious notes in Sunday School or Differential Equations as fast in ink as I might on the keyboard (about 115wpm before mistakes, 90wpm after corrections) but can include chemistry symbols (get HR to deal with _that!_) and short spurts of music notation without resorting to any other applications.

That’s what’s important to me.

Mr. Crash

Yet another reason why Techcrunch isn’t worth reading these days.

Tabloid Tech.

Stephen B

Another former Newton User here and long time HWR advocate. I have used HWR across a range of platforms from the Newton (of course!), XP Tablet edition, Sharp Zaurus, and every iteration of the Pocket PC/Windows Mobile platform.

HWR on the XP Tablet edition was pretty good and it sounds like Vista is even better. On the Windows Mobile Platform the latest generation of Transcriber does a very good job IMHO. But, as pointed out by John J D’Alessandro above, none of them can match the Newton for the total integration of the HWR into how the OS worked. The Newton was all about HWR and, indeed the Newton HWR is still just about the best I have ever used.

It is the lack of integration that makes HWR so much less usable in pretty much every other OS I have tried it with – again IMHO.

Dave P

I tend to agree with you but I would make a few other points.

First, for document creation (i.e. creating a file with more than a page of text) typing is still the best mode of text input (although I haven’t tried speech recognition for several years). However, once the document is created, ink notes and ink comments are far more usable and easier to input; either during meetings or while reviewing documents.

As has been pointed out, inking during meetings to take notes is better, in most respects, than typing. What also needs to be stressed is that it is far less obtrusive to the meeting than typing.

With presentations, ink presents a mechanism that just does not exist with typing. Not only can one use ink to stress points on a slide but one can use it to capture comments and distribute them after a presentation.

Lastly, ink seems ideally suited to the small form factor. While there are those who can thumb with ease and even those who can use T9 pads at an incredible speed (esp if u rite 4 meaning nt 4 spell chk) for me, inking on my OQO is far more usable (and natural) than pulling out the keyboard.

All in all, ink has its place and it would be a shame if it was lost in the drive towards touch.

Dave

Lorie Ghamy

For best inking experiencewithout keyboard :

1) Use a biggest pen for UMPC like Samsung Q1 (a criterium with wacom pin)not the native needle !

2)Use a virtual desktop (Virtual Dimension is the best with a light edge setting like TIP and i work with 4 desktops).

3) Use CLCL (nakka.com) to have a copy-paste with multiplefiles (texts & visible photo or multiples files tomove… and Templates)

4) Use Dictionnary to increase special or technical word with a text file backup.

5) Use Strokeit for mouse gestures…

Steve Hoffman - Active Ink Software

James

I agree that the handwriting recognition is good, but could be improved. I find that for data entry, the HWR works great but for long strings of text, it’s not perfect. Part of the problem is that the tablet PC’s dictionary does not accomodate all the words in one’s vocabulary. There are certain streets, surnames, acronyms, abreviations and special terms that aren’t included and thus difficult for the HWR to get right.

John J D'Alessandro

I was an avid Newton 2000 user. The big hindrance of the Windows Vista Toshiba Portege M400 I am using to type this to you is the interface.

HWR was integrated into the Newton, not slapped on. If I wrote “5” in a cell in a spreadsheet, a computer-print-perfect “5” appeared in that cell. In Windows Vista, I have to click in cell, pop out the TIP, write “5”, (possibly click “Insert).

Writing on a Newton was like writing on a pad, with the problem of errors but the benefits of digital electronic manipulation. Writing on Windows Vista is like using a special keyboard that only works on the screen in some places some times in some ways. It is powerful, still, but much more clumsy.

I miss that dang Newton.

tlewis

James,
Great points.

The truth is that the entire “tabletpc interface” pretty much sucks. It offers few winning usability arguments for the common user, although it could. And should.

I’m a geek and it took me months of ardent struggles with the form factor to get my first tabpc to bend to my will. Hell, 4.5 years and 4 tabpcs later I’m still battling. I stick with it because as an artist (see ArchiMark’s sage post above), the tabpc is a dream come true.

But for a lay-newbie there aren’t many of the “aha”-gasms one gets using an Apple product. We need a smoother integration of hardware and software. Instead we get the usual Microsoft practice of taking the OS up to a point, then dropping the follow-thru onto the 3rd party apps and OEMs. While that model (sort of) works with the broad market, it’s simply a dysfunctional fantasy for MS to think that’s the way to sell tablets.

The TabOS feels like a lumpy, bumpy layer awkwardly lying atop Windows (famously lumpy itself). A tabpc needs an OS shell that is deep and thorough…that works on the detail-level.

One teeth-grinding example: the “Penabled” thing. There are in my Control Panel at least 5 different little driver-apps that influence how I move around my screen…

1) Tablet and Pen Settings
2) Pen Tablet Properties (wacom’s add-on for pressure in PhotoShop, etc)
3) Touch Panel
…and the usual…
4) Keyboard
5) Mouse

Confusing enough for the poor dude who has just dropped 2 grand. But add it to the fact that the pen-based drivers are often quirky and some of the apps don’t easily believe in the other apps without counseling (Touch is easily traumatized by Pen Tablet), and you’ve created built-in user agony. This halfway thinking is crippling a potentially elegant and industry-changing form factor.

Considering all the hype and Gate’s ardor for tabletpcs (not to mention the price bump for tab features), the clunky niche state of the tabtech is bewildering…and shameful.
T

Demetri

I also agree on the actual usability of inking. Another aspect that I have noticed is how using a tablet pc for inking or a laptop for note taking can also be a hinderance to the flow of a meeting or conversation. That is why I have had a hard time justifying the expense of buying a tablet pc for myself. In my last job I was able to get a tablet pc as my work computer, but found that I used it more as a normal computer rather than an inkable pc.

If the new line of smartpens gets a handwriting recogntion feature, I might be more inclined to invest and use one of those.

Ah, the search continues.

Steve Krug

Great commentary, James. You’re exactly right: handwriting is just plain slow compared to typing, so it’s usually painful to use handwriting recognition for more than one or two sentences, especially when you factor in the time it takes to correct even one error. (I’d never thought about one excellent point you make: typing errors will usually be flagged by your spellchecker, handwriting recognition errors will always be “real” words, hence harder to spot.)

In theory, what handwriting recognition is best for on a tablet is quick bits of input: a note, a to-do item, a phone number, or a URL. Unfortunately, these usually contain things that aren’t generic words (think about a URL, or proper name), where handwriting recognition is at its least effective.

What *is* faster than typing, though, is speech recognition. With a little bit of training (very little, with the latest versions of Dragon NaturallySpeaking) and a little bit of practice in articulating clearly, I can dictate much faster than my 55 WPM typing. Which is why it’s particularly frustrating that Microsoft and the tablet PC manufacturers have put so little effort into OS software and built-in microphones that just work.

Yes, speech recognition does generate the occasional error (although I’ve dictated this entire comment without any mistakes so far, at least as far as I know). But what nobody seems to have noticed is that the combination of pen input and speech recognition works remarkably well: you dictate at a pretty fast clip, and when there is a mistake you select it with the pen and either re-dictate it, write the word, or correct the letters with the TIP keyboard.

I’ve been looking for years for a tablet that has really good built-in array microphones that cancel background noise, focusing on you while you’re talking…even in a somewhat noisy environment. Instead I get microphones that *generate* a lot of noise. And an operating system that seems to constantly choose the wrong microphones for input, reset the volume levels according to its own whims, etc., etc.

Oh, well.

borax99 (Alain C.)

Hmmm, interesting post James. The more I use my tablets the most I think that HWR is indeed a secondary feature of Tablets. On the other hand, I find inking impossible to resist. After much time with devices used pretty much 100% in tablet mode, I am seriously considering going over to a full-size convertible (Fuji T4220, T2010 or HP 2710p).

Equally undeniable is that, once you have tasted the flexibility of being able to ink whenever you want, using a regular laptop is like going for batting practice with one arm tied behind your back !

James Kendrick

Good points all and I should reiterate that I’m not talking here about the usefulness of inking in general. I ink notes for hours every day and wouldn’t trade that for the world. No, I’m talking about handwriting recognition which by definition is the conversion of ink to digital text.

Inking notes works so well for me because I NEVER convert them to text because the searching is so good. There’s just no need to convert ink to text most of the time.

Jake

As with the comments above, I agree with everything you’ve said James. One other thing that I think stops me from taking a tablet into meetings (as opposed to a pen and paper) is the fact that the PC itself will slow me down. Conversations move swiftly in most meetings (especially impromptu ones) and I can’t really afford to wait for my tablet to wake up or for One Note to load with the risk that something might freeze or slow me down in some other way. If tablets could include a 100% reliable instant-on capability and software was 110% reliable I would love to take notes on a tablet but we’re not quite there yet.

Andrew Wilcox

Ink is great when you are writing whilst sat on the sofa as I am now. I have hardly used in my notebook (the paper sort) for the last 3 years. I do all my notes in ink in MindManager on a TC4400 or previously a TC1100. It’s a pity Mindjet have not developed their ink to text conversion to match the quality of Vista.

I have tried pen flicks and Vista Ink to Text in mouse mode but iris not as fluid as ink mode.

Ink may be slower than typing but that sometimes improves my writing.

ArtRage is a great painting package for Tablet PC.

NigelG

Isn’t the problem that Joe Public (and even some tech jounrnos) think “Handwriting recognition” means writing stuff and turning it into text. Which is very niche and certainly not something I would consider, I just reach for the keyboard.

However what tablet fans have always known is that SEARCHING handwritten text is what matters for most people not CONVERTING. HR is what powers OneNote for inkers. And doing it offline is what EverNote are promoting too. Take a snapshot of some written text and we will convert it for you. HR for the masses.

However….why was inking on the Apple Newton so much fun ? Sure, the recognition was not very good (compared to Vista) but as an experience it is still very hard to beat. Is it because of the usage, you were not trying to get it to write a thesis, but to simply remember to “Call Betty at 7” ?

Doug Carmichael

Taking notes in seminars where graphics are important, the ability to sketch is wonderful. the scrolling infinitely long page is terrific.

hand writen and drawn semimnar notes in a tab make finding easy. Compared to paper, which is usually lost after a month..

Also for writing poetry in the semi darkness, the slate mode is wonderful.

Eric

I think handwriting recognition is, at best, a secondary use for a tablet. The inking and pen are more of a flexibility addition. That is, tablet’s are all about new ways to interact with your computer: drawing, scribbling, taking notes are all possible with the tablet but are barely possible on a traditional laptop. This is especially true of presentations (as in education). Try interacting with a group during your PPT presentation when they ask questions–if you don’t have the exact slide, you’ll be doing lots of explaining. I, myself, get great use out of all the input forms, though I haven’t touched a mouse in about 2 years.

Gavin Miller

I do ink meeting notes and convert to text later. Also, in slate mode I prefer Vista’s handwriting recognition to the on screen keyboard for entering web addresses and similar. That’s about the limit of it and typing is always the preferred option for me.

Amit Agarwal

Agree with your observations James. If I have to write more than a couple of words on the tablet, I reach for the keyboard.

Primarily use the device for reading PDFs.

Alan

I agree, notetaking with a tablet pc can become
cumbersome at best. While the tablet form factor is a nice way to interact
with the screen, I also don’t see it as a universally necessary entity.

This is why I don’t understand why tabletPcs are not more heavily marketed towards professional artists, animators, and graphic designers, where pen input is not only more suitable, but is by and large preferred. Most
will use a typical wacom tablet because of its lower price than a cintiq, but in most cases a cintiq would be a substantially better tool. I believe tablet pcs can bridge this gap, but there are not enough tablet pcs with dedicated graphics and large enough screens to serve as a primary computer for these applications.

My two cents.

ArchiMark

James, as usual, you hit the nail on the head, or is it stylus?… ;-)

Good points about Tablet usage….agree with your assessment, that reco is great but still not yet good enough to make one just naturally choose it over keyboard if you have both options avaiable.

Hopefully, one day this will change, and either the situation will reverse or least one will need to think which mode is best for what I’m going to do….

Comments are closed.