GigaOM » IMAP

IMAP4 : The mobile application protocol we didn’t know we had

Brian McConnell — Sun, 05 Jun 2011 19:03:46 +0000

IMAP4 (Internet Mail Access Protocol) is a little discussed but widely used protocol for accessing and managing email. As it turns out, IMAP is also a great way to deliver information to mobile applications, by repurposing mobile IMAP clients for use in a wide range of mobile applications. While this is a bit of a hack, since we’re using a protocol designed for one purpose for an unintended use, it solves several problems that plague mobile applications.

IMAP is designed to do one thing and do it well — deliver email despite the intermittent connections that were prevalent when it was developed. For that reason is baked into nearly every email client and mobile device. Email delivery is also one of the most thoroughly tested services on mobile devices, and is designed to operate as an always on, background service.

IMAP was designed at a time when mobile Internet apps did not exist, and when most Internet connections were dialup connections. As with mobile apps running on cellular networks, email clients of the time needed to make best use of the intermittent connection they had. Without going into too much detail, IMAP splits accessing an email account into several tasks:

Logging in and retrieving a list of folders (inbox, outbox, etc)
Retrieving a summary of messages within a folder, but not the messages themselves
Retrieving an individual message and/or attachments

This approach enables the mail client to quickly find out what’s in a mailbox, and then retrieve the messages needed, without retransmitting information that is not needed or that has been sent before. This maps nicely to what many mobile apps do, and can be optimized in the same way.

In this paradigm, an information service presents an IMAP interface that email clients can connect to. Let’s use an upcoming event notification service as an example, and walk through the steps an IMAP client would use to display continuously updated event information tied to your current location. The client would go through the following steps:

Login with the user’s credentials, request a list of folders. The server replies with a list of folders such as “San Francisco : Mission”, “San Francisco : Civic Center”, and so on.
The client subscribes to some of all of these folders, and fetches a list of message headers. The message headers contain basic information about upcoming events. IMAP’s server side search function will also be a useful way to fetch customized result sets.
The client fetches message bodies and attachments in the background, or when the
user opens an item, depending on how the email client is configured.
When the user opens a message, he or she sees an HTML formatted email, with links,
attachments, etc all preloaded, so there is usually no visible network delay because most of the message was preloaded in the background.

Why give IMAP a try?

Of course, you can do all of this with a native application that calls a web service in the background and selectively loads and caches information ahead of time. The problem is that it requires much more work on the part of the developer, and on some mobile platforms, background processes are tricky to manage (e.g. they’ll be shut down by the mobile OS to conserve memory or battery power), whereas using IMAP, one can simply subscribe to additional IMAP email account for the services the user is following. Most mobile email clients also notify the user of new or unread messages, and provide a simple notification mechanism, another bonus. Meanwhile, the service provider can track which users have read which messages, news items, etc.

Another advantage of using IMAP as a transport protocol, and using a built in IMAP agent on the phone, is it is already there, and is one of the core services on mobile devices. Your web app can behave like just another IMAP inbox the device can subscribe to, so it will be synced along with other mail accounts. Likewise, your users can subscribe using their phone’s built in mail client, or you can “skin” an existing client for look and feel, without forcing your developers to build the whole thing from scratch.

The missing piece.

Most of the pieces needed to do this are already available. The protocol is widely used and is built into nearly every mail app in some form. Only one piece is not widely available, an IMAP2HTTP utility that presents an IMAP interface to mobile apps, while communicating with an upstream web service via RESTful HTTP queries that are easy for web app developers to deal with.

Jesse Vincent’s Perl library, NET::IMAP::SERVER, enables web apps to serve data as IMAP mailboxes. In addition to libraries for popular programming languages, there is an opportunity for mobile development tools vendors, such as Titanium and PhoneGap to host IMAP-to-HTTP gateways that provide a solution for web app developers, much like the SMTP2WEB service enables web apps to process email. (Hint: it would also be nice if Google provided a similar handler for App Engine, much as it has done for inbound email processing and XMPP/Jabber messaging).

IMAP will primarily be useful for applications that continuously retrieve information, such as news readers, background search tools, and similar applications that can present information as a set of items in folders. It is a bit of a hack, but it’s a useful hack that can improve the user experience for many mobile applications. It won’t replace native apps, but will provide developers with a fast and inexpensive way to provide mobile access, and then make a decision about whether to invest in native apps further on in the product development cycle.

Brian McConnell is an entrepreneur, and the founder of the Worldwide Lexicon, an open source translation platform. Prior to founding WWL, he founded several telecom and mobile startups (where the balkanization of the mobile software industry was one of his biggest pet peeves).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Simplifying Email

Charles Hamilton — Thu, 18 Jun 2009 18:00:18 +0000

As web workers, we are often asked to help friends and relatives fix computer problems. For me, the majority of these problems seem to be related to email. It’s ironic, as email is now less popular than social networks.

So why is email such a hassle?

It’s more than 30 years old. Email has come a long way, but its underlying protocols haven’t changed much since the 1970s.
It’s really three different systems. Sending (SMTP) and receiving (POP or IMAP) are totally separate functions, and are often handled on different servers. That’s why I often hear comments like “I can receive, but I can’t send” from clients.
It’s being used for a lot of things it was never designed to do, like send images and attachments, highly formatted messages, signatures and calendar entries.
It’s been overrun by spam, and even well-designed spam filters aren’t perfect, and cause unwanted side effects, like messages that get misidentified as spam, or just go away.
Email software is too complex. These programs that were originally built for offline use; that is, they were set up so that users could read and write messages without being connected to the internet. Sending and receiving would happen in batches. That made sense when internet connections were slow, expensive and charged by the minute. Now that most people have always-on connections like cable or DSL, that process is less necessary. Desktop email client software is a pain to set up and use; as someone who helps many people with email, Outlook is the bane of my existence.
Many of us connect to the Internet in more than one place — at work, at home, and on cell phones. It can be very frustrating to realize that we’ve left the message we needed to reply to at the office.
Many of us have more than one email address. I try to keep my work and personal email separate, plus I have a series of email addresses that I use when registering on websites that might try to send spam. And I have several email addresses that were given to me, such as the ones that are automatically created when signing up for instant-messaging services like Yahoo, AIM and Windows Live/MSN.

What can be done to overcome these problems? Here are some tips that might help you and your clients and friends be more productive.

Get your email on the web. Dump your desktop email software, and switch to Gmail/Google Apps or another online provider like Yahoo. If your Internet connection is unreliable, Google Gears lets you work offline.
Create a master inbox. If you have multiple email accounts, you can set up forwarding to receive and send email from one place.
Use IMAP. If you need mobile access to your email, set up your phone software to use IMAP, not POP. By using IMAP, your messages will sync automatically in all of the places you check your mail.
Use social networks. It seems like all of my friends under 30 don’t do email anymore, but they’re on Facebook a lot.
Use instant messaging. For short, simple conversations, IM can be very efficient. In a few seconds, you can schedule a meeting or a lunch date. It’s much faster than email or phone conversations.
Use file-sharing services for sending large documents. There are lots of such services, and new ones are popping up all the time, including Fluxiom and FileShareHQ. And Dropbox and the new Opera Unite service allow you to share files directly from your computer.
Organize your electronic communications. Celine wrote about this recently, and I’ve talked about it, too.

Oh, and one more:

Remember your passwords. This is obvious to you and me, but I’ve heard “I didn’t know I had a password” way too often. Online services like LastPass, or programs like 1Password, SplashID, or the free KeePass (Windows and phones; also available for Mac and Linux) can be lifesavers.

How do you keep email simple?

Image by stock.xchng user chris27.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Zoho Mail Goes Offline & Mobile

Imran Ali — Sat, 11 Oct 2008 14:17:05 +0000

As Google deepened its support for offline access via IMAP this week, Zoho, its closest competitor in the web office space, was publicly unveiling its own support for offline and access, ironically using Google’s own Gears platform.

Curiously Zoho decided that to bring users’ mailboxes offline, Gears was a better technological platform for offline access than the IMAP protocol; though we’re assured IMAP is coming.

Regardless, the offline features seem pretty comprehensive despite currently being restricted to Gears for Firefox and Internet Explorer and with most online features being available offline – messages, images & attachments are optionally available and a clever connectivity detection feature automatically determines whether a network is visible, flipping between offline and online modes as appropriate, with offline messages queued for later deliver when connectivity becomes available.

A Gears configuration dialog allows users to select the number of messages to download initially, how many Sent Items should be stored for offline access.

Finally, though Zoho is pitching mobile access alongside offline support, in reality Zoho Mail is currently only optimised for the iPhone.

Though the offline support appears to work well enough – as do other Gears-enabled services such as Google Reader – mainstream offline access seems a little too fragmented for comfort.

Where standards such as IMAP are pretty much guaranteed to work across multiple platforms, clients and services, supporting entire classes of application, such as email. AIR, Gears, Prism and Silverlight all seem to be mechanisms for locking users into a vendor platform, forcing developers to have to divide their efforts in supporting them.

As an industry, we should be thinking more responsibly about data portability through standards…a complex endeavor, but in the longer term much more viable and stable than the collection off offline platforms and browser combinations we see today.

Read more about Zoho’s offline and iPhone support here…

(Incidentally, I’ll be meeting Zoho’s evangelist, Raju Vegesna, later this month at O’Reilly’s Web 2.0 Expo Europe. If you have questions for Raju, please drop me a message!)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Google Labs Gives Gmail an IMAP Functionality Boost

Kevin C. Tofel — Fri, 10 Oct 2008 18:34:34 +0000

Gmail, a staple of our work environment, becomes a little more like “big-boy” IMAP courtesy of Google Labs. Hit the Labs button in your Gmail account and you’ll see a new “Advanced IMAP Controls” function. Enable it and you can begin to take advantage of the following new features:

Auto-Expunge: it sounds like a dirty word, but expunging is a good thing for data that’s no longer needed. Up to now, Gmail would simply mark items for deletion in clients and not immediately delete them. Enabling the Auto-Expunge feature kicks those deleted messages to the curb and gets them out of your e-mail system.
Delete Forever: one the benefits of Gmail is that like an elephant, it never forgets. All of your mail is archived in the Gmail / All Mail label or folder unless you manually clean it out. This often results in wasted space, messages in multiple locations and more bandwidth required for mail synching. You can now expunge mail forever by setting Gmail to simply move it to Trash upon deletion.
Selective Syncing: you can now pick and choose which labels or folders you want synchronized between Gmail and your web or desktop client. This comes in handy for archived folders that don’t change much. For example, I still want my “CES 2008″ folder data, but I don’t need to see it every waking minute of the day. I’ve disabled that one for now because I’ve got an active “CES 2009″ folder for planning purposes.

Each of these features can be useful, but for me the ability to say good-bye to the All Mail label usage is key. I often work remotely over wireless broadband connections where there’s a 5GB monthly data cap and 1.5-hours of work already uses near 45MB. Why should I be synching the near 53,000 e-mails in my All Mail folder when I’m already managing the ones I need with custom labels?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

TV Tops, But PC Video Watching Grows

Chris Albrecht — Thu, 05 Jun 2008 12:55:09 +0000

While oldteevee remains the top video-watching dog for most people, the PC is gaining ground, according to a recent study by Ipsos MediaCT. Ipsos found that, among U.S. video downloaders and streamers, the amount of video consumed on a TV set dropped to 70 percent in February 2008 from 75 percent in February 2007.

While oldteevee was dropping, video watching on PCs grew to 19 percent in 2008, up from from 11 percent in 2007. According to Ipsos, of the 52 percent of Americans 12 and older who have ever streamed or downloaded video content, roughly one out of every five hours of video content is watched on the PC.

And the rise in PC watching is across all age groups.

The study also found that video watching is still a home-based experience for most. Teenagers had the highest use of a portable media device to consume video content, while the mobile phone continues to be the red-headed step-child of the video universe, nabbing just one percent of consumption (but that trend should change as those two devices continue to merge).

Because of this home-based watching Ipsos sees an opportunity for set top boxes like Apple TV and the Netflix Roku, which are trying to bridge the web to PC gap.

Hat tip to eMarketer.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Jailbreaking Hotmail

Imran Ali — Sat, 26 Apr 2008 17:00:10 +0000

With Microsoft’s announcement this week that Outlook and Outlook Express will no longer support desktop access to Hotmail accounts raises some interesting questions on data portability.

After 30th June, Microsoft’s Windows Live Mail application will be the only means by which desktop and offline access to Hotmail accounts will be supported. This effectively means that a Hotmail user’s messages continue to be imprisoned within a closed ecosphere of services and applications. OK, smart people won’t be using Outlook, Outlook Express or Hotmail, but millions do and many have years of messages archived that they may wish to continue accessing outside a web-based interface.

However, there are some unofficial mechanisms that can not only continue to provide offline and desktop access, but also standards-based access into most email clients

IzyMail has close to a decade of experience in delivering webmail gateway services. A one-off payment of $17.95 buys a one-year subscription that equips any Hotmail, Yahoo, AOL and Gmail account with a POP and IMAP ‘wrapper’, so users can send, receive and synchronise messages using most desktop clients, such as Thunderbird, Apple Mail and Outlook

Thunderbird’s WebMail extension plays a similar role to IzyMail, though is of course limited only to Thunderbird and doesn’t necessarily support the various permutations of a service like Hotmail (Live Hotmail, Hotmail Classic etc.) but does support a wider range of services

Apple Mail fans can use a similar plugin – HTTP Mail – to retrieve Hotmail messages and folder.

Of these solutions, I’d recommend IzyMail’s standards-based approach, freeing the user to employ any both leading mail protocols with the widest range of clients. Indeed, IzyMail saw a huge spike in subscriptions with the launch of the iPhone last Summer as users utilised POP and IMAP webmail gateways to synchronise their email with their phones.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

He’s gettin’ old… er, I mean: Happy Birthday James!

Kevin C. Tofel — Thu, 07 Feb 2008 14:23:25 +0000

OK, he might not be getting his present on time… c’mon Fujitsu, don’t you have his birthday in Plaxo, Outlook or something?!? Just because there’s no gift from the FedEx man today doesn’t mean we can’t all wish James a Happy Birthday and many more years of mobile tech blogging. Happy Birthday dude! Oh and I hate to be the spokesperson for the group, but I’m guessing in another year or two it’ll be time for a “more mature” looking avatar. Just kidding! ;)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Move Networks, Acinion Raise Millions

Liz Gannes — Tue, 02 Oct 2007 17:39:19 +0000

Online video infrastructure companies are in high demand these days, and they’re taking advantage of the opportunity to haul in cash. Dan Primack of Private Equity Hub has the scoop on two deals: Move Networks recently raised $34 million, and Acinion took in $16 million.

This brings Move’s total venture capitalization up over $45 million from firms like Steamboat Ventures and Hummer Winblad Venture Partners, while Acinion now has raised around $21 million from firms like Globespan Capital Partners, IDG Ventures Boston and Sigma Partners (new to this round).

We’ve been impressed with American Fork, Utah-based Move, which provides video streaming platforms for customers such as ABC, the CW, and FOX.

However, a Move spokesperson claimed in an email to NewTeeVee that Move would soon be “the undisputed big money raiser in 2007 in online video space.” I’m not sure that’s a title anyone should be boasting about, but it’s clearly untrue if Primack is right about the number: Dailymotion also raised $34 million, Joost has $45 million, Hulu has $100 million.

Acton, Mass.-based Acinion is said to be making servers specialized for high-definition video, though we haven’t heard of it launching a product or signing any customers yet.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Microsoft Zune getting wireless download feature?

Kevin C. Tofel — Thu, 10 May 2007 15:50:35 +0000

Back in April, I wrote up three features that I think would make the Zune a more compelling product. If you believe in rumors, some with blurry pics, at least one of the three items might be around the corner. Engadget has two photos showing a beta firmware build for the Zune that provides the ability to download music from open WiFi hotspots. Don’t know if it’s true, of course but I still believe the Zune team was forward-thinking by including wireless capability in the Zune. I personally have never turned on my Zune’s wireless radio because it doesn’t provide a feature I want. Now, if I could just get that XBox 360 video content on my Zune, I’d really be a happy camper. ;)

While we’re on the subject of Zune: what features would you want in ‘the perfect handheld audio/video device’?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Kingston unveils tiny 9-in-1 USB card reader

James Kendrick — Wed, 09 May 2007 12:39:06 +0000

The folks at Kingston have alerted me to the release of a very cool and tiny USB card reader. The MobileLite reader is sold alone for just $8.99 or with either a 1 or 2 GB SD card bundle or a 1 GB MicroSD (prices below). Take a look at the photo of the MobileLite and you’ll see why I can’t wait to get one of these in my hot little hands, it is about as small as can be and still read 9 different card formats. Here’s the deets from Kingston, note the way it handles drive letters:

The new MobileLite USB Reader works with nine different memory card formats: microSD; miniSD; full-size Secure Digital; SDHC; MMCmicro; MMCmobile, RS-MMC; MMCplus and full-size MMC. When in use, MobileLite appears as three drive letters (for example G:, H: and I:) and data transfer is as simple as drag and drop.

MobileLite Card Reader with 1GB SD Card Bundle p/n: FCR-ML+SD/1GB = $ 22.99 MSRP
MobileLite Card Reader plus 2GB SD Card Bundle p/n: FCR-ML+SD/2GB = $ 34.99 MSRP
MobileLite Card Reader plus 1GB microSD Card Bundle p/n: FCR-ML+SDC/1GB = $ 23.99 MSRP

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.