Why Sync Is So Difficult

[qi:020][qi:026] As we increasingly struggle to manage our data spread, both on our devices and in the cloud, even Google (s goog) is going offline, for remote access and online storage aren’t enough. There is a need for sync with local duplicates, one the likes of SugarSync, Dropbox, Apple’s (s aapl) MobileMe and Microsoft’s (s msft) LiveMesh are aiming to fulfill.

I worked at SugarSync for almost three years (I no longer have any financial ties to the company), so I know the double-edged sword of sync firsthand. When you sell a sync product, you sell magic. (We free the data from their physical devices; just forget where you last edited the file, it’s gonna be everywhere.) But once it’s implemented, there’s no magic anymore, and the engineer is left to deal with asynchrony, slow bandwidth, third-party applications, and file systems that have different semantics.

Here’s why:

Push sync is deeply asynchronous

Conceptually, you have redundant copies of your data on various devices, and a service that keeps the copies in sync. That means that when a change is made on one device, you replay the change to the others. So when a file is edited on one device, it needs to be updated on the others. In the meantime, you can’t guarantee that the old version of the file remains untouched on the other devices. It could be edited, moved or deleted, which is when conflicts arise. It’s well known that concurrent programming is difficult; sync is just an extreme example.

Sync matches different data models

You can’t actually sync identical duplicates of your data because they live on different devices. So you have to translate the data to the local models. File names alone don’t sync properly from a Mac to Windows without careful unicode transformations, so imagine what becomes of extended attributes, resource forks, ACLs, etc. Even if you’re not cross-platform, most file systems can be set to either case-sensitive or case-insensitive. So you have to come up with an extensible strategy to deal with the different models, and there is infinite testing involved.

Sync messes with third-party applications

Applications tend to misbehave in various ways with their documents. When a sync product attempts to update a file with a newer version from another device, it can’t always know whether or not the file is open for reading or editing. In that case, the application may become unstable. Syncing app data is also dangerous.

Sync is hard to test

Sync maintains redundant copies of the user’s data through incremental updates. The devices are in sync as long as the redundant copies are consistent. Developers and testers will usually assume in their testing that the initial state is in sync; they will make a change on one side and see that the other side changes accordingly. So as soon as an error is introduced in the system (you’re out of sync to start with), you’re in a non-tested scenario. The system needs to recover from its own errors.

Several of these problems are on the client side, which is why building a sync client is hard — even on the best sync platforms out there, such as Sync Services or Live Framework.

Some of these difficulties are just inherent to sync, but the technology is maturing. Sync so far has meant something different to everybody. Even the industry’s main players still sell byproducts of sync rather than sync itself, which then compete with backup, online storage, photo-sharing or music-streaming services. Sync is a great enabler for all these connected services, one that’s becoming central to the personal cloud story of more and more companies.

Jean-Gabriel Morard is a software engineer living in Paris.