Your other point is about the fact that some workloads don’t partition well, and you need to do replication. You’re right — that’s a trickier nut to crack. It arises in scientific computation a lot. I’m not will convinced that social nets are clearly in that bin. I worked with LinkedIn to parallelize a number of their analytic jobs on top of Greenplum using both SQL and MapReduce, and they partitioned quite smoothly.

There’s historically been a tendency to think in the following terms: “I need to parallelize Algorithm X, and it has a lot of intrinsic data sharing, so I need an innovative/expensive parallel architecture since data-parallelism won’t work”. In many cases, it can be more fruitful to think in these terms: “I have a cost-effective data-parallel infrastructure, so I need an innovative Algorithm X’ that approximates Algorithm X.” This mindset is taking hold in a number of quarters, and there’s quite a lot of sophisticated things you can do cheaply if you embrace that approach.

It’s true that Map-Reduce does wonders for offline processing of data generated as a byproduct of activities on the web, phone etc. But it needs to be pointed out that social network data models create online data management problems that are not parallelizable. This is simply because user-friend-shareditem relationships generate networks of relationships that can’t be partitioned as easily as the hierarchical trees we were accustomed to tackling in the business world. There is a lot of talk of sharding/partitioning etc. Yet it remains true that the underlying data models in social networks combined with the ~100million users and their data represent a whole different class of online data management problem not addressed by map-reduce, column stores … and all the offline data management technologies that are correctly getting a lot of attention. In the meanwhile, Twitter, Facebook, etc. have to face massive online data management problems not addressed by the vendors or the research community.

There are surprising implications when one does look deeper at the problem. Simple back of the envelope calculations re: scaling data items belonging to ~100million network-meshed users suggests that only natural way to partition the data is for users to have their own client side data stores and run queries locally when those queries mostly involve their small subset of data. Also the ratio of (CPU+disk ) to data quantity is much more favorable. Offloading subsetted queries is the only way for online social network data management to scale.

This creates the next generation of client-server architecture where ~100 million client-side databases (possibly on SQLite) each of ~100MB-10G need to be synchronized in real time with say the Facebook backend. Realistic scalable solutions in this area, i.e. web driven online data management for ~100mill users, are being completely ignored by the Oracles/MySQLs of the world and also by the database research community and yet these problems can be very lucrative …. Of course there is a lot of money in managing offline Big Data but who will help solve the pain of exploding online data management – what I’ve called Data 2.0 since 2005 ?

The current advances in memory with 2TB RAM in 2U box for ~30K may provide temporary relief but the underlying meshed-network data models need serious research and engineering and for now the

1) When it first appeared, this post had a different title, which had an unpleasant whiff of MapReduce marketing. I honestly don’t know how that title got there, but it wasn’t what I put down when I submitted the text. Hopefully the fix to the title now gets the flavor a little closer to where it should be. (It also makes R. McLoy’s post a little confusing, since it refers to that earlier title which is now gone…)

2) For me, the most interesting thing about MapReduce (by far!) is that people are so interested in it. We’re only going to make progress in parallel computing with paradigms that programmers embrace. SQL and MapReduce are two successful examples (arguably the only two) in the parallel programming space. MapReduce is the new kid on that block, relatively speaking. So we should learn from that, channel the energy to more general directions, and make more progress. You have to bring people along with you on this stuff.

3) On other languages/models: Josh is right: the MS work on LINQ and co. — and the MSR work on C-Omega — are some of the most interesting things in the space of embedding declarative nuggets into traditional code, and moving toward a more parallelizable world. Definitely worth watching. In the comments on the first post, somebody mentioned Erlang, which is another very interesting and somewhat different design point in the parallel computing space. Also well worth looking at — I recommend the Erlang book, it’s a pleasure to work through, and eye-opening. Finally, I’m pretty fired up about our research work on Overlog, and the work we’re beginning on Lincoln. These languages push the data-centric style of SQL and MapReduce into a much richer programming model. The Overlog work is documented to some degree at https://trac.declarativity.net. The work on Lincoln is still in the pipe, and has an eye on attracting programmers, not just researchers. Watch that space.

4) Yuvamani: Yep, Google’s MapReduce is a batch system, because that’s what Google wanted at the time. Hadoop copied that. But it’s all relatively easy to change, with little impact on the programming model. BigTable and company are indeed very interesting. I expect an integration of low-consistency storage with other query processing languages in the next year or so. We’ll see which of the DBMS vendors gets there first, or if it happens by gluing a query processor over a system like that. Facebook’s Hive is one open-source step in that direction (though it’s got a pretty thin query processor at this stage). The question of how much query optimization to do under the covers is kind of an eternal tradeoff between complexity and control. We’ll see how it plays out in the emerging application domains.

5) Ronald: I have to agree. Mother Nature has been amazing people for millenia. But it’s been hard for us to systematize computational insights from that — so far. There are some pretty intriguing tidbits in the research about DNA computing and the like, but it will be quite some time before you can, say, implement Spore that way :-). In general, we need to place bets on both short-term and long-term research, and keep a clear eye on the potential of each.

Most webapps like flickr / youtube / amazon / blogs /twitter have found that the main problems that they face scaling is caused by the database. SQL while being easy to learn is also opaque in the sense that it is really not known what will happen behind the scenes, Query plans change leaving massive website slowdowns etc etc. Also parallel / distributed databases are not really out there – mysql or oracle are not really parallel on commodity hardware.

Thus Facebook / Amazon / Google are producing a simple parallel databases for webapps, I am talking of Cassandra / Dynamo and BigTable obviously. There is also CouchDB etc

Why is this important. You assume that the MapReduce will interact with parallel database through SQL. The problem is 1) Mapreduce is not built for interactive apps. RDBMS are built for these classes of apps. 2) There are no good (meaning industry standard) distributed RDBMSs out there to power the parallel future you envision.

SQL or no SQL is a smaller problem. Any language used here could be SQL like to take advantage of the ease of use and the number of people out there who know SQL. However the open question is what will the parallel db look like is still open.

Finally, I think the title’s a bit misleading: while you can certainly credit MapReduce with attacking the problems of parallel data processing, there are plenty of endeavours involving parallel systems where it doesn’t apply. The resurgence of interest in actor systems is probably helping more widely when it comes to tackling concurrency in general.