Damian Black, CEO, SQLstream
(c) 2012 Pinar Ozger pinar@pinarozger.com

Ever wonder how a site like Mozilla.org could display a constantly updating download counter when it released Firefox 4 in early 2011, showing exactly how many copies of the popular browser had been retrieved by users? SQLstream CEO Damian Black explained at GigaOM’s Structure conference in San Francisco Thursday that the secret is dataflow — the parallel processing of data.

Dataflow was invented in the ’70s, when computer scientists first made advances in parallel computing. “Parallel computing in the ’70s was ahead of its time,” said Black, but the technology has made huge advances since, and is now ready for real-world applications. “Dataflow has finally come of age,” said Black.

But what is dataflow exactly, and how does it differ from other types of data processing, including Hadoop Map Reduce? The latter breaks down streams of data into large chunks and then processes one chunk after another. “That leads to quite a bit of latency,” Black explained. The relational streaming approach favored by Black’s company SQLstream instead processes streams immediately. “As soon as data comes it, it will flow out with minimal latency,” he said.

However, Black reminded his audience at Structure that this isn’t really a choice of one of two approaches. Mozilla’s Glow counter used Hadoop for historical, batch-based processing for long-term data analysis and at the same time relies on real-time dataflow processing to deliver that neat, constantly updating counter.

Check out the rest of our Structure 2012 coverage, as well as the live stream, here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Startup growth and the new recruiting ecosystem
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Using the gaming world as an example, Black likened this to a continuously updated real-time leaderboard. “By streaming big data, analyzing and processing millions of data bits and reacting to the output,” a massive amount of data can be queried in a far shorter time period.

But this approach isn’t unique to online gaming, Black suggested. Any data that fits what he called “S3 data” can take advantage of this approach; specifically data from sensors, systems and services.

That suggests a broad range of uses, especially as more connected consumer devices gain the use of sensors. And the number of services is expanding too; Black noted that streaming queries are ideally suited to services such as text messaging, Twitter and even real-time pricing information. The days of instant price changes based on true supply and demand may be a SQL query away.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media