How to query Big Data quickly: Stream those queries!


Over time, we’re generating massive amounts of new data, thanks to a growing number of connected devices and services. As this Big Data gets bigger, it becomes a challenge to gain insights through traditional database queries: More data to sift through means more lag time between a query and actionable results. Parallel processing is one way to solve this problem, but streaming queries can help.

Damian Black of SQLstream at Structure:Data 2012

(c) 2012 Pinar Ozger.

SQLstream Structure:Data

Using the gaming world as an example, Black likened this to a continuously updated real-time leaderboard. “By streaming big data, analyzing and processing millions of data bits and reacting to the output,” a massive amount of data can be queried in a far shorter time period.

But this approach isn’t unique to online gaming, Black suggested. Any data that fits what he called “S3 data” can take advantage of this approach; specifically data from sensors, systems and services.

That suggests a broad range of uses, especially as more connected consumer devices gain the use of sensors. And the number of services is expanding too; Black noted that streaming queries are ideally suited to services such as text messaging, Twitter and even real-time pricing information. The days of instant price changes based on true supply and demand may be a SQL query away.

Watch the livestream of Structure:Data here.

Watch live streaming video from gigaombigdata at

Comments are closed.