As a refresher, the big difference between SQL on Hadoop and the Hadoop connectors that were popular a couple years ago is that SQL-on-Hadoop products query the data where it resides — in HDFS or HBase — rather than pulling it into a relational database environment to analyze it. We have been talking for months about the emergence of a large SQL-on-Hadoop market, but IBM’s name was conspicuously absent from that discussion. The company has Hadoop software called BigInsights and lots of SQL expertise, so it only made sense that IBM would get into the game at some point.

Details on Big SQL are still pretty sparse save for a few high-level blog posts and an instructional video (embedded below), but it looks to take the standard approach, as Cloudera is doing with Impala, of enabling access through traditional tools via JDBC and ODBC drivers.

Ultimately, I think the advent of big data will enable some new types of querying techniques quite a bit different than the SQL queries we’ve come to know and love over the past couple decades. But SQL is still the language du jour and might never go away, so there’s a lot of value to be had if people can put their SQL skills to work on data stored inside Hadoop or other environments, and if companies can work toward a nirvana where all the data is stored in a single place rather than across database environments.

That IBM got this message and got into the game isn’t surprising at all, but it is important. Lots of large companies buy IBM’s software.  If it wants them to follow it into the world of big data and Hadoop, it has to give them the tools they need to use it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

In practice, Cascading is generally used as a higher-level method than MapReduce for writing Hadoop jobs, although it’s technically a framework that could support any number of distributed-processing frameworks. It’s used by a number of notable users, including Etsy, Airbnb and Climate Corporation. In February, the Cascading project expanded its scope to address the growing SQL-on-Hadoop trend with a project called Lingual.

Software veteran Gary Nakamura is taking on the role of Concurrent CEO, replacing Cascading creator Chris Wensel, who’ll stay on as the company’s CTO.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

With the latest updates — announced in a blog post by BigQuery Product Manager Ju-kay Kwek on Thursday — users can now join large tables, import and query timestamped data, and aggregate large collections of distinct values. It’s hardly the equivalent of Google launching Compute Engine last summer, but as (arguably) the inspiration for the SQL-on-Hadoop trend that’s sweeping the big data world right now, every improvement to BigQuery is notable.

BigQuery is a cloud service that lets users analyze terabyte-sized data sets using SQL-like queries. It’s based on Google’s Dremel querying system, which can analyze data where it’s located (i.e., in the Google File System or BigTable) and which Google uses internally to analyze a variety of different data sets. Google claims queries in BigQuery run at interactive speeds, which is something that MapReduce — the previous-generation tool for dealing with such large data sets — simply couldn’t handle within a reasonable time frame or level of complexity. Of course, if you want to schedule batch jobs, BigQuery lets you do that, too, for a lower price.

This constraint — and therefore the potential benefits of something like Dremel and its commercial incarnation, BigQuery — wasn’t lost on the Hadoop community, which itself had been largely reliant on MapReduce processing for years. In the past year, we’ve seen numerous startups and large vendors pushing their own Dremel-like (or MPP-like) technologies for data sitting in the Hadoop Distributed File System. If you happen to be in New York next week, you can hear some of the pioneers in this space talk about it at our Structure: Data conference.

Background aside, the ability to join large data sets in BigQuery is probably the most-important of the three new functions. Joins are an essential aspect of data analysis in most environments because pieces of data that are relevant to each other don’t always reside within the same table or even within the same cluster. And joining tables of the size BigQuery is designed for can take a long time without the right query engine in place.

How to do a join in BigQuery

Kwek offers an anecdote from Google that shows why joins, and the new aggregation function, are important:

[W]hen our App Engine team needed to reconcile app billing and usage information, Big JOIN allowed the team to merge 2TB of usage data with 10GB of configuration data in 60 seconds. Big Group Aggregations enabled them to immediately segment those results by customer. Using the integrated Tableau client the team was able to quickly visualize and detect some unexpected trends.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• The fourth quarter of 2012 in cloud

Based in San Francisco and Santa Cruz, Calif., Looker has developed LookML, a proprietary language based on the SQL programming language for relational data, to enable users with little to no development background to make their own custom SQL queries of sales data. Once a customer signs up, Looker’s analysts review the customer’s data and, over a few days, custom-build the tool with options for querying the customer’s various databases in specific ways, said Lloyd Tabb, Looker’s founder and CEO. Once in place, Looker can run on a customer’s on-premise hardware or on hosted servers.

Many companies offer business analytics tools, and they come in different flavors. Oracle, IBM and other legacy IT vendors offer data warehouse appliances, although those can take engineers months to implement, Tabb said. Other BI vendors, including GoodData, Mixpanel and Tableau, can store customer data in the clouds, but those display bigger-picture trends in event data and don’t correlate well with user data, which aren’t in clouds, Tabb said. And then there’s Redshift, Amazon Web Services’ new data warehouse product.

Despite all the competition, Tabb believes Looker has a place in the market. The company, which is emerging from stealth mode about a year after its establishment, has more than 20 customers, including Simply Hired.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

Yara, Greenplum’s co-founder and senior vice president of products, has a not-so-coded message for his big data market competitors, too. Put simply, he doesn’t think they stand a chance against his company, and he served notice on Monday morning with the unveiling of the company’s new Pivotal HD Hadoop distribution and Project Hawq in a staged event at San Francisco’s Dogpatch Studios.

Pivotal HD is a completely re-architected Hadoop distribution that has been natively fused with Greenplum’s analytic database (that’s the Project Hawq part), but Yara thinks it’s a bigger deal than just another SQL-on-Hadoop play. In an interview last week, Yara told me that Project Hawq is the manifestation of Greenplum’s decision to sell itself to EMC in 2010, a move he thought would would kickstart his company’s founding vision of becoming the leading big data platform.

Building a data platform costs money, and lots of it

But before the details, a little history. Greenplum’s flagship product is an analytic database powered by a massively parallel processing (MPP) and query engine. The company had raised nearly $100 million in venture capital around this technology since launching in 2003, but doing business in the enterprise software world is hard and expensive, and Greenplum needed more money.

Yara (left) with Pivotal Labs CEO Rob Me and Om Malik at Structure: Data 2012 (c) 2012 Pinar Ozger. pinar@pinarozger.com

“I thought it was going to take another couple hundred million dollars in investment for us to complete the technical vision we had and go to market,” Yara explained. But finding that kind of money wasn’t so easy in an investment environment where everyone was gaga over social apps like Facebook and Zynga. When EMC approached with a deal like it gave VMware in 2003 — essentially near complete independence bolstered by a huge R&D and marketing budget — Greenplum couldn’t refuse.

Yara said Greenplum had known for a while that Hadoop was the key to any big data strategy going forward, but that it would take some time to build up its own technology. So, in 2011, it entered into a reseller agreement with Hadoop startup MapR to offer a premium product to appease enterprise customers while Greenplum’s engineers got to work on what would become Pivotal HD. That deal with MapR is still in place, but it’s no longer the focal point of Greenplum’s Hadoop strategy.

Big investment, big aspirations

The technology inside Pivotal HD is what companies should come to expect from a Hadoop distribution, Yara explained. It’s essentially the Greenplum Database with its POSIX file system ripped out and replaced by the Hadoop Distributed File System. Whatever users can do on Greenplum’s flagship database, they can do on Pivotal HD, only they can run Hadoop MapReduce jobs and house an HBase database, too.

And when SQL-like features become an important part of Hadoop because it’s so broadly installed that users are now seeking out broader utility, “that’s when the bar gets raised in terms of the amount of capability that’s required,” Yara said. He said Pivotal HD includes years worth of investment in Hadoop cluster-management technology and professional support, too, and that they will cost half as much as what Cloudera and Hortonworks charge. It’s designed to run smoothly wherever customers want it to — physical servers, virtual servers or even cloud servers.

Because they’re so new, he said, competitive SQL-on-Hadoop offerings such as Cloudera’s Impala can only handle about 20 percent of real-world workloads. Looking back at the capital investment in analytics and big data technologies past, things like Netezza, Teradata and Aster Data, Yara proffered, “I don’t think you could build [a full SQL-on-Hadoop] system for less than $25 to $50 million over five years.” (Some of those new technologies, by the way, will have a chance to state their cases during a Structure: Data panel on March 21 that’s all about Hadoop as the next-generation business intelligence platform.)

Greenplum, by contrast, rebuilt its entire R&D team to focus on bringing 10 years of database technology to Hadoop. “We literally have over 300 engineers working on our Hadoop platform,” Yara said. “… We’re bringing all the power of EMC and VMware behind it.”

The data warehouse is the new mainframe

Looking past his competitive boasting, though, it’s easy to see Yara’s greater point when you ask him what all this Hadoop talks means for the data warehouse business on which Greenplum was built. He points to the mainframe business that fell from its high perch decades ago but still drives billions a year in revenue. A single MPP database system is still faster on certain workloads than SQL on Hadoop, but that gap will close over time and  “I do think the center of gravity will move toward HDFS,” he said.

Josh Klahr, a Pivotal HD product manager, noted the importance of being able to process all of a company’s data right in a single scalable data store rather than operating numerous systems. He pointed to one customer that’s storing a petabyte of data in Greenplum Database but wants to grow its data volume to 20 petabyes over the next few years and needs something like Hadoop to do that both financially and technically. He said Netflix’s decision to store all its data in Amazon S3 and bring analytic services to it is a good indicator of where the market is headed.

A few years ago, Yara acknowledged, embracing Hadoop as the future might have been a scary proposition. However, he said, “Now, if you don’t embrace Hadooop as the new database platform, if you’re a database vendor, that’s a grave mistake.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

In fact, there are now so many products and projects pushing SQL queries and interactive data analysis on Hadoop — including two more announced this week — that it’s getting hard to keep track. But I’ll do my best.

Of course, Facebook began this whole movement to bring SQL database-like functionality to Hadoop when it created Hive in 2009. Hive, now an Apache project, includes a data-management layer and SQL-like query language called HiveQL. It has proven rather useful and popular over the years, but Hive’s reliance on MapReduce makes it somewhat slow by nature — MapReduce scans the entire data set and moves a lot of data over the network while processing a job — and there hasn’t been much effort to package it in a manner that might attract mainstream users.

And keep in mind that this next generation of SQL-on-Hadoop tools aren’t just business intelligence or database products that can access data stored in Hadoop; EMC Greenplum, HP Vertica, IBM Netezza, ParAccel, Microsoft SQL Server and Teradata/Aster Data (which this week released some cool new features for just this purpose) all allow some sort of access to Hadoop data. Rather, these are applications, frameworks and engines that let users query Hadoop data from inside Hadoop, sometimes by re-architecting the underlying compute and data infrastructures. The beauty of this approach is that data is usable in its existing form and, in theory, doesn’t require two separate data stores for analytic applications.

Data warehouses and BI: The Structure: Data set

I’m highlighting this group of companies first, not because I think they’re the best (although that might well be), but because I’m truly excited about the panel they’ll be featured on at our conference next month. The panel is moderated by Facebook engineering manager Ravi Murthy– a guy who knows his way around a database — so they’ll have to answer some tough questions from one of the most-advanced and most-aggressive Hadoop and analytics tools users out there:

Apache Drill: Drill is a MapR-led effort to create a Google Dremel-like (or BigQuery-like) interactive query engine on top of Hadoop. First announced in August, the project is still under development and in the incubator program within Apache. According to its web site, “One explicitly stated design goal is that Drill is able to scale to 10,000 servers or more and to be able to process petabyes of data and trillions of records in seconds.”

Hadapt: Hadapt, which actually launched at Structure: Data in 2011, was the first of the SQL on Hadoop vendors and is somewhat unique in that it has a real product on the market and real users in production. Its unique architecture includes tools for advanced SQL functions and a split-execution engine for MapReduce and relational tasks, and both HDFS and relational storage. In October, the company announced a tight integration with Tableau Software around advanced visual analytics.

Platfora: Technically not a SQL product, Platfora is red-hot right now and is trying to re-imagine the world of business intelligence for a big data world. Essentially an HTML5 canvas laid atop Hadoop and an in-memory, massively parallel processing engine, the company’s software, which it unveiled in October, is designed to make analyzing data stored in Hadoop a fast and visually intuitive process.

Qubole: Qubole is an interesting case in that it’s essentially a cloud-based version of the popular Apache Hive framework launched by the guys who created Hive while working at Facebook. Qubole claims it auto-scaling abilities, optimized Hadoop code and columnar data cache make its service run much faster than Hive alone — and running on Amazon Web Services makes it easier than maintaining a physical cluster.

Data warehouses and BI: The rest

Citus Data: Citus Data’s CitusDB isn’t just about Hadoop, but rather wants to bring the power of its distributed Postgres implementation to all types of data. It relies on Postgres’s foreign data wrappers feature to convert disparate data types into the database’s native format, and then on its own distributed-processing technology to carry out queries in seconds or less. Because of its Postgres foundation, CitusDB can join data from different data sources and retains all the native features that come with that database.

Cloudera Impala:  Cloudera’s Impala might just be the most-important SQL-on-Hadoop effort around because of Cloudera’s expansive installation and partner footprints. It’s a massively parallel processing engine that bypasses MapReduce to enable interactive queries on data stored in either HDFS or HBase, using the same variant of SQL that Hive uses. However, because Cloudera doesn’t build applications, it’s relying on higher-level BI and analytics partners to provide the user interface.

Karmasphere: Karmasphere is one of the first startups to build an analytic application atop Hadoop, and in its 2.0 release last year the company added support for SQL queries of data in HDFS. Like Hive, Karmasphere still relies on MapReduce to process queries, which means it’s inherently slower than newer approaches. However, unlike Hive, Karmasphere allows for parallel queries to run at the same time and includes a visual interface for writing queries and filtering results.

Lingual: Lingual is a new open source project from Concurrent (see disclosure), the parent company of the Cascading framework for Hadoop. Announced on Wednesday, Lingual runs on Cascading and gives developers and analysts a true ANSI SQL interface from which to run analytics or build applications. Lingual is compatible with traditional BI tools, JDBC  and the Cascading family of APIs.

Phoenix: Phoenix is a new and relatively unknown open source project that comes out of Salesforce.com and aims to allow fast SQL queries of data stored in HBase, the NoSQL database built atop HDFS. Its stated mission: “Become the standard means of accessing HBase data through a well-defined, industry standard API.” Users interact with it through JDBC interfaces, and its developers claim its sub-second response times for small queries and seconds-long response for querying tens of millions of rows.

A sample of Phoenix via the SQuirreL client

Shark: Shark isn’t technically Hadoop, but it’s cut from the same cloth. Shark, in this case, stands for “Hive on Spark,” with Hive meaning the same thing it does to Hadoop, but with Spark being an in-memory platform designed to run parallel-processing jobs 100 times faster than MapReduce (a speed improve over traditional Hive that Shark also claims). Shark also includes APIs for turning query results into a type of data format amenable to machine learning algorithms. Both Shark and Spark are developed by the University of California, Berkeley’s AMPLab.

Stinger Initiative: Launched on Wednesday (along with a security gateway called Knox and a faster, simpler processing framework called Tez), the Stinger Initiative is a Hortonworks-led effort to make Hive faster — up too 100x — and more functional. Stinger adds more SQL analytics capabilities to Hive, but the most-important aspects are infrastructural: an optimized execution engine, a columnar file format and the ability to avoid MapReduce bottlenecks by running atop Tez.

Operational SQL

Drawn to Scale: Drawn to Scale is a startup that has built an operational SQL database on top of HBase. The key word here is database, as its product, called Spire, is modeled after Google’s F1 designed to power transactional applications as analytic ones. Spire has a fully distributed index and queries are sent only to the node with the relevant data, so reads and writes are fast and the system can handle lots of concurrent users without falling down.

Splice Machine: Database startup Splice Machine is also trying to get into the operational space by building its Splice SQL Engine atop the naturally distributed HBase database. Splice Machine focuses its message on transactional integrity, which is really where it separates itself from scalable NoSQL databases and analytics-focused SQL-on-Hadoop efforts. It relies on HBase’s aut0-sharding feature in order to making scaling an easy process.

Feature image courtesy of Shutterstock user hauhu.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

Big data has opened companies’ eyes to the importance of analytics and alternative data stores, but combining the two often means learning new languages, using multiple tools and probably sacrificing the performance they’re used to from analytic platforms.

Citus Data’s flagship product, called CitusDB, is actually built atop PostgreSQL and its first iteration was designed for Google Dremel-like scale and speed on relational data. Thanks to a feature called “foreign data wrappers,” though, it’s able to run SQL on numerous data types (e.g., CSV, log and JSON files) that don’t comport with how Postgres formats data natively. So, while CitusDB now officially supports the Hadoop Distributed File System in addition to Postgres, it is by no means limited to them.

Matt Ocko, managing partner at Data Collective and one of Citus Data’s early investors, says the database can technically support any data source with an ODBC driver, and even could query something like log files straight from a data store. In fact, Citus is working on extending its support to MongoDB — a capability that’s in beta right now. Ocko is also particularly impressed with CitusDB’s ability to act like a fabric connecting all these data sources rather than making users query each independently and then manually join the data. He cited a demonstration in which CitusDB carried out a query that required executing a join across Postgres and Hadoop.

The other big thing about CitusDB is that it’s not just flexible but fast, too. Ocko said CitusDB has outperformed Oracle’s vaunted Exadata machine on a TPC-H benchmark test with data stored primarily on hard disk. That Postgres-Hadoop query he referenced completed in just a few seconds while running on the Amazon EC2 cloud.

CitusDB is so fast, Citus Co-founder Umur Cubukcu told me, because of how it’s architected. It moves the computation to where the data is rather than trying to move data across the network, and it has some impressive load-balancing the resource-management abilities baked in. If, for example, it needs data housed on a slow-running node in order to complete a task, the software will look for that data elsewhere rather than just wait for the congested resource to free up.

In the case of Hadoop, MapReduce brings the computation to the data, too, but every job requires a scan over the entire dataset. This is why early SQL-on-Hadoop tools such as Hive are still relatively slow. Citus software engineer Carl Steinbach, who came to the company from Cloudera, said CitusDB is between 3 and 20 times faster than Hive depending on the query type.

It’s actually much faster for short queries that might be typical in an interactive environment, but he acknowledged those aren’t really what Hive was designed to do.

Rather, CitusDB’s real competition is the spate of SQL-on-Hadoop projects, products and startups of which it’s now a part. We’ll have a whole session dedicated to this topic at Structure: Data next month, and there isn’t enough room for everything on the market right now — Aster Data, Platfora, Cloudera (with Impala), Apache Drill, Drawn to Scale and Hadapt, to name several.

These are impressive technologies (at least in theory where they’re still under development), and Citus would be remiss to ignore them. But, aside from the ability to query multiple data sources, the company has something the others don’t, Cubukcu said: It has the Postgres community and all the features they’ve built into that database already. Things like connectors, authentication, full-text search and PostGIS for geospatial data that go beyond just running fast queries.

“When you’re talking about an enterprise-class database,” Steinbach said, “you’re talking about more than a query execution engine.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• Real-­time query for Hadoop democratizes access to big data analytics

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Takeaways from the second quarter in cloud and data

ScaleArc, for its part, wants to let customers keep their MySQL, Oracle and SQL Server databases by making them faster and more intelligent. Its product, called iDB, sits between an application and its database and does a whole lot of good things — load-balancing, caching, SQL analytics and real-time dashboards among them. Recognizing the validity of certain new approaches for serving often-accessed data in a hurry (e.g., memcached) iDB’s cache actually uses a NoSQL database, although it’s backed up by the product’s standard security and high-availability features.

But for a company like ScaleArc, as well as the database products it aims to improve, the biggest threat probably isn’t NoSQL at all. Rather, they’re facing a new breed of SQL products — the so-called NewSQL movement — that promise everything developers and companies love about SQL with the scalability and speed that NoSQL can deliver. Among these offerings are Drawn to Scale’s HBase-based Spire, the SQL-into-C++ MemSQL and OLTP engine NuoDB, although there are many more.

Still, it seems clearer now than at any point in the past few years that SQL truly is not going anywhere, despite the early hype from the NoSQL camp. The most-important question that SQL shops might have to ask themselves now is not whether to stick with the technology they know and (maybe) love, but which flavor to go with to suit their particular needs around scale and performance. Thanks to technologies like ScaleArc, they might not have to swap their database at all.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The fourth quarter of 2012 in cloud
• Big data 2013: key trends and companies to watch