That’s the problem that David Crawford is trying to fix with his cloud analytics startup MetricaDB, one of our Structure 2013 LaunchPad finalists. It’s a software-as-a-service (SaaS) tool for individual analysts who don’t care whether the data is held in a NoSQL database or behind a Salesforce or Google Analytics API – they just want to deal with it in one place.

Crawford’s a one-man band at the moment and the 8-month-old MetricaDB is a bootstrapped affair, but it seems pretty effective for such an early product (albeit a product that’s already evolved from simply enabling SQL queries against MongoDB data).

After signing up, you click on the different SaaS products you use to gain a view on your customer – Mailchimp, Sendgrid, Salesforce, Stripe and so on – and end up with a SQL console that puts all that data in straightforward tables. Then you can run SQL queries (MetricaDB is built on Postgres) or what have you and export the results to Excel.

So, for example, a user could look at support tickets on his or her company’s cloud CRM system and join it to the Stripe billing system to see how the biggest spenders feel they’re being treated.

“You’re not looking up the API libraries or running a script,” Crawford said. “You attach multiple services and they all just appear as tables in the same database – pulling the data together and doing something across the different sources is just a SQL join.”

Crawford is pitching MetricaDB as a product for individual analysts who are “tired of waiting for engineers to get the data they want and have them deliver an Excel spreadsheet” – down the line he wants to add enough features to make it a fully-fledged enterprise play, but right now he’s attacking the market from the bottom up.

“What I’ve learned about security in the SaaS world is it’s largely a brand perception issue,” Crawford said. “I need to build the brand so people will be comfortable giving me access to their proprietary data and knowing I’ll steer it well.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• The fourth quarter of 2012 in cloud
• A near-term outlook for big data

The company took so long to officially launch, CEO Eric Kish told me, because it took such a long time to build. The 28msec history goes like this: The early investors are database industry veterans (one was employee No. 6 at Oracle) who, at some point in 2006, envisioned an explosion in data formats and databases. Their solution was to create a platform able to extract data from any of these sources, transform it into a standard format, and then let users analyze it using a single query language that looks a lot like the SQL they already know. 28msec is based on the open source JSONiq and Zorba query languages and will be available as a cloud service.

That’s about all Kish is willing to spill right now with regard to the technology.

As for the company itself, it has been staffed thus far primarily by Ph.Ds. in query technologies from ETH Zurich in Switzerland, where co-founder Donald Kossmann is a professor. Every year since 28msec was founded, it has hired one of his graduates to help build the product. The company brought on Kish, a serial entrepreneur, as CEO in 2012.

28msec was originally based in Zurich, but is in the process of shifting its base to Palo Alto, where Kish lives. It has raised $5.5 million in capital from friends and family, and already has paying customers.

As for the name, 28msec, it’s a reference to the time it takes for a database to access data stored on a hard disk. After the headquarters, maybe that name will be the next thing to change given the prevalence of flash and RAM as database storage media. “Seven years later,” Kish acknowledged, “it’s not relevant anymore.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• The fourth quarter of 2012 in cloud
• Infrastructure Q1: Cloud and big data woo enterprises

What is Accumulo?

The technological linchpin to everything the NSA is doing from a data-analysis perspective is Accumulo — an open-source database the agency built in order to store and analyze huge amounts of data. Adam Fuchs knows Accumulo well because he helped build it during a nine-year stint with the NSA; he’s now co-founder and CTO of a company called Sqrrl that sells a commercial version of the database system. I spoke with him earlier this week, days before news broke of the NSA collecting data from Verizon and the country’s largest web companies.

Adam Fuchs

The NSA began building Accumulo in late 2007, Fuchs said, because they were trying to do automated analysis for tracking and discovering new terrorism suspects. “We had a set of applications that we wanted to develop and we were looking for the right infrastructure to build them on,” he said.

The problem was those technologies weren’t available. He liked what projects like HBase were doing by using Hadoop to mimic Google’s famous BigTable data store, but it still wasn’t up to the NSA requirements around scalability, reliability or security. So, they began work on a project called CloudBase, which eventually was renamed Accumulo.

Now, Fuchs said, “It’s operating at thousands-of-nodes scale” within the NSA’s data centers. There are multiple instances each storing tens of petabytes (1 petabyte equals 1,000 terabyes or 1 million gigabytes) of data and it’s the backend of the agency’s most widely used analytical capabilities. Accumulo’s ability to handle data in a variety of formats (a characteristic called “schemaless” in database jargon) means the NSA can store data from numerous sources all within the database and add new analytic capabilities in days or even hours.

“It’s quite critical,” he added.

What the NSA can and can’t do with all this data

As I explained on Thursday, Accumulo is especially adept at analyzing trillions of data points in order to build massive graphs that can detect the connections between them and the strength of the connections. Fuchs didn’t talk about the size of the NSA’s graph, but he did say the database is designed to handle months or years worth of information and let analysts move from query to query very fast. When you’re talking about analyzing call records, it’s easy to see where this type of analysis would be valuable in determining how far a suspected terrorist’s network might spread and who might be involved.

Stewart Baker, former NSA general counsel under George W. Bush, wrote on his blog Thursday that this type of data could also be used for for general pattern recognition — the kinds of stuff that targeted advertisers love to do. Only, instead of the system serving someone an ad because of what they’ve been searching for and the operating system they’re using, Baker presented the hypothetical of “[an] American who makes a call to Yemen at 11 a.m., Sanaa time, hangs up after a few seconds, and then gets a call from a different Yemeni number three hours later.”

The big legal question here is around probable cause and whether the government should further investigate this caller based on call patterns similar to those of known terrorists, but the big data question is around false positives. Baker’s hypothetical might appear pretty cut and dry but, data scientist Joseph Turian explains, call records in general probably don’t offer too strong of a signal and could lead to situations where innocent behavior patterns looks a lot like nefarious ones. “But once you start connecting the dots with other pieces of information you have from other sources,” he said via email, “you can start making more predictions.”

This is where a program like PRISM, the NSA’s reported effort to collect data straight from the likes of Google, Facebook and Apple could come into play. If you’re able to tie a name or web account to a phone number, you can figure out all sorts of information. If you can prove that certain people are radical Islamists, for example, you can start to infer more things about the others in that social graph.

And if Sqrrl’s capabilities are any indicator of what Accumulo is supporting within the NSA, the agency can perform a lot of simpler functions on its data as well. In addition to graph processing, said Ely Kahn, Sqrrl’s co-founder and VP of business development, their product includes pre-packaged analytic capabilities around SQL queries and full-text search, and also supports streaming data. This means Sqrrl’s version can support any number of interesting use cases — from processing data as it hits the system to keeping a massive index that can be searched in the same way someone searches the web.

How much data is the NSA collecting? Follow the money

We’re not quite sure how much data the two programs that came to light this week are actually collecting, but the evidence suggests it’s not that much — at least from a volume perspective. Take the PRISM program that’s gathering data from web properties including Google, Facebook, Microsoft, Apple, Yahoo and AOL. It seems the NSA would have to be selective in what it grabs.

Assuming it includes every cost associated with running the program, the $20 million per year allocated to PRISM, according to the slides published by the Washington Post, wouldn’t be nearly enough to store all the raw data — much less new datasets created from analyses — from such large web properties. Yahoo alone, I’m told, was spending over $100 million a year to operate its approximately 42,000-node Hadoop environment, consisting of hundreds of petabytes, a few years ago. Facebook users are generating more than 500 terabytes of new data every day.

Using about the least-expensive option around for mass storage — cloud storage provider Backblaze’s open source storage pod designs — just storing 500 terabytes of Facebook data a day would cost more than $10 million in hardware alone over the course of a year. Using higher-performance hard drives or other premium gear — things Backblaze eschews because it’s concerned primarily about cost and scalability rather than performance — would cost even more.

Even at the Backblaze price point, though, which is pocket change for the NSA, the agency would easily run over $20 million trying to store too many emails, chats, Skype calls, photos, videos and other types data from the other companies it’s working with.

Actually, it’s possible the intelligence community is taking advantage of the Backblaze designs. In September 2011, Backblaze CEO Gleb Budman says, he met with CIA representatives who discussed that agency’s five-year plan “to centralize data services into a large private cloud” and how Backblaze’s technology might fit into it. Its plans for analyzing this data, as illustrated in the slide below (and discussed by CIA CTO Ira “Gus” Hunt at Structure: Data in March), seem to mirror what the NSA has in mind.

Whatever type of gear the NSA is using, though, and how ever much it’s spending on the Verizon data or PRISM specifically, we do know the agency is spending a lot of money on its data infrastructure. There are those dozens (at least) of petabytes of overall data in Accumulo, and the agency is famously building a 1-million-square-foot, $1.5 billion data center in Utah. It recently began construction on a 600,000-square-foot, $860 million facility in Maryland.

Policies are in place

Sqrrl’s Kahn — who previously served as director of cybersecurity strategy at the National Security Staff in the White House — says even with all the effort it’s putting into data collection and analysis, the NSA really is concerned about privacy. Not only are there strict administrative and legal limitations in place about when the agency can actually search through collected data (something Stewart Baker explains in more detail in a Friday blog post), but Accumulo itself was designed with privacy in mind.

The system itself is designed to make sure there’s not a free-for-all on data, another individual familiar with Accumulo said.

It has what Kahn and Sqrrl CTO Fuchs described as “cell-level” security, meaning administrators can manage access to individual pieces of data within a table. Furthermore, Fuchs explained, those policies stick with the data as it’s transformed as part of the analysis process, so someone prohibited from seeing it won’t be able to see it just because it’s now part of a different dataset. When data would come into the NSA from the CIA, he said, there were policies in place around who could see it, and Accumulo helped enforce them.

Even agencies within the Department of Homeland Security are using or experimenting with Accumulo, Kahn added, because proposed legislation would put them in charge of ensuring privacy as cybersecurity data exchanges hands between the government and private corporations.

It’s ironic he acknowledged, but Accumulo actually flips the presumed paradigm that stricter security and privacy regulations mean less sharing. That might be a shallow victory for citizens concerned about their civil liberties, but data collection and sharing don’t seem likely to stop any time soon. At least it’s something.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?
• Will the Real-Time Web Bring High Performance to a System Near You?

At its core, the new standard — which encompasses the MongoDB API, data representation (BSON), query language and wire protocol — appears to be all about establishing a way for mobile and other next-generation applications to connect with enterprise database systems such as IBM’s popular DB2 database and its WebSphere eXtreme Scale data grid. MongoDB is already immensely popular among web and mobile developers who must deal with semi- and unstructured data, but its lack of transactional integrity (among other things) means MongoDB isn’t often deployed for “mission-critical” applications that require ACID compliance and consistent performance.

In theory, the new standards would MongoDB-based applications easily and securely access mission-critical database systems. This could usher in a new wave of flexible applications that add significant value by spanning multiple data systems. According to a press release, “Customers can begin to use these new features later this summer by pairing eXtreme Scale with MongoDB, and by running their MongoDB applications on DB2 directly.”

The companies are also seeking participation from other parties interested in developing standard methods for interacting with MongoDB.

However, there’s a bigger shift at play here than the development of a new database standard, and it has everything to do with IBM’s planned acquisition of cloud provider SoftLayer, also announced on Tuesday. If IBM wants to remain relevant as server sales and application platforms move to the cloud, it has to embrace the new business and application-development models that come along with cloud computing. IBM’s stable of enterprise developers might not be deploying mobile apps on Parse or Google any time soon, but they will look for alternative platforms if IBM doesn’t at least try to keep up with a changing landscape.

Coincidentally, SoftLayer and 10gen already have a strong partnership around hosting MongoDB applications in the cloud.

If IBM is still an IT kingmaker, that bodes very well for MongoDB, as well as for the OpenStack cloud computing platform that IBM is also backing. If IBM’s influence in this realm is slipping, though, one could argue that IBM needs MongoDB and OpenStack more than they need it.

I am awaiting comment from IBM and/or 10gen for more details on the scope of their partnership and this standard, and will update when I hear more.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• The fourth quarter of 2012 in cloud
• Infrastructure Q1: Cloud and big data woo enterprises

AWS has a managed NoSQL database service called DynamoDB that’s replicated across three availability zones to ensure its stays up. Google’s Cloud Datastore sounds eerily similar, according to the product’s website (although Google calls its product “NoSQL-like). It’s fully managed, built for speed and scale and is replicated across data centers. For some queries, Google even promises that Cloud Datastore will support ACID transactions.

Although the services advertise similar features in terms of availability and scalability, they’re quite different technically. Cloud Datastore is based on Google’s BigTable database (and a library called Megastore on top of it) while DynamoDB is based on Amazon’s Dynamo database. You can get details on Datastore  and how it works here. Pricing information is available here.

If its goal is to compete with AWS, though, Google’s cloud platform still has a long way to go. Yes, it has most of the key services in place and even some seeming advantages in certain areas, but it’s lacking the incredible breadth of services AWS offers — everything from virtual server instances to a devops service to a hosted data warehouse. It’s also lacking a seven-year reputation for being an all-around reliable platform and an ever-growing list of large-enterprise users.

Of course, there’s also an argument to be made that Google doesn’t really have to compete with AWS at all when it comes to cloud computing. AWS made a name for itself by  taking all the new workloads from startups and corporate developers who wanted to build new types of applications and didn’t want to deal with the IT department; Google has the same opportunity ahead of it. New programming languages like Go and the unique nature of the rest of Google’s services, Cloud Datastore included, could make it the go-to place for a class of developers that likes to push the envelope in terms of application design.

Oh, and Google has a little ace up its sleeve called Android. If someone is so inclined to develop mobile applications for the most-popular mobile operating system on the planet, there are worse places to host them.

This post was updated at 5:35 p.m. to clarify that DynamoDB and Cloud Datastore are based on different underlying technologies.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011

The MongoDB Backup Service, now in limited release with general release slated for the summer, lets customers determine how often they want to back up their databases at colocation facilities 10gen uses. If a user wants to back up every six hours, for example, then that user has many options to choose from in the way of restoring a database to a previous state. They can choose the version from six, 12, 18 or 24 hours ago. Restores require two-factor authentication and work across multiple shards. Customers pay only for the amount of backup that they use.

10gen, based in New York and Palo Alto, Calif., expects the service to be a hit not necessarily with big companies but with small and medium-sized businesses. “It allows them to focus on building out applications instead of worry about this operational part of the infrastructure,” said Kelly Stirman, director of product marketing at 10gen. Regardless of company size, the feature could be valuable for anyone working in Mongo with larger data sets.

Beyond that, backing up means users can move data from a production environment into a testing environment to look for issues so their production environment won’t be affected.

While many MongoDB users already back up their databases, the systems are typically homemade, Stirman said. The MongoDB Backup Service, by comparison, is more reliable.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Cloud and data first-quarter 2013: analysis and outlook
• How devops can reduce cycle times

The reunion comes courtesy of a merger between open source database services firm SkySQL (which supports both MySQL and MariaDB deployments for customers ranging from Harvard to Shutterstock) and a company called Monty Program — yes, as in Monty Widenius, who named MySQL after his oldest daughter My and its fork after his younger daughter, Maria.

So now we have Widenius and other ex-MySQLers such as Colin Charles back together with players such as MySQL co-founder David Axmark and former MySQL sales director Magnus Stenberg. Actually, that’s underselling the magnitude of what’s happened here: out of the 70 employees of the fused operation (which is continuing under the SkySQL name), 50 used to be at the original MySQL firm.

Open appeal

At the same time, MariaDB seems to be capitalizing on the disillusionment of some in the open source community with Oracle’s stewardship of MySQL — doing things like releasing extensions for the commercial version but not the free version was never going to win favor in that scene. Wikipedia migrated to MariaDB in the last few days, and the Fedora and OpenSUSE Linux distros will both make the jump in their next releases.

The MariaDB Foundation, which is busy sorting out its governance structure and which now claims SkySQL as an early member, also took on former Sun Chief Open Source Officer Simon Phipps as its CEO a week ago.

“It is a pleasure to have a company representing the reunited core team of our code base joining the Foundation at its inception,” Phipps said in a statement this week.

MariaDB the “bridge”

The fused team has a unique NewSQL proposition: not only is MariaDB fully compatible with MySQL, but it can also interface with newer NoSQL databases such as Cassandra and LevelDB. According to SkySQL CEO Patrik Sallner, SkySQL will continue to service both MySQL and MariaDB customers and won’t be forcing anyone to jump to MariaDB — but he expects many customers to make that leap nonetheless:

“Right now, because MySQL belongs to Oracle, it’s not necessarily perceived as independent. Linux is the default operating system in most enterprise contexts. Oracle, IBM and Microsoft control the vast majority of business in databases and most companies have at least two of these, which are not compatible with each other. And, as companies deploy new applications, they use new [NoSQL] database technologies to meet their needs.

“We believe that MariaDB has an opportunity to become a truly independent and interoperable open source database, meaning we can provide a solution that’s a neutral ground for companies. … Our aspiration is to start building this into a new form of database platform that ties together other databases in a seamless manner. By providing a bridge, we believe we can create more innovation.”

Sallner noted that there isn’t currently a great deal of difference between MySQL and MariaDB, apart from the latter’s “pluggable” approach to storage engines. “Using the SQL language allows us to be compatible with other databases, and we have a connect engine which allows us to add on-the-fly support for other data formats,” he said.

As a next step, Sallner said he hoped to see other database providers join the MariaDB Foundation, in order to maintain this open common ground. “We’re not competing against DB2 or Oracle or Microsoft today — we’re all serving different needs,” he said. So does he want to sign up Oracle itself? “That’ll be a stretch, but it would be a huge sign of success,” he laughed.

It’s not all bonhomie, though — Sallner reckons large internet companies will engage with MariaDB in a way that they haven’t with Oracle’s MySQL.

“We believe those companies are willing to contribute the work they’ve done back to MariaDB,” he said. “Facebook and Twitter have contributed substantial new features to MariaDB. They probably wouldn’t have contributed that to Oracle.”

UPDATE (10.55am PT): This piece originally and incorrectly stated that Widenius is the new SkySQL CTO, whereas he is in fact the CTO of the MariaDB Foundation. Widenius is on the board of SkySQL, but his role is non-operational.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How devops can reduce cycle times
• A near-term outlook for big data
• Is the Future of Enterprise Completely Open Source?

That’s why as Hadoop keeps thundering toward its destination as the de facto data platform for next-generation applications, companies such as Cloudera and Hortonworks that are making a killing off it might want to stop and thank the guys from Powerset for building HBase. Because the database — a columnar Google BigTable clone that runs on top of the Hadoop Distributed File System — is so fast and scalable, it’s helping Hadoop find a home in companies and with applications that HDFS and MapReduce alone might not have been able to penetrate so easily.

The latest HBase user I’ve come across is Gravity, the interest graph company that powers content recommendations for some of the biggest publishers on the web.

From big MySQL at MySpace to big data with HBase

Its co-founders were all senior executives at MySpace, including Gravity CTO Jim Benedetto, who was SVP of technology for the social networking pioneer. He was actually MySpace’s first architect and helped build platform’s MySQL database. Although MySpace never reached Facebook’s scale, it did have 150 millions users at its peak, all able to store unlimited numbers of wall posts, messages and photos. Benedetto eventually oversaw a 600-instance cluster that required about 30 database adminstrators to keep it up and running.

Benedetto (center) at Structure: Data 2012. (c) Pinar Ozger

So naturally, when it came time to build out the Gravity architecture, Benedetto opted for the MySQL he knew so well. Until about three years ago, he told me recently, that database held about 95 percent of the company’s data. At some point, though, Benedetto and his team realized they were spending way too much time keeping their MySQL environment up insteading of building new things, so it was time for a change.

It ultimately opted for HBase, but the decision wasn’t easy. “For us,” Benedetto said, “our data and algorithms are our company,” so making the move from a relational database to a column-based database that can serve MapReduce jobs was nerve-racking. After all, he explained, “You never want to migrate your data … and if you have to, you never want to migrate it more than once.” In fact, he added, “you’re not going back.”

But Benedetto says the move to HBase as Gravity’s primary data store has been “life-saving,” and it’s arguably a more important component of the company’s infrastructure than is Hadoop MapReduce. HBase handles the company’s real-time recommendation algorithms, and it does it across the entire Gravity platform rather than on a site-by-site basis. And although it’s not banking-grade when it comes to the consistency of transactions, Benedetto says it’s about 99.95 percent consistent in real time. Later on, batch MapReduce jobs swoop in and pick up whatever HBase dropped earlier, and process it all against the company’s graph algorithms.

An example of an interest graph from Gravity,

Scalable for sure, and getting easier to use

And although it took some serious engineering effort to get HBase operational when Gravity began working with it three years ago, Benedetto thinks HBase is getting to the point (as is rival NoSQL database Cassandra, he acknowledged) where one could safely call it “enterprise-ready.” Right now, he noted, “You’re not gonna to see HBase in a company that just buys Oracle because Oracle is the name and Oracle has been around for 20 years,” but for web startups that hope to reach a certain scale and even for existing companies that are running into the MySQL wall, he sees a shift occurring.

“The web farm is the easiest part of your infrastructure to scale because all it does is cost more money,” Benedetto explained. Databases, on the other hand, require a lot of thinking about how to migrate data, shard the database and otherwise make a piece of software likely designed for a handful of servers, max, spread across dozens or hundreds. HBase really eases the scaling process, as well as the subsequent management, he said. Now, Gravity’s 100-node HBase cluster has only two operations engineers dedicated to it.

Indeed, there are startups trying to capitalize on HBase by using it to power SQL and even MongoDB-compliant databases that can scale beyond what most relational databases can do.

Aside from scale HBase might soon start catching on because of the work companies like Gravity have been doing to make it more user-friendly. It might scale easily, but, as Benedetto noted, it’s not always easy to get started with — especially without some deep understanding of the intricacies of the underlying HDFS infrastructure. Last year, eBay VP of Experience, Search and Platforms Hugh Williams told me that although HBase is one of the big data tools the company is most excited about, it’s also the area where he’d like to see the most improvement.

To help alleviate some of the learning curve, Gravity has developed an open-source tool called HPaste that lets developers access data and run jobs on HBase data using Scala rather than the more-bloated Java programming language on which Hadoop and HBase are built. One of the biggest benefits of HPaste, Benedetto said, is that it lets new HBase developers see the data in a way that makes sense to them: HBase stores everything in byte arrays, he explained, and “when a human tries to read a byte array, it looks like ancient hieroglyphics.”

The Kiji architecture

Elsewhere, a startup called WibiData has created an open-source framework called Kiji that aims to provide a collection of high-level APIs that should make it easier to store different data types in and develop applications on HBase. The company envisions Kiji being to HBase what the Spring Framework has become to Java over the course of the past decade.

Hadoop’s weapon for the mainstream?

But user experience aside, a lot of companies already invested in Hadoop — aside from expert users such as Facebook — are starting to see the promise of HBase and are incorporating it into their architectures.

WibiData co-founder Christophe Bisciglia, who also co-founded Hadoop pioneer Cloudera in 2008, gave me his take on the state of HBase while discussing its role in the future of Hadoop earlier this year. ”If you talk to anyone from Cloudera or any of the platform vendors, I think they will tell you that a large percentage of their customers use HBase. It’s something that I only expect to see increasing,”  he explained. “… HBase is gonna be what takes Hadoop from an ETL and BI platform into a real-time application platform.”

The Cloudera Hadoop stack (Gravity uses Cloudera’s distro).

Benedetto appears to agree. He considers Hadoop as a whole incredibly important, almost on par with what Amazon Web Services did for computing resources, because it lets startups use commercial-grade open source software to do data storage and processing that previously was only available to massive web companies. “More and more … the shining star in that suite is HBase,” he said. “If I were Oracle, I’d be scared.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How to use big data to make better business decisions
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

On Wednesday, MarkLogic’s success was validated again, as the company announced a $25 million round of venture funding, bringing the total it has raised to $71.2 million. Sequoia Capital and Tenaya Capital led the round; CEO Gary Bloom and other MarkLogic executives also contributed.

MarkLogic like to tout the fact that it’s geared for enterprise use. Features such as high availability, replication, clustering and ACID compliance help differentiate the company from other NoSQL databases, Bloom told me. And although the company is taking in revenue and looks robust enough to go public now, Bloom said he would rather boost revenues to the point that MarkLogic could sustain success after an IPO.

Rather than go after the revenues that open-source NoSQL databases generate, Bloom said he wants to take away database marketshare from legacy companies peddling SQL databases, including IBM, SAP and Bloom’s previous employer, Oracle. That means MarkLogic salespeople will have to convince slower-to-change enterprises on the reality that relational databases might not be the best choice if they want to take advantage of unstructured data. MarkLogic also will have to put up with fellow NoSQL players that are adding enterprise functions, such as MongoDB,

But if MarkLogic’s plan turns out to be fruitful, a public offering could come within a year or two, Bloom said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Putting Big Data to Work: Opportunities for Enterprises
• A near-term outlook for big data