Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Big data 2013: key trends and companies to watch

Like a bad episode of Hoarders, people love to store all things digital, most of which will never be accessed again. And, like a bad episode of Storage Wars, our love of storing crap means we need more places to store it. Today’s content has outgrown even the hydro-electric dam powered Mega Data Centers built just yesteryear. Increasingly, operators are turning to distributing their information across multiple geographically dispersed data centers. As the number, size, and distances between the data centers have steadily grown, timing distribution and accuracy has likewise grown in importance in keeping the data centers in sync.

In a previous article I discussed new standards being developed to increase the accuracy of timing for the internet and other IP-based networks. Current systems and protocols offer milliseconds of accuracy. But that just isn’t enough as we depend more on real-time information and compute, storage and communications networks become more distributed. While people often cite the importance of timing on mobile backhaul networks in the next-genration LTE-Advanced networks,there has been less publicity around the need for these new timing technologies in the continued growth of data centers.

The rise of Hadoop in an age of digital garbage

Massive storage of data appears to occur in periods, very analogous to dinosaur evolution. A database architecture will rise to the forefront, based upon its advantages, until it scales to the breaking point and is completely superseded by a new architecture. At first, databases were simply serial listed values with row/column arrangements. Database technology leapt forward and became a self-sufficient business with the advent of relational databases. It appeared for a while relational databases would be the end word in information storage, but then came Web 2.0, social media, and the cloud. Enter Hadoop.

A centralized database works, as the name suggests, by having all the data located in a single indexed repository with massive computational power to run operations on it. But a centralized database cannot hope to scale to the size needed by today’s cloud apps. Even if it could, the time needed to perform a single lookup would be unbearable to an end user at a browser window.

Hadoop de-centralizes the storage and lookup, as well as computational power. There is no index, per se. Content is distributed across a wide array of servers, each with their own storage and CPU’s, and the location and relation of each piece of data mapped. When a lookup occurs, the map is read, and all the pieces of information are fetched and pieced together again. The main benefit of Hadoop is scalability. To grow a database (and computational power), you simply keep adding servers and growing your map.

Even Hadoop is buried under mounds of digital debris

It looked like Hadoop would reign supreme for generations to come, with extensions continuously breathing new life into the protocol. Yet, after only a decade, databases based upon Hadoop such as Facebook are at the breaking point. Global traffic is growing beyond exponential, and most of it is trash. Today’s databases look more like landfills than the great Jedi Archives. And recently hyped trends such as lifelogging suggest the problem will get much worse long before it gets better.

The main limitation of Hadoop is that it works great within the walls of a single massive data center, but is less than stellar once that database outgrows the walls of a single data center and has to be run across geographically separated databases. It turns out the main strength of Hadoop is also its Achilles heel. With no index to search, every piece of data must be sorted through, a difficult proposition once databases stretch across the globe. A piece of retrieved data might be stale by the time it reaches a requester, or mirrored copies of data might conflict with one another.

Enter an idea keep widely dispersed data centers in sync — Google True Time. To grossly oversimplify the concept, True Time API adds time attributes to data being stored, not just for expiration dating, but also so that all the geographically disparate data centers’ content can be time aligned. For database aficionados, this is sacrilegious, as all leading database protocols are specifically designed to ignore time to prevent conflicts and confusion. Google True Time completely turns the concept of data storage inside out.

Introducing Spanner

In True Time, knowing the accurate “age” of each piece of information, in other words where it falls on the timeline of data, allows data centers that may be 100ms apart to synchronize not just the values stored in memory locations, but the timeline of values in memory locations. In order for this to work, Google maintains an accurate “global wall-clock time” across their entire global Spanner network.

Transactions that write are time stamped and use strict two phase locking (S2PL) to manage access. The commit order is always the timestamp order. Both commit and timestamp orders respect global wall-clock time. This simple set of rules maintains coordination between databases all over the world.

However, there is an element of uncertainty introduced into each data field, the very reason that time has been shunned in database protocols since the dawn of the data itself.

Google calls this “network-induced uncertainty”, denoted with an epsilon, and actively monitors and tracks this metric. As of summer 2012, this value was running 10ms for 99.9 percent (3 nines) certainty. Google’s long term goal is to reduce this below 1ms. Accomplishing this will require a state of the art timing distribution network, leveraging the same technologies being developed and deployed for 4G LTE backhaul networks.

A modest proposal

While True Time was most likely developed to improve geographic load balancing, now that accurate time stamping of data exists, the possibilities are profound. The problems associated with large databases go beyond simply managing the data. The growth rate itself is unsustainable. Data storage providers must do more than grow their storage, they must also come up with ways to improve efficiencies and ebb the tsunami of waste that is common in the age of relatively free storage.

It’s a dangerous notion, one simply must challenge the basic tenet that all data is forever. Our minds don’t work that way, why should computers? We only hold on to key memories, and the further the time from an event, the fewer the details are held. Perhaps data storage could work similarly. Rather than delete a picture that hasn’t been accessed in a while, a search is performed for similar photos and then only one kept. And as time passes, perhaps rather than simple deletion, a photo is continuously compressed, with less information kept, until the photo memory fades into oblivion. Like that old Polaroid hung on the refrigerator door.

Jim Theodoras is director of technical marketing at ADVA Optical Networking, working on Optical+Ethernet transport products.

Dinosaur image courtesy of Flickr user Denise Chen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• 9 Companies that Pushed the Infrastructure Discussion in 2010

That’s why as Hadoop keeps thundering toward its destination as the de facto data platform for next-generation applications, companies such as Cloudera and Hortonworks that are making a killing off it might want to stop and thank the guys from Powerset for building HBase. Because the database — a columnar Google BigTable clone that runs on top of the Hadoop Distributed File System — is so fast and scalable, it’s helping Hadoop find a home in companies and with applications that HDFS and MapReduce alone might not have been able to penetrate so easily.

The latest HBase user I’ve come across is Gravity, the interest graph company that powers content recommendations for some of the biggest publishers on the web.

From big MySQL at MySpace to big data with HBase

Its co-founders were all senior executives at MySpace, including Gravity CTO Jim Benedetto, who was SVP of technology for the social networking pioneer. He was actually MySpace’s first architect and helped build platform’s MySQL database. Although MySpace never reached Facebook’s scale, it did have 150 millions users at its peak, all able to store unlimited numbers of wall posts, messages and photos. Benedetto eventually oversaw a 600-instance cluster that required about 30 database adminstrators to keep it up and running.

Benedetto (center) at Structure: Data 2012. (c) Pinar Ozger

So naturally, when it came time to build out the Gravity architecture, Benedetto opted for the MySQL he knew so well. Until about three years ago, he told me recently, that database held about 95 percent of the company’s data. At some point, though, Benedetto and his team realized they were spending way too much time keeping their MySQL environment up insteading of building new things, so it was time for a change.

It ultimately opted for HBase, but the decision wasn’t easy. “For us,” Benedetto said, “our data and algorithms are our company,” so making the move from a relational database to a column-based database that can serve MapReduce jobs was nerve-racking. After all, he explained, “You never want to migrate your data … and if you have to, you never want to migrate it more than once.” In fact, he added, “you’re not going back.”

But Benedetto says the move to HBase as Gravity’s primary data store has been “life-saving,” and it’s arguably a more important component of the company’s infrastructure than is Hadoop MapReduce. HBase handles the company’s real-time recommendation algorithms, and it does it across the entire Gravity platform rather than on a site-by-site basis. And although it’s not banking-grade when it comes to the consistency of transactions, Benedetto says it’s about 99.95 percent consistent in real time. Later on, batch MapReduce jobs swoop in and pick up whatever HBase dropped earlier, and process it all against the company’s graph algorithms.

An example of an interest graph from Gravity,

Scalable for sure, and getting easier to use

And although it took some serious engineering effort to get HBase operational when Gravity began working with it three years ago, Benedetto thinks HBase is getting to the point (as is rival NoSQL database Cassandra, he acknowledged) where one could safely call it “enterprise-ready.” Right now, he noted, “You’re not gonna to see HBase in a company that just buys Oracle because Oracle is the name and Oracle has been around for 20 years,” but for web startups that hope to reach a certain scale and even for existing companies that are running into the MySQL wall, he sees a shift occurring.

“The web farm is the easiest part of your infrastructure to scale because all it does is cost more money,” Benedetto explained. Databases, on the other hand, require a lot of thinking about how to migrate data, shard the database and otherwise make a piece of software likely designed for a handful of servers, max, spread across dozens or hundreds. HBase really eases the scaling process, as well as the subsequent management, he said. Now, Gravity’s 100-node HBase cluster has only two operations engineers dedicated to it.

Indeed, there are startups trying to capitalize on HBase by using it to power SQL and even MongoDB-compliant databases that can scale beyond what most relational databases can do.

Aside from scale HBase might soon start catching on because of the work companies like Gravity have been doing to make it more user-friendly. It might scale easily, but, as Benedetto noted, it’s not always easy to get started with — especially without some deep understanding of the intricacies of the underlying HDFS infrastructure. Last year, eBay VP of Experience, Search and Platforms Hugh Williams told me that although HBase is one of the big data tools the company is most excited about, it’s also the area where he’d like to see the most improvement.

To help alleviate some of the learning curve, Gravity has developed an open-source tool called HPaste that lets developers access data and run jobs on HBase data using Scala rather than the more-bloated Java programming language on which Hadoop and HBase are built. One of the biggest benefits of HPaste, Benedetto said, is that it lets new HBase developers see the data in a way that makes sense to them: HBase stores everything in byte arrays, he explained, and “when a human tries to read a byte array, it looks like ancient hieroglyphics.”

The Kiji architecture

Elsewhere, a startup called WibiData has created an open-source framework called Kiji that aims to provide a collection of high-level APIs that should make it easier to store different data types in and develop applications on HBase. The company envisions Kiji being to HBase what the Spring Framework has become to Java over the course of the past decade.

Hadoop’s weapon for the mainstream?

But user experience aside, a lot of companies already invested in Hadoop — aside from expert users such as Facebook — are starting to see the promise of HBase and are incorporating it into their architectures.

WibiData co-founder Christophe Bisciglia, who also co-founded Hadoop pioneer Cloudera in 2008, gave me his take on the state of HBase while discussing its role in the future of Hadoop earlier this year. ”If you talk to anyone from Cloudera or any of the platform vendors, I think they will tell you that a large percentage of their customers use HBase. It’s something that I only expect to see increasing,”  he explained. “… HBase is gonna be what takes Hadoop from an ETL and BI platform into a real-time application platform.”

The Cloudera Hadoop stack (Gravity uses Cloudera’s distro).

Benedetto appears to agree. He considers Hadoop as a whole incredibly important, almost on par with what Amazon Web Services did for computing resources, because it lets startups use commercial-grade open source software to do data storage and processing that previously was only available to massive web companies. “More and more … the shining star in that suite is HBase,” he said. “If I were Oracle, I’d be scared.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Defining Hadoop: the Players, Technologies and Challenges of 2011

On the Airbnb Nerd Blog on Thursday, engineers Andy Kramolisch and Ben Hughes, who worked on Neighborhoods and previously founded NabeWise, a neighborhood guide for American cities, explained the back-end process of aligning locations with neighborhoods.

Behind the scenes, Kramolisch said, a cartographer carves out the borders of neighborhoods. Then it’s time to match up hosts’ homes with the neighborhoods listed on Airbnb. On the site’s back end, Kramolisch said, the latitude and longitude of available homes are regularly associated with the various neighborhoods in a given city, if those neighborhoods are represented on Airbnb, through an internal system called Glop, short for Genome Location Pipeline. “For example, say you list your place, which is located at (12.333568650219718, 45.43647998034738). The next time Glop runs, it will correctly identify your listing as being in San Marco,” he said.

It’s not as if Neighborhoods works with “insane amounts of data,” Kramolisch said. Still, up-to-date data on places to stay in neighborhoods needs to be served up quickly, so users aren’t kept waiting in front of their screens. Data changes fast, and an SQL database wouldn’t work because of “mass updates,” Kramolisch said. So an internal NoSQL database in cooperation with Amazon Web Services’ managed DynamoDB NoSQL database service was considered. But DynamoDB couldn’t handle Airbnb’s storage needs. So the engineers turned to the Memcached key-value store for quickly serving up data by keeping it in memory.

In going with Memcached, Airbnb is making the same choice as Facebook, Etsy and other companies that operate at webscale. Location is the top criterion for Airbnb travelers, Kramolisch says, and the fast service Memcached enables — 35 milliseconds on average, to be precise — is the kind of solution that could help Airbnb focus on giving customers more of what they want from the site, when they want it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• Themes for a connected world: GigaOM RoadMap review
• Flash analysis: Collaborative consumption – a first look at the new web-sharing economy

On Wednesday, MarkLogic’s success was validated again, as the company announced a $25 million round of venture funding, bringing the total it has raised to $71.2 million. Sequoia Capital and Tenaya Capital led the round; CEO Gary Bloom and other MarkLogic executives also contributed.

MarkLogic like to tout the fact that it’s geared for enterprise use. Features such as high availability, replication, clustering and ACID compliance help differentiate the company from other NoSQL databases, Bloom told me. And although the company is taking in revenue and looks robust enough to go public now, Bloom said he would rather boost revenues to the point that MarkLogic could sustain success after an IPO.

Rather than go after the revenues that open-source NoSQL databases generate, Bloom said he wants to take away database marketshare from legacy companies peddling SQL databases, including IBM, SAP and Bloom’s previous employer, Oracle. That means MarkLogic salespeople will have to convince slower-to-change enterprises on the reality that relational databases might not be the best choice if they want to take advantage of unstructured data. MarkLogic also will have to put up with fellow NoSQL players that are adding enterprise functions, such as MongoDB,

But if MarkLogic’s plan turns out to be fruitful, a public offering could come within a year or two, Bloom said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Putting Big Data to Work: Opportunities for Enterprises
• A near-term outlook for big data

ParElastic sits in between the application and the underlying database and lets developers scale without having to resort to complicated sharding or maybe even moving the database back in-house where they can run it on a bigger server. Architecturally, Founder and CEO Ken Rugg told me, ParElastic’s Database Virtualization Engine is similar to a parallel database system, although it functions more like middleware that manages multiple database instances as one and is designed for operational rather than analytic workloads.

Because it intelligently balances database load and distributed data across servers, ParElastic is ideal for multitenant situations where multiple users, applications or services are accessing the database simultaneously, Rugg added.

Now, anyone familiar with the next-generation database market might think they’ve heard this story before, and they kind of have. The NoSQL database movement rode into town on the promise of high scalability, and the NewSQL movement furthered that story by bringing scale-out performance to SQL. Some of these databases are even available as cloud services.

However, Rugg explained, there’s a big difference between these options and what ParElastic does. Namely, while NoSQL and NewSQL options require deploying an entirely new database and likely rewriting some application code, ParElastic’s software just overlays customers’ existing cloud databases. Rugg said about half of its early users are running standard MySQL versions on Amazon Web Services, while the rest are spread across cloud providers such as Rackspace, Joyent and LiquidWeb.

Some ParElastic users actually manage existing SQL services such as Amazon’s Relational Database Service and Google Cloud SQL. One even uses it to manage an in-house database environment. And technically, Rugg noted, ParElastic could manage cross-cloud database deployments but, because of the inherent latency hit that would entail, “we wouldn’t recommend that.”

However, he said, the biggest beneficiary of ParElastic aside from the company itself might well be AWS. It is by far the most widely used cloud in the world, but when users reach the limites of their single database instances, Amazon usually tells them to look into sharding or perhaps transitioning to DynamoDB. “None of those are really too friendly for Amazon keeping their customers moving forward in their cloud,” Rugg said.

Further, although certain cloud providers offer better CPU, IO or network performance than AWS does (Rugg cited Rackspace as being particularly strong on IO performance, for example, and ProfitBricks as looking promising on the network front), “Amazon is sort of the lowest common denominator in a number of ways,” Rugg explained. The economics and performance requirements vary from application to application, of course, but ParElastic could help stitch together a number of commodity AWS instances to provide suitable performance at a lower cost than might be possible using the biggest, fastest instances from other providers.

Having watched the cloud market unfold as it has, though, Rugg and ParElastic aren’t banking on AWS — which has a reputation for launching services that compete with startup ecosystem partners — as the future of the business. By supporting other cloud providers that are gaining acceptance (aside from the ones Rugg noted, Google has been impressing some with the performance of its Compute Engine service), ParElastic is in a pretty good position to handle whatever cloud-database market shifts might occur.

“Even if Amazon comes out and says ‘We’re going to replace you with something we built back in the lab,’ that puts us in a great position in terms of validating the market,” Rugg said.

ParElastic’s existing investors — Point Judith Capital,  CommonAngels and LaunchCapital — also participated in the Series A round, which brings the company’s total venture capital to $8.7 million.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Big data 2013: key trends and companies to watch
• Understanding and managing the cost of the cloud

According to 10gen CEO Max Schireson, Carey’s presence will be important for the company, which has more than 200 employees and should top 500 in the next couple years. While she helps grow and build the corporate infrastructure in the finance, legal and HR departments,  Schireson can spend more time working directly with customers, partners and products.

Carey told me she’s excited about getting back into a high-growth company, especially one like 10gen that has a disruptive technology and open-source business model. Granted, open-source business models do bring their own unique set of challenges on top of those associated with startup businesses, she noted, but they also open up doors for new business.

“I think 10gen has all the elements there to make that a really good fit for me,” Carey said.

Schireson said Carey’s hire isn’t indicative of a forthcoming 10gen IPO, but that is something on which the company is focused. However, he added, some great companies have went public rather late in their lives, so 10gen isn’t rushing that decision and is instead keeping its attention of product development. The company has raised $81 million in venture capital since launching in 2007.

The addition of Carey is just the latest in a series of executive hires at 10gen that includes a handful of veteran big data and database industry vice presidents, as well as Senior Vice President of Worldwide Sales Phillip Carty.

Despite some criticism of its ability to scale beyond a handful of nodes, MongoDB is easily the most-used NoSQL database around, largely because it’s so easy for even novice NoSQL developers to work with and performs well with smaller data volumes. And the MongoDB ecosystem that 10gen helped catalyze is big, growing and potentially very lucrative. There are a handful of popular cloud services around such as MongoLab and MongoHQ, and Rackspace just bought its way into the mix by acquiring ObjectRocket.

Schireson said 10gen might consider some acquisitions that help advance the company’s MongoDB mission, but isn’t really looking at buying its way into the position of a one-stop NoSQL shop right now.

“Right now we’re 100 percent focused on MongoDB,” he said. “Everything we do is somehow related to that.”

This story was updated at 9:22 a.m. to correct the amount of venture capital 10gen has raised to $81 million from $86 million. The company changed the official amount between the time this post was written and the time of its publication.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How devops can reduce cycle times
• AWS Storage Gateway jolts cloud-storage ecosystem

Although, of all LinkBench’s features — and you can read all about them in a Facebook Engineer wall post from Monday morning — probably the biggest is that it’s open source and built to be extensible. One of the biggest problems with benchmarks overall is that they rarely align with actual production workloads inside the companies that are supposed to care about them. In this case, for example, a benchmark for measuring the performance of Facebook’s massive MySQL+memcached+Flashcache database architecture against its massive social graph and transaction activity would be all but worthless unless someone was just planning to rebuild Facebook.

I’ve written in the past that perhaps crowdsourced benchmarks are the wave of the future: essentially a compiled set of statistics and best practices as more companies test different database (or Hadoop) technologies on different hardware setups against different workloads and publish the results. Everything will of course vary by the exact details within any given environment, but it would be a good way to get a sense of how a particular stack might, or perhaps should, fare.

But an open source benchmark tuned for a specific use case — social graphs — by probably the world’s foremost expert on that use case is interesting, too. Anyone else trying to serve data from their own social graphs can benefit from some of LinkBench’s more-prominent features, such as its ability to generate “large synthetic social graphs,” while tuning it to the specifics of their own infrastructure. After all, it might be that your app has different requirement around reading versus writing data, and it’s very possible you’re not using MySQL, either.

Or maybe you are using MySQL and want to see how a newer database technology might handle your graph workload. That, by the way, is one of the reasons Facebook built LinkBench, according to this post.

At any rate, the social web is all about graphs, and database performance really matters for anyone trying to build a service that stays online and provides a pleasant user experience. Say what you want about Facebook, but its services perform, so the bar is set high for anyone trying to dethrone it or at least to build something than can attract an equally large and devout following.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Is the Future of Enterprise Completely Open Source?
• How devops can reduce cycle times
• Migrating media applications to the private cloud: best practices for businesses

Emil Eifrem, CEO of Neo Technology, touted the power of Neo4j and other graph databases to show relationships among disparate varieties of data with nodes, edges and key-value properties. (Think of Facebook’s Graph Search as one version.) The style takes inspiration from the connections among neurons and synapses inside the brain, Eifrem said. But, like other NoSQL databases, Neo Technology’s Neo4j product doesn’t use the SQL programming language, which could limit its adoption among enterprises.

Damian Black, CEO of SQLstream, touted his database’s use of SQL, calling it “lingua franca for data management.” Sure, it isn’t the easiest language to use. Still, “you know it’s going to save, it’s going to work,” he said. “It’s auto-optimizing. It’s proven.” Plus, it might be easier to find developers who can use it. As specialized databases get more attention, that’s become a more important point, said the moderator of the talk, GigaOM Research Analyst David Linthicum.

Different databases have different sweet spots. For Ryan Garrett, vice president of product of MemSQL, it’s comparing real-time data — from the trading floor, say — with recent historical data from perhaps a day or a week ago. Andrew Cronk, CEO of TempoDB, said his database excels at crunching time-series data in long columns coming off of many new connected devices.

Black believes storing data in memory provides a clear advantage. “It’s obviously going to be faster if you’re pulling it from memory,” he said. Eifrem took issue with that notion, saying that Neo4j runs on wherever sufficient memory is available. “Generally speaking, we want to be as horizontal as possible,” he said.

Legacy database vendors such as Oracle still command large swaths of the database market. But specialized databases such as the four on display here could keep chipping away as data sets get larger and larger. Because there are so many flavors, a few databases could become leaders, rather than just one, as they really do have different strengths and weaknesses and use-case sweet spots. At least they do for now.

Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below.

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Listening platforms: finding the value in social media data
• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012

10gen had greater business adoption in mind last year when it raised $42 million and vowed to focus on research and development to improve MongoDB. Now, around 60 percent of customers are enterprises, said Kelly Stirman, the company’s direct of product marketing.

Once signed up for the MongoDB Enterprise software, customers can use their own on-premise hardware to run an extension of the MongoDB Monitoring Service to track MongoDB deployments with more than 100 metrics and receive alerts. MongoDB Enterprise comes certified for deployment on several operating systems. It also supports the Kerberos authentication protocol, which is popular among insurance companies and banks, and can hook in to customers’ existing monitoring services, such as Nagios. And it introduces roles for giving certain abilities to certain database users.

New features in the MongoDB 2.4 release available to all users include full-text search for querying the database, an option to evenly shard data across machines, more accurate measurements of the distance between locations, the ability to count items in a database 20 times faster than before and the ability to maintain and query leaderboards of, say, the top 50 scorers in a baseball league.

The NoSQL database market is crowded, and differentiation is important. That’s why it’s a good thing 10gen, which is based in New York and Palo Alto, Calif.-and has other offices in Australia, England, Ireland and Spain, will increase its headcount by 75 percent in the next year, Stirman said. The time between product releases is getting shorter and shorter, he said, which means that still more improvements could be just a few months away.

Feature image courtesy of Flickr user junyaogura.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How devops can reduce cycle times
• A near-term outlook for big data