Feedly Co-founder Cyril Moutran

So engineers moved static content off the Feedly servers inside a data center and, somewhat ironically, onto Google App Engine, which scales very nicely, Moutran said. Dynamic content stayed put on the Feedly servers, which store terabytes of data, including indexed content from the feeds users subscribe to.

Less than a week later, Moutran said, “we saw another really, really crazy spike.” The site went down again. Developers took a look at the code that communicates between the client and Feedly servers, and tried to make the client more efficient, thereby reducing the load hitting the servers. “Then we had to order some more hardware,” Moutran said — load balancers, to be specific.

That second outage came on a Monday. As it turned out, Feedly gets more traffic on Monday than on any other day, and generally speaking traffic is higher on weekdays than on weekends. Desktop traffic picks up at around 8 a.m. local time and decreases around 6 p.m. Why? Many Feedly users look to the service “not so much in a casual context but more to catch up with what’s going on with the industry,” Moutran said. People use Feedly for work, in other words. Lawyers, designers, and writers are typical business users.

As many more users get on board — more than 3 million had joined since the Google announcement as of April 2, on top of 4 million users active before the announcement — more feeds pile up. The number of feeds is now up to 100 million, Moutran said.

With many more business users and a greater variety of content, monetization is a bigger question, and Feedly feels it must accelerate its efforts in that direction. The company, which is based in Palo Alto, Calif., and has 10 employees, is now looking at how it will introduce a premium or pro version later this year. Feedly could also add a way to take revenue by providing streams of publishers’ premium content inside the desktop and mobile versions of the application.

While plenty of people find Twitter handy for getting news, the migration of millions to Feedly shows the desire for a strong RSS reader still exists. If that desire keeps steady and if Feedly can keep adding features that interest users, it could turn Google’s trash into Feedly’s treasure.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The promise of SDNs in the enterprise
• A near-term outlook for big data
• The new IT manager, part 1

Drawn to Scale’s flagship product, called Spire, is a distributed data platform that’s built atop an optimized version of the Hadoop-based HBase database. HBase is what lets Spire scale cheaply and easily across. Its fully distributed index is what lets Spire read and write data at speeds that other approaches to scaling databases (e.g., sharding) can’t handle while maintaining the ability to handle rich queries.

To date, the company has been focused on letting users run massive SQL databases, but it has finally completed a lengthy process of rewriting parts of MongoDB to work with Spire, Founder and CEO Bradford Stephens (who’ll be participating in our Structure: Data event this week in New York) told me. The company had been keeping the work under tight wraps “because we didn’t know how long it was going to take to build,” he added.

“Our big vision for the market is providing people with a universal data platform,” Stephens said. After SQL — which accounts for the vast majority of databases in existence — MongoDB is a logical next step (although Spire also supports queries using Hadoop MapReduce). It’s the most-widely used NoSQL database by a longshot, but although many users love its functionality and tooling, the database is notoriously poor at scaling to meet the demands of big data or high performance.

“You just sort of top out once you max out the memory,” Stephens explained, adding that MongoDB often starts getting inefficient as it’s forced to scale across 50 or 10 servers. “[T]hat’s where we start getting efficient.”

Now, without changing a single line of code, he claims, MongoDB users can import their data onto Spire and start handing 200-plus terabytes with ease. Of course, he noted, this doesn’t mean MongoDB users will abandon the database entirely. It might be they keep it for running applications that don’t require it to scale beyond a single server, and then use Spire to store big data for analytical purposes.

Initially, Spire will just support data importation and the basic CRUD (create, read, update, delete) functions of MongoDB, Stephens said. Later this year, assuming users want it, Drawn to Scale will implement MongoDB’s native MapReduce functionality as well as its management features.

As data volumes and data stores continue to proliferate, though, Drawn to Scale isn’t the only startup trying to provide a one-stop shop experience. At least for analytics, Citus Data is building a Postgres-based database capable of analyzing SQL, Hadoop and MongoDB data, although each data store remains external. And there’s a whole group of companies merging SQL and Hadoop for analytic workloads that might be wise to consider supporting operational data stores such as MongoDB, as well.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?

It should come as no surprise, then, that the companies serving the country’s 1.3 billion people with their social media, e-commerce and information-discovery needs are very, very big, too. Here are some statistics that demonstrate their scale.

Alibaba Group

Taobao, the eBay-like e-commerce line of business from Chinese internet giant Alibaba Group, does a lot of business. On a single day — Nov. 11, 2011 — the company did a whopping 19 billion Yuan (or approximately $3.05 billion) in sales. According to Alibaba Group CTO and Alibaba Cloud Computing President Wang Jian, the company site surpassed the 1 trillion Yuan (about $160 billion) mark for 2012 revenue at the end of November. Alipay, the company’s version of PayPal, handles about 3 billion Yuan (about $480 million) in transactions every day.

By comparison, eBay posted $3.4 billion in revenue for the entire third-quarter this year. Amazon.com, with which Taobao also competes (although Alibaba also has a business-to-consumer division called Tmall), closed its third quarter with $13.8 billion in revenue.

The women’s shoe department on Taobao

Of course, Taobao and Alipay are just two of Alibaba’s expansive portfolio of services, which includes a much-publicized (although recently reduced) partnership with Yahoo.

That type of business means Aliaba needs a lot of servers. In a single year not too long ago, Jian told me, the company bought more servers than it had in previous five years combined. If you charted Alibaba’s server count now versus five years ago, he added, the previous number would look like zero. How big is its database? Enough to store data for more than 800 million items for sale.

Baidu

The Chinese search giant is ranked fifth in the Alexa internet rankings (behind Facebook Google, YouTube and Yahoo), which is evidence of its popularity. All those users, I’m told, result in an annual server growth approximately equal to the previous three years combined. It has been reported that Baidu is planning possibly the world’s largest data center — spanning 120,000 square meters, costing $1.6 billion, housing 100,000 servers (totaling 700,000 CPUs and 3 million cores) and storing 4,000 petabytes of data.

Tencent

Somtimes compared with Facebook in the United States (although it’s actually quite different), Tencent boasted more than 717 million users for its popular QQ messaging service as of September 2011. That number has surely grown. The company says its highest-ever number of concurrent users was more than 176 million, although there are often tens of millions (if not more than 100 million people) using it at any given time. An individual with some knowledge of the company’s infrastructure told me Tencent adds about 100,000 servers per year.

Tencent usage at 5:51 local time on Jan. 10, 2012.

Weibo

Weibo, the Twitter-like platform from internet new-school internet company Sina had more than 400 million users as of April 2012. That’s about twice the number Twitter claims. And the Chinese use Weibo a lot, for everything from micro-blogging to self-publishing. It might actually be a more important tool in China than Twitter is in the United States, sources told me, because while the government can censor official news outlets, it can’t possibly control the stream of information coming off Weibo. And that will mean even more growth.

Not (yet) innovators

However, despite their sheer scale, Chinese internet companies are, by most accounts, less technologically inclined than their American counterparts. The biggest reason — one I heard time and time again — is that these companies tend to view themselves as traditional businesses rather than technology companies, and that employees often strive to work up the management ladder rather than remain career engineers. This inevitably affects R&D budgets, makes companies less willing to take risks and reduces the pool of employees that really, deeply understand complex systems.

As an example, one might look at the server situation within China’s big four internet companies. Alibaba’s Jian told me that although his company is running all white boxes in its data centers now, it had a lot of legacy IBM gear in its data centers five years ago. I heard the same thing about Baidu. Tencent, someone told me, had 10,000 webscale servers fail in six months last year and is considering a move back to traditional boxes.

However, maybe these companies are coming around on innovation beyond just buying more-efficient gear. Tencent, Baidu and Alibaba, for example, are all members of the Facebook-led Open Compute Project for designing webscale hardware. Tencent and Baidu actually created their own rack-design specification, called Project Scorpio, that is being merged into Open Compute’s Open Rack design in 2013. They still don’t build their own servers like Google and Facebook do, preferring instead to push their custom specs on server makers, but many innovative American companies, including eBay, do the same thing.

Power specs of Open Rack

One has to assume that a closer working relationship between engineers at American and Chinese internet companies will spur even more changes in the tech culture there. Although technical talent comes relatively cheap in China, perhaps they’ll realize that highly skilled, forward-thinking engineers (and data scientists, for that matter) are something worth hanging onto and rewarding with high salaries.

As Facebook VP Frank Frankovsky told PCWorld in July as the Open Rack designs were unveiled, “We compete with those guys, but on the infrastructure side, if we can make our infrastructure more efficient, it makes everyone that much better. Where we differentiate our business is in the service we provide to our end users.”

That differentiation comes from in large part from an incredible investment in research and technology. If they want to be considered thought leaders in their field — and if they want to expand significantly into cloud computing (as Alibaba and Sina clearly want to do) — China’s internet companies will have to start matching their immense scale with demonstrated technological prowess.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Flash analysis: the future of Yahoo
• The promise of SDNs in the enterprise
• Facebook’s IPO filing: ideas and implications

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Migrating media applications to the private cloud: best practices for businesses
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

As with most of its changes to Hadoop over the years — including the recently unveiled AvatarNode — Corona came to be because Hadoop simply wasn’t designed to handle Facebook’s scale or its broad usage of the platform. What kind of scale are we talking about? According to Facebook engineers Avery Ching, Ravi Murthy, Dmytro Molkov,‎ Ramkumar Vadali, and Paul Yang in a blog post detailing Corona on Thursday, the company’s largest cluster is more than 100 petabytes; it runs more than 60,000 Hive queries a day; and its data warehouse has grown 2,500x in four years.

Further, Ching and company note — echoing something Facebook VP of Infrastructure Engineering Jay Parikh told me in September when discussing the future of big data startups — Hadoop is responsible for a lot of how Facebook runs both its platform and its business:

Almost every team at Facebook depends on our custom-built data infrastructure for warehousing and analytics, with roughly 1,000 people across the company — including both technical and non-technical personnel — using these technologies every day. Over half a petabyte of new data arrives in the warehouse every 24 hours, and ad-hoc queries, data pipelines, and custom MapReduce jobs process this raw data around the clock to generate more meaningful features and aggregations.

So, what is Corona?

In a nutshell, Corona represents a new system for scheduling Hadoop jobs that makes better use of a cluster’s resources and also makes it more amenable to multitenant environments like the one Facebook operates. Ching et al explain the problems and the solution in some detail, but the short explanation is that Hadoop’s JobTracker node is responsible for both cluster management and job-scheduling, but has a hard time keeping up with both tasks as clusters grow and the number of jobs sent to them increase.

Further, job-scheduling in Hadoop involves an inherent delay, which is problematic for small jobs that need fast results. And a fixed configuration of “map” and “reduce” slots means Hadoop clusters run inefficiently when jobs don’t fit into the remaining slots or when they’re not MapReduce jobs at all.

Corona resolves some of these problems by creating individual job trackers for each job and a cluster manager focused solely on tracking nodes and the amount of available resources. Thanks to this simplified architecture and a few other changes, the latency to get a job started is reduced and the cluster manager can make fast scheduling decisions because it’s not also responsible for tracking the progress of running jobs. Corona also incorporates a feature that divvies a cluster into resource pools to ensure every group within the company gets its fair share of resources.

The results have lived up to expectations since Corona went into full production in mid-2012: the average time to refill idle resources improved by 17 percent; resource utilization over regular MapReduce improved to 95 percent from 70 percent (in a simulation cluster); resource unfairness dropped to 3.6 percent with Corona versus 14.3 percent with traditional MapReduce; and latency on a test job Facebook runs every four minutes has been

Despite the hard work put into building and deploying Corona, though, the project still was a way to go. One of the biggest improvements currently being developed is to enable resource management based on CPU, memory and other job requirements rather than just the number of “map” and “reduce” slots needed. This will open Corona up to running non-MapReduce jobs, therefore making a Hadoop cluster more of a general-purpose parallel computing cluster.

Facebook is also trying to incorporate online upgrades, which would mean a cluster doesn’t have to come down every time part of the management layer undergoes an update.

Why Facebook sometimes must re-invent the wheel

Anyone deeply familiar with the Hadoop space might be thinking that a lot of what Facebook has done with Corona sounds familiar — and that’s because it kind of is. The Apache YARN project that has been integrated into the latest version of Apache Hadoop similarly splits the JobTracker into separate cluster-management and job-tracking components, and already allows for non-MapReduce workloads. Further, there is a whole class of commercial and open source cluster-management tools that have their own solutions to the problems Corona tries to solve, including Apache Mesos, which is Twitter’s tool of choice.
However, anyone who’s familiar with Facebook knows the company isn’t likely to buy software from anyone. It also has reached a point of customization with its Hadoop environment where even open-source projects from Apache won’t be easy to adapt to Facebook’s unique architecture. From the blog post:

It’s worth noting that we considered Apache YARN as a possible alternative to Corona. However, after investigating the use of YARN on top of our version of HDFS (a strong requirement due to our many petabytes of archived data) we found numerous incompatibilities that would be time-prohibitive and risky to fix. Also, it is unknown when YARN would be ready to work at Facebook-scale workloads.

So, Facebook plods forward, a Hadoop user without equal (save for maybe Yahoo) left building its own tools in isolation. What will be interesting to watch as Hadoop adoption picks up and more companies beging building applications atop it is how many actually utilize the types of tools that companies like Facebook, Twitter and Quantcast have created and open sourced. They might not have commercial backers behind them, but they’re certainly built to work well at scale.

Feature image courtesy of Shutterstock user Johan Swanepoel.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011

According to co-founder Alex Loddengaard, Levy began building MemCachier as a side project in mid-2011, and he hosted a private beta version in the Heroku add-on market so a friend could easily access the service. The team at Heroku saw the service, liked it and encouraged Levy to pursue it for real. Levy, who’s still in the middle of getting a Ph.D. from Stanford, called Loddengaard (who taught Levy while a teaching assistant at the University of Washington) and fellow Stanford Ph.D. candidate David Terei for help, and MemCachier launched in April 2012.

Alex Loddengaard

Landing Loddengaard wasn’t too tough. He had quit his job at software-development firm Atlassian, after beginning his career at Google and then following his boss Christophe Bisciglia to Hadoop pioneer Cloudera, where Loddengaard was a pre-funding employee. (MemCachier, by the way, now shares office space with Bisciglia’s new company, WibiData, in the former Atlassian headquarters.) He was living off his savings, had “built a bunch of stupid web apps that you never heard of” and was trying to figure out what to do next, he told me. And then Levy called.

Memcached, and MemCachier, are everywhere

Memcached is a popular open-source key-value system that speeds up web applications by caching certain data in the memory of distributed systems rather than on disk in the database itself. Facebook is widely cited as the largest user for the hundreds of terabytes it’s now storing in memcached, but, Loddengaard said, “Every company that needs to scale uses memcached.”

Aside from the core open source version, developers might choose the Couchbase’s eponymous NoSQL database (into which the popular memcached implementation Membase Server has been integrated) or its hosted Membase service called Memcache, which is available on Heroku. Another hosted option is AWS’s ElastiCache, a membased-compliant service available to developers building web applications on the Amazon EC2 cloud.

Since starting off on Heroku, MemCachier has already expanded to the AppHarbor and Cloud Control platforms, but Wednesday’s expansion represents  the company’s first real introduction to the public, Loddengaard said. Now, MemCachier is also available on AppFog, CloudBees and DotCloud — three popular PaaS offerings — as well as Amazon EC2.

Growing isn’t always easy

Moving to Amazon’s cloud, in particular, also meant a change in pricing to reflect a different class of user (e.g., AWS mega-user Netflix) than most PaaS offerings attract. Whereas MemCachier’s options on Heroku range from 100MB to 10GB in size, Amazon users can get up to a 100GB instance. Loddengaard said most Amazon EC2 users use more than a gigabyte of RAM for memcached, and ElastiCache actually starts out at 1.3GB.

MemCachier’s architecture, simplified.

Loddengaard acknowledges that trying to woo developers away from ElastiCache service on Amazon’s own platform won’t necessarily be easy, but he thinks the difference in approach between the two services favors MemCachier for a particular class of developers — those who don’t want to manage their infrastructure too closely. Whereas ElastiCache still requires users to manage their instances, as is the norm with Amazon’s lower-level infrastructure-as-a-service platform, MemCachier is about “no operations whatsoever,” he said. “Developers shouldn’t spend any time operating servers over developing software.”

That mindset has proven effective so far. Thanks to word of mouth alone, the bootstrapped MemCachier has been growing steadily in terms of revenue and users, now claiming more than 1,500 developers, but its broader footprint and some proactive marketing should mean sharp upticks in both areas. However, a jump in users — especially the larger ones that might come from Amazon EC2 — will probably require MemCachier to grow beyond its current three-person team. Of course, there are worse problems to have.

Feature image courtesy of Shutterstock user optimarc.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Platform as a Service in 2012

More often than not, they’re conducted on highly optimized systems running workloads that don’t necessarily mirror what anyone actually runs in production. If the results in question come from vendors rather than third parties, there’s a good chance they’ve only been published because the vendor was able to achieve the desired result. Thinking your experience will be equally as fast is like watching a fishing show on television and then hitting the water expecting bite after bite.

However, cloud computing and the advent of popular open source software such as Hadoop and NoSQL databases could change the way we do benchmarks. With relatively little cost and effort, anyone can conduct their own tests to see how their specific applications and configurations run on their specific infrastructure. Throw in a platform to share these results, and you have crowdsourced performance benchmarks free from vendor hype and the vacuum-like conditions of standardized tests.

Ideally, it ends working a lot like the crowdsourced medical platforms I’ve come across lately, PatientsLikeMe and the forthcoming Lucine Biotechnology. Rather than rely on claims from drug companies or even doctors whose knowledge is limited to published research, users share their own real-world experiences with drugs, symptoms and side effects, and learn from others like them what they might expect.

It’s might get worse before it gets better

However, with so many Hadoop distributions in the market now, and so much money at play, I’d prepare to hear a lot more chest-beating in the months to come about whose implementation is actually fastest. It actually has been going on for a while – in December, I detailed a series of purported and contested records from SGI, MapR and HPCC Systems on the Terasort benchmark — and not all the voices have yet to be heard. Like SGI before them, hardware partners such as Dell, HP and Cisco probably want to prove their reference architectures are the best.

And VMware has already published a study claiming that Hadoop actually performs faster on its vSphere hypervisor than on bare metal. If Hadoop workloads really do move to virtual machines, Hadoop vendors are going to have to prove themselves there, too. VMware’s study ran CDH3 (Cloudera’s third-generation distribution), but Hortonworks has been working closely with VMware lately and might have something to say. Of course, EMC Greenplum is actually under the same corporate umbrella as VMware and can’t afford to be seen as slower than the competition on virtualized servers.

In cloud computing, too, providers have spent the past few years arguing against the idea of cloud servers as commodities by claiming their systems offer the best performance. There have been plenty of boasts (sometimes, perhaps, misleading) and quite a few attempts to benchmark cloud system and network performance (GigaOM Pro subscription req’d) With that much of the cloud market still up for grabs, we’re not yet done hearing about whose cloud is the biggest and the fastest (see, for example, Google’s emphasis on performance when it launched Compute Engine last month).

But it should get better

Despite all the effort by vendors and cloud providers to claim superiority, though, the truth is that it’s easier than ever for users of next-generation software and services to run their own tests. Hadoop or NoSQL databases aren’t expensive Oracle software that needs to run on scale-up, big-iron systems; they’re all free to download and can run on small clusters of commodity boxes. For applications that are going to run in the cloud, renting a few instances from a cloud provider might only cost a few bucks.

Sure, it might take a little time to configure everything (although software vendors might be willing to help), but isn’t that effort worth it in the end?

Use cases and test results that demonstrate the value of crowdsourcing in-the-wild performance metrics are everywhere on corporate technology blogs across the web. Earlier this month, for example, Medialets discussed how it tested its Hadoop workload (on a cluster of rented physical machines) and found that Cloudera actually outperformed the supposedly faster MapR for that job. This week, video-transcoding service Zencoder shared some interesting (if not always surprising) results when it compared Amazon Web Services’ highest-powered cloud instances to Google Compute Engine’s best.

Aggregated and indexed on a single platform, these types of experiences could help quiet the boasts from vendors and industry organizations touting their latest benchmark results. I’d argue it matters a lot more to a systems architect to know the production throughput of someone running a similar application on similar resources than to know how fast a generic workload ran in a lab on a setup he doesn’t have. Add some analytics to this information, and ideal configurations for different application types and data volumes might begin to emerge.

Cloud computing and open source software have freed IT practitioners from so much legacy vendor baggage over the past few years. Isn’t it time to free them from inane benchmark boasting, too?

Feature image courtesy of Shutterstock user Suzanne Tucker.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Benchmarking the Cloud: Your Mileage May Vary
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

The results, detailed in a blog post on Tuesday, suggest that Google Compute Engine’s real problem right now might just be a lack of high-performance instances. Its current workhorse — an 8-core Intel Sandy Bridge instance with 30GB of memory and 22 compute units — can’t hang with the Amazon Cluster Compute Instances that Zencoder uses for its transcoding workloads. The largest of those is a 16-core dual-CPU Intel Xeon instance providing 60.5GB of memory and 88 compute units running atop a 10 Gigabit Ethernet platform.

As Zencoder ramped up the workloads, the performance differences became clear:

Compute Engine didn’t fare any better when Zencoder tested transfer speeds between the cloud storage platform and the cloud computing platform. Whereas rates between Amazon S3 and Amazon EC2 topped out at 1,458.32 Mbps, the rate between Google Cloud Storage and Google Compute Engine peaked at 202.6 Mbps. In fact, the post’s author writes, “it appears that GCS is slower than S3, and GCE transfer is slower than EC2, such that even if you’re using Google for compute, you may be better off using S3 for storage.”

While the results are interesting because they’re the first real apple-to-apples comparison I’ve seen between Compute Engine and EC2 (BuildFax cloud architect Joe Emison’s pre-release benchmarks were pulled from his Compute Engine review on InformationWeek), they need to taken as what they are. They are, as Zencoder points out, tests of a specific CPU-bound workload — the performance of which Google could improve by adding higher-powered instances — and don’t take into account the difficulties of running at massive scale — a capability Google touted when it launched Compute Engine in June.

And, the author notes, Compute Engine is generally a quality platform, “especially [with regard to] disk I/O, boot times, and consistency, which historically haven’t been EC2′s strong suit.”

This might actually be the more-important measure for most potential Compute Engine users. As GigaOM contributor James Urquhart wrote recently, “If Google can deliver a service that eliminates most of the I/O and network performance inconsistencies that AWS customers currently experience, I can guarantee you there are many major compute customers of AWS that will want to give Compute Engine a test run.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How direct-access solutions can speed up cloud adoption
• Cloud computing 2013: how to navigate without a map

And it wants to make this transition as easy as possible for customers that don’t want to get their hands dirty working with Hadoop or trying to turn a relational database into something it isn’t.

Drawn to Scale’s product, called Spire, is built atop HBase (a distributed NoSQL database that uses the Hadoop Distributed File System), so it scales with ease. Because it uses SQL, users can write robust queries in a familiar manner. Because it uses a distributed index and knows exactly where to go to find the right data, it’s fast as heck. Because it leverages Chef to configure the database, there’s no need to learn the finer points deploying and managing a Hadoop cluster.

“We’re sort of the only game in town when it comes to cloud-scale databases that run on top of Hadoop,” says Drawn to Scale Founder and CEO Bradford Stephens. There are plenty of companies that have lots of data — petabyes of it, in some cases — and want to find new ways to analyze it, but they don’t necessarily want to learn how to deploy Hadoop or write MapReduce workloads.

As of Tuesday, Drawn to Scale has another carrot to offer prospective customers because Spire now ships with MapR’s M3 distribution pre-integrated as the product’s Hadoop platform. Although it’s the bane of open source devotees such as fellow Hadoop vendors Cloudera and Hortonworks because it uses a proprietary file system in place of HDFS, MapR is gaining quite a following because of the performance benefits its technology brings. In fact, Drawn to Scale is just the latest MapR partner after Amazon Web Services recently tagged it for inclusion in its Elastic MapReduce offering and Google did the same for its Compute Engine cloud.

Bradford Stephens

When you’re building a distributed database, you need a fast distributed file system, he added, and MapR is the fastest Hadoop file system around. Because it’s fully API-compliant with HDFS, though, M3 still works just fine with Spire’s HBase underpinnings. “Despite our small size,” Stephens said, “we are more than willing to pick a horse and really advocate loudly for it.”

Although Drawn to Scale is relatively young — it’s still in private beta and just raised a $925,000 first round in March — large companies are already buying into its message. Already, Stephens told me, it’s engaged in seven paid pilot programs with large credit card companies, telcos and other “traditional” types of enterprises. One of them is already trying to negotiate a deal for a 1,000-node production deployment, he added. The company’s skeleton crew of mostly engineers couldn’t keep up the support workload if it opened the floodgates on all the inbound interest, Stephens said.

That kind of demand isn’t surprising when you consider Hadoop’s promise as the platform for a new generation of data-driven platforms. Whether or not anyone actually wants to use MapReduce — the programming framework that made Hadoop famous — they do want to take advantage of its nearly limitless scale on cheap, commodity hardware. When companies such as Drawn to Scale bake Hadoop into an otherwise useful product as a technological component rather than as the product itself, customers should experience the best of both worlds.

Feature image courtesy of Shutterstock user Jakub Pavlinec.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?
• Big data 2013: key trends and companies to watch