That Netflix is a heavy Hadoop user is hardly news, though. In June, I explained just how much data Netflix collects about users and some the methods it uses to analyze that data. Hadoop is the storage and processing engine for much of this work.

As blog post author Sriram Krishnan points out, however, Hadoop is more than a platform on which data scientists and business analysts can do their work. Aside from their 500-plus-nod cluster of Elastic MapReduce instances, there’s another equally sized cluster for extract-transform-load (ETL) workloads — essentially, taking data from other sources and making it easy to analyze within Hadoop. Netflix also deploys various “development” clusters as needed, presumably for ad hoc experimental jobs.

And while Netflix’s data-analysis efforts are pretty interesting, the cloud makes its Hadoop architecture pretty interesting, too. For starters, Krishnan explains how using S3 as the storage layer instead of the Hadoop Distributed File System means, among other things, that Netflix can run all of its clusters separately while sharing the same data set. It does, however, use HDFS at some points in the computation process to make up for the inherently slower method of accessing data via S3.

Netflix also built its own PaaS-like layer for Amazon Elastic MapReduce, which it called Genie. This lets engineers submit jobs via a REST API and without having to know the specifics of the underlying infrastructure. This is important because it means Hadoop users can submit jobs to whatever clusters happen to be available at any given time (Krishnan goes into some detail about the resource-management aspects of Genie) and without worrying about the sometimes-transient nature of cloud resources.

We’ve long been pushing the intersection of big data and cloud computing, although the reality is that there aren’t really a lot of commercial options that mix user-friendliness and heavy-duty Hadoop workload management. There’ll no doubt be more offerings in the future — Infochimps and Continuuity are certainly working in this direction, and Amazon is also pushing its big data offerings forward — but, for now, leave it to Netflix to build its own. (And if you’re interested in custom-built Hadoop tools, check out our recent coverage of Facebook’s latest effort.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 9 Companies that Pushed the Infrastructure Discussion in 2010
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Pinterest Operations Engineer Ryan Park had the stage to himself for a session on Wednesday, while Flipboard Chief Architect Greg Scallan and Yelp Engineering Manager Jim Blomo teamed up with Kleiner Perkins Caufield Byers Partner Ray Bradford to form a trifecta of wisdom on Thursday.

Know — and measure — your costs

Flipboard’s Scallan had a paradoxical lesson for the audience when it comes to managing cloud-based infrastructure: Embrace the cloud, but be afraid of the cloud. Yes, it’s flexible and affordable if done right, but all it takes is poor planning or a handful of servers left running ad infinitum, and the costs can begin to grow out of control. That’s why Flipboard assigns members of its engineering team the title of “chief miser,” which means they’re the ones who decide that applications are using the right resources and using them wisely.

Thanks to a variety of practices, including its miserly ways, Scallan said Flipboard is now running about 900 instances at any given time. That’s down from a peak of about 1,500.

Some stats on Flipboard’s AWS usage

One way to help ensure this sort lean operation is to understand your business inputs and outputs, Kleiner Perkins’s Bradford explained. He suggests companies ask, for example, what it costs them to serve a free user on their platform and how does that change with scale or affect the experience they can offer premium users. Pick metrics that really matter, he said (e.g., infrastructure cost per user per month) and then consider how long your current  architecture can sustain that cost before it’s time to retool.

The secret weapon: Source your instances wisely

Pinterest, Yelp and Flipboard all swear by AWS’s pre-paid Reserved Instances in order to save money over the long haul. In fact, Flipboard’s Scallan said, the e-reading startup sees cost savings of about 80 percent over three years by using heavy-duty Reserved Instances instead of on-demand instances for its base workloads, and the break-even point might be only eight or nine months. Pinterest’s Park cited savings of about 70 percent over three years using them.

The trick is queuing another job to take up the waste.

Yelp’s Blomo said his company is a heavy Elastic MapReduce (EMR) user, peaking at more than 350 Elastic MapReduce instances when many developers run their Hadoop jobs simultaneously or when it’s doing nightly analysis of its log files. In order to keep costs in check, Yelp uses Reserved Instances whenever possible to save on hourly bills and has implemented a job-flow pooling system to keep Hadoop jobs running continuously as resources become available. This helps avoid the situation where a job completes in 61 minutes, for example, thus triggering the charge for a full hour of resources even though it only used a minute worth of the second hour.

In order to best gauge when it should use what type instance, Yelp created a tool called EMRio that analyzes past usage to determine what resources are the most-efficient choice for any given job.

The results of EMRio

When it comes to optimizing costs on AWS, though, Pinterest appears to have it all figured out — even how to make use of the somewhat tricky Spot Instances that are priced based on demand and can be terminated without notice if the market price outgrows a user’s bid. Park explained how Pinterest uses the heck out of Reserved Instances and created its own auto-scaling “watchdog” service that decides whether to use Spot Instances or on-demand instances when more resources are required.

Ryan Park dropping knowledge — and graphs

Although Spot Instance prices occasionally spike through the roof, Park’s experience is that they typically remain stable and can result in “massive” savings if you know how to use them effectively. Using Spot Instances to power Pinterest’s approximately 80 front-end servers costs only about $20 per hour, he said. All told, Pinterest has reduced its daily computing bill to about $440 from about $1,200.

All this being said, though, Park, Blomo and Scallan all acknowledged that the flexibility of being able to mix on-demand, reserved and spot servers might not be all it’s cracked up to be if you don’t understand how they all work. Reserved Instances are inflexible in terms of size and region once you reserve them, and Spot Instances must be used wisely for jobs or applications that can handle their easy come, easy go nature. And now there’s even more to consider because Reserved Instances can be resold via AWS’s spot marketplace.

“It gets a little tricky,” Blomo said.

Pick your challenges

Although decisions such database type and structure are largely architectural, there might be elements of cost efficiency at play, as well. Maybe Kleiner Perkins’s Bradford put it best while leading off the session with Scallan and Blomo. Bradford presented a slide containing a simple quote from Instagram Founder Mike Krieger: “Your users around the world don’t care that you wrote your own database.” Sometimes, Bradford added, it might be best to use what works — maybe even a managed service — rather than whatever’s trending highest on Hacker News.

Pinterest’s Park expressed a similar sentiment during his session, citing a lesson his team learned about trying out too many new databases. The site used to use MongoDB, Cassandra, Redis and other databases simultaneously, but learning all the new technologies and managing them became burdensome. Now, he said, Pinterest uses good, old-fashioned MySQL (granted, it sharded MySQL 4,000 times) and memcached — as well as Redis — because they have strong communities and new engineers are more likely to know how to work with them.

After explaining EMRio and some other custom-built Hadoop tools to the crowd, Yelp’s Blomo noted that companies should carefully consider whether the time and money it takes to build stuff will actually result in commensurate savings once those tools or systems are in production. That can require some tough balancing of criteria such as cost, performance, flexibility and user experience.

But it’s important to use human resources wisely. As Bradford said during his presentation, “There’s no free lunch when it comes to developer time.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• How direct-access solutions can speed up cloud adoption
• Understanding and managing the cost of the cloud

Matt Wood

He and I sat down this week at AWS’s inaugural Re: Invent conference and talked about many things, including what he’s seen in the field and where cloud-based big data efforts are headed. Here are the highlights.

The end of contstraint-based thinking

Not so long ago, computer scientists understood many of the concepts that we now call data science, but limited resources meant they were hamstrung in the types of analysis they could attempt to do. “That can be very limiting, very constraining when you’re working with data,” Wood said.

Now, however, data storage and processing resources are relatively inexpensive and abundant — so much so that they’ve actually made the concept of big data possible. Cloud computing has only made those resources cheaper and more abundant. The result, Wood said, is that people working with data are undergoing a shift from that mindset of limiting their data analysis to the resources they have available to one where they think about business needs first.

If they’re able to get past traditional notions of sampling and days-long processing times,  he added, individuals can focus their attention on what they can do because they have so many resources available. He noted how Yelp gave developers relatively free rein early on the use of Elastic MapReduce, saving them from having to formally request resources just “to see if the crazy idea [someone] had over coffee is going to play out.” Yelp was able to spot a shift in mobile traffic volume years ago and get a headstart on its mobile efforts because of that, Wood added.

Data problems aren’t just about scale

Generally speaking, Wood said, solving customers’ data problems isn’t just about figuring out how to store ever greater volumes for every cheaper prices. “You don’t have to be at a petabyte scale in order to get some insight on who’s using your social game,” he said.

In fact, access to limitless storage and processing is a solution to one problem that actually creates another. Companies want to keep all the data they generate, and that creates complexity, Wood explained. As that data piles up in various repositories — perhaps in Amazon’s S3 and DynamoDB services, as well as on some physical machines with a company’s data center — moving it from place to place in order to reuse it becomes a difficult process.

Wood said AWS built its new Data Pipeline Service in order to address this problem. Pipelines can be “arbitrarily complex,” he explained — from running a simple piece of business logic against data to running whole batches through Elastic MapReduce — but the idea is to automate the movement and processing so users don’t have to build these flows themselves and then manually run them.

The cloud isn’t just for storing tweets

People sometimes question the relevance of cloud computing for big data workloads, if only because any data generated on in-house systems has to make its way to the cloud over inherently slow connections. The bigger the dataset, the longer the upload time.

Wood said AWS is trying hard to alleviate these problems. For example, partners such as Aspera and even some open source projects enable customers to move large files at fast speeds over the internet (Wood said he’s seen consistent speeds of 700 megabits per second). This is also why AWS has eliminated data-transfer fees for inbound data, has turned on parallel uploads for large files and created its Direct Connect program with data center operators that provide dedicated connections to AWS facilities.

And if datasets are too large for all those methods, customers can just send AWS their physical disks. “We definitely receive hard drives,” Wood said.

Collaboration is the future

Once data makes its way to the cloud, it opens up entirely new methods of collaboration where researchers or even entire industries can access and work together on shared datasets too big to move around. “This sort of data space is something that’s becoming common in fields where there are very large datasets,” Wood said, citing as an example the 1000 Genomes project dataset that AWS houses.

DNAnexus’s cloud-based architecture

As we’ve covered recently, the genetics space is drooling over the promise of cloud computing. The 1000 Genomes database is only 200TB, Wood explained, but very few project leads could get the budget to store that much data and make it accessible to their peers, much less the computation power required to process it. And even in fields such as pharmaceuticals, Amazon CTO Werner Vogels told me during an earlier interview, companies are using the cloud to collaborate on certain datasets so companies don’t have to spend time and money reinventing the wheel.

No more supercomputers?

Wood seemed very impressed with the work that AWS’s high-performance computing customers have been doing on the platform — work that previously would have been done on supercomputers or other physical systems. Thanks to AWS partner Cycle Computing, he noted, the Morgridge Institute at the University of Wisconsin was able to perform 116 years worth of computing in just one week. In the past, access to that kind of power would have required waiting in line until resources opened up on a supercomputer somewhere.

The collaborative efforts Wood discussed certainly facilitate this type of extreme computation, as does AWS’s continuous efforts to beef up its instances with more and more power. Whatever users might need, from the new 250GB RAM on-demand instances to GPU-powered Cluster Compute Instances, Wood said AWS will try to provide it. Because cost sometimes matters, AWS has opened Cluster Compute Instances and Elastic MapReduce to its spot market for buying capacity on the cheap.

But whatever data-intensive workloads organizations want to run, many will always look to the cloud now. Because cloud computing and big data — Hadoop, especially — have come of age roughly in parallel with each other, Wood hypothesized, they often go hand-in-hand in people’s minds.

Feature image courtesy of Shutterstock user winui.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• It’s time for cloud security and big data to come together
• Buying into big data appliances
• 9 Companies that Pushed the Infrastructure Discussion in 2010

In the cloud, however, the choices are never as simple as black or white.

Amazon versus Rackspace is a matter of control

Discussing its forthcoming Hadoop service during a phone call on Friday, Rackspace CTO John Engates highlighted the fundamental product-level differences between his company and its biggest competitor, AWS. Right now, for users, it’s primarily a question of how much control they want over the systems they’re renting — and Rackspace comes down firmly on the side of maximum control.

John Engates

For Hadoop specifically, Engates said Rackspace’s service will “really put [users] in the driver’s seat in terms of how they’re running it” by giving them granular control over how their systems are configured and how their jobs run (courtesy of the OpenStack APIs, of course). Rackspace is even working on optimizing a portion of its cloud so the Hadoop service will run on servers, storage and networking gear designed specifically for big data workloads. Essentially, Engates added, Rackspace wants to give users the experience of owning a Hadoop cluster without actually owning any of the hardware.

“It’s not MapReduce as a service,” he added, “it’s more Hadoop as a service.”

The company partnered with Yahoo spinoff Hortonworks on this in part because of its expertise and in part because its open source vision for Hadoop aligns closely with Rackspace’s vision around OpenStack. “The guys at Hortonworks are really committed to the real open source flavor of Hadoop,” Engates said.

Rackspace’s forthcoming Hadoop service appears to contrast somewhat with Amazon’s three-year-old and generally well-received Elastic MapReduce service. The latter lets users write their own MapReduce jobs and choose the number and types of servers they want, but doesn’t give users system-level control on par with what Rackspace seems to be planning. For the most part, it comports with AWS’s tried-and-true strategy of giving users some control of their underlying resources, but generally trying to offload as much of the operational burden as possible.

Elastic MapReduce also isn’t open source, but is an Amazon-specific service designed around Amazon’s existing S3 storage system and other AWS features. When AWS did choose to offer a version of Elastic MapReduce running a commercial Hadoop distribution, it chose MapR’s high-performance but partially proprietary flavor of Hadoop.

It doesn’t stop with Hadoop

Rackspace is also considering getting into the NoSQL space, perhaps with hosted versions of the open source Cassandra and MongoDB databases, and here too it likely will take a different tact than AWS. For one, Rackspace still has a dedicated hosting business to tie into, where some customers still run EMC storage area networks and NetApp network-attached storage arrays. That means Rackspace can’t afford to lock users into a custom-built service that doesn’t take their existing infrastructure into account or that favors raw performance over enterprise-class features.

Rackspace needs stuff that’s “open, readily available and not unique to us,” Engates said. Pointing specifically to AWS’s fully managed and internally developed DynamoDB service, he suggested, “I don’t think it’s in the fairway for most customers that are using Amazon today.”

Perhaps, but early DynamoDB success stories such as IMDb, SmugMug and Tapjoy suggest the service isn’t without an audience willing to pay for its promise of a high-performance, low-touch NoSQL data store.

Which is better? Maybe neither

There’s plenty of room for debate over whose approach is better, but the answer for many would-be customers might well be neither. When it comes to hosted Hadoop services, both Rackspace and Amazon have to contend with Microsoft’s newly available HDInsight service on its Windows Azure platform, as well as IBM’s BigInsights service on its SmartCloud platform. Google appears to have something cooking in the Hadoop department, as well. For developers who think all these infrastructure-level services are too much work, higher-level services such as Qubole, Infochimps or Mortar Data might look more appealing.

The NoSQL space is rife with cloud services, too, primarily focused on MongoDB but also including hosted Cassandra and CouchDB-based services.

In order to stand apart from the big data crowd, Engates said Rackspace is going to stick with its company-wide strategy of differentiation through user support. Thanks to its partnership with Hortonworks and the hybrid nature of OpenStack, for example, Rackspace is already helping customers deploy Hadoop in their private cloud environments while its public cloud service is still in the works. “We want to go where the complexity is,” he said, “where the customers value our [support] and expertise.”

Feature image courtesy of Shutterstock user Graeme Shannon.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Amazon’s DynamoDB: rattling the cloud market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Viewed alone, IDC’s forecast offers impressive enough numbers — a $77 million Hadoop market growing at a 60 percent compound annual growth rate until it hits the $812.8 million mark in 2016. The number would be higher, the report concludes, if Hadoop’s open source status didn’t drive down the prices that vendors pushing proprietary products could charge. Indeed, with vendors such as Hortonworks pushing the all-open-source approach to Hadoop, others do have to keep their license fees in check.

According to separate emails from report co-authors Carl Olofson and Dan Vesset, IDC’s revenue calculations take into account software, maintenance and software-as-a-service revenue. That means they account for the usual suspects such as Cloudera, MapR and Hortonworks at the distribution layer (GigaOM Pro subscription req’d), as well as the myriad vendors working a layer above on Hadoop-based applications, databases and management tools. IDC also accounted for cloud-based Hadoop services such as Amazon Web Services’ Elastic MapReduce (and, presumably, upstarts such as Infochimps and Mortar Data).

Time will tell if IDC’s forecast is accurate, but it’s definitely welcome. Hadoop isn’t a new version of the relational database, as some have suggested, but a whole new platform that sits beside and likely won’t replace legacy software. In theory, then, any money spent on Hadoop is new money. And because it began as an open source project and is coming of age in the cloud computing era, it’s not too easy to look to technologies past for guidance.

An impending Hadoop explosion in the cloud

It’s in the cloud where things could get particularly interesting. Olofson noted that he and Vesset estimated  ”little if any revenue for [Elastic MapReduce] in 2011,” although I’m not certain I agree. That service is actually quite popular and accounts for some serious Hadoop use — some users run several thousand nodes at a time. If it’s generating even a few million dollars — just a fraction of AWS’s estimated overall revenue — that’s a not-insignificant piece of a $77 million Hadoop space.

But cloud computing is already having a meaningful impact on Hadoop in other ways that will only expand. One is as the de facto deployment model of choice for many web startups, more and more of which are finding a way to make big data a part of their business model. Whether they’re big data applications or just applications that use big data, they will likely use Hadoop, and they’re likely not going to pay a lot of money for it. If they’re not hosting and managing their own Apache Hadoop cluster, they’ll probably use a cloud-based Hadoop offering, which could mean significant growth in that segment of the ecosystem.

The externalities of Hadoop

Moreover, every company — startup or established — that offers a service powered in some part by Hadoop adds to the platform’s overall economic impact. Hadoop is a big data storage-and-processing framework as well as a positive-externality generator. Facebook, Twitter, Yahoo, Etsy, ipTrust, BloomReach, Skybox, Climate Corporation, Zions Bancorporation — every sale made, fraud thwarted or page view generated thanks to Hadoop means a healthier economy. The dollar amount directly attributable to Hadoop probably isn’t calculable, but it’s likely rather large and growing.

It’s difficult to even fathom the external effects of Hadoop in 2016. A projected $812.8 million market means Hadoop is becoming fairly ubiquitous, especially if one considers how many companies are using free software in addition to those paying for it. Already, I’ve been told, the majority of Fortune 500 companies are at least experimenting with Hadoop. It might not be a godsend, and might be just the starting point of a meaningful big data strategy, but Hadoop is going to have a major impact on how businesses do business.

Image courtesy of Flickr user RachScottHalls.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 2012: The Hadoop infrastructure market booms
• Takeaways from the second quarter in cloud and data
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• Big data 2013: key trends and companies to watch
• Cloud computing 2013: how to navigate without a map

This is a huge increase — about 200x — since 2007, when Etsy signed on with Splunk, and was capturing a mere 5GB of data per day. Etsy’s usage of Splunk has probably evolved, too, from a focus on troubleshooting (i.e., noticing a problem and tracking down the cause) to a focus on what Splunk calls “operational intelligence.” Because users can search and analyze server logs and other machine-generated data pretty much as it streams in, they can, for example, monitor traffic patterns in real time to uncover ongoing issues that might be causing visitors to drop off pages or leave the site.

Splunk isn’t Etsy’s only big data solution — it’s also a big Hadoop user. Etsy runs dozens of Hadoop workflows each night on Amazon’s cloud-based Elastic MapReduce Hadoop service. According to this very detailed (and technical) presentation (PDF here, video here) explaining Etsy’s Hadoop usage, it ran nearly 5,000 Hadoop jobs in May 2011 to analyze both internal operational data as well as external activity such as customer behavior. Etsy actually uses MATLAB within its Elastic MapReduce clusters to analyze the data and perform predictive analytics. The presentation also highlights Etsy’s experimentation with Tableau to visually display the results of its internal data after it has been cleaned up by Hadoop.

At the product level, Hadoop powers Etsy’s Taste Test feature that helps the site determine what products best suit a particular customer’s tastes. It also helps with a feature that analyzes Facebook profile information in order to let visitors shop for their friends. At Hadoop World next week, an Etsy engineer will discuss how Etsy uses Hadoop to improve its search recommendation engine.

Operationally, Hadoop helps Etsy analyze server logs to figure out what customers are doing on the site and how they’re accessing it.

Etsy, like so many other companies — especially on the web — is both drowning in data and trying to leverage it. That’s why we’re seeing such a huge focus on Hadoop among all varieties of enterprise data-management vendors, and why you can’t escape the omnipresent references to “big data.” Although, as Etsy proves, dealing with big data requires a multi-pronged approach that goes well beyond simply deploying a Hadoop cluster and watching the insights pour in.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Creating value out of machine-driven big data
• Big data 2013: key trends and companies to watch
• Takeaways from the second quarter in cloud and data