GigaOM » Amazon S3

Hadoop startup Qubole raises $7M for Hive as a Service

Derrick Harris — Tue, 23 Apr 2013 12:00:32 +0000

Qubole, the startup from former Facebook engineers Ashish Thusoo and Joydeep Sen Sarma, just closed a Series A investment round for its service, which lets users run a variety Hadoop jobs — including Hive, MapReduce and Pig — in the Amazon Web Services cloud. Hive is the data warehouse system and SQL-like language for Hadoop that Thusoo and Sen Sarma helped create while at the social-networking company. Charles River Ventures and Lightspeed Ventures led the round, which brings the company’s total venture capital investment to $7 million, including its seed round in late 2011.

Qubole launched in June 2012 and opened its platform for public consumption in December, Thusoo told me, and has processed about half a petabyte of customer data since then. Thus far, the platform’s biggest users have been in the advertising technology, e-commerce and application-development spaces. A common use case (and one detailed in a blog post by Qubole customer MediaMath) is to create pipelines that use Hadoop to process unstructured data before pushing it into relational databases such as MySQL, Vertica or Infobright for more-traditional business-intelligence applications.

Ashish Thusoo at Structure: Data 2013, (c) Albert Chau, itsmebert.com

However, Thusoo added, Qubole also has connectors for getting data out of certain other data stores, such as MongoDB, and is working on letting customers import data via API from services such as Omniture and Google analytics.

Being in the cloud — especially Amazon’s cloud — could actually pay big dividends, too, and not just because it lets Qubole scale clusters automatically and lets users avoid the operational headaches of maintaining a Hadoop cluster. Companies are already using Amazon S3 to store a lot of data — more than 2 trillion objects at this point — and that’s Qubole’s choice for a storage system, as well. As companies move more of their big data workloads to the cloud, S3 serves as a cheap, easy and generic storage platform to which they can connect various services and applications.

In January, for example, Netflix detailed its cloud-based Hadoop platform that consists of numerous services but relies on Amazon S3 as the source-of-truth data store.

Netflix’s Hadoop architecture.

If there’s one big question about Qubole, though, it has to be the emergence of a rather-large SQL-on-Hadoop market since the company launched. Although Hive has been an important part of the Hadoop stack over the past few years, its MapReduce foundation is beginning to show its age in terms of query speed, and the new breed of database startups pushing SQL analytics atop Hadoop are quick to point this out.

Thusoo has certainly noticed this activity, but he stills sees Qubole as being in a good position. For starters, he said, the company is looking at interactive analytics projects such as Impala and Shark to see how they might integrate with the Qubole platform, and Hadoop startup Hortonworks is leading the Stinger project to drastically boost the speed of Hive itself.

Further, there’s the fact that Qubole itself has already optimized its platform to run, on average, about five times faster than Hive would normally run on Amazon Elastic MapReduce alone.

“We’re also keeping a close tab on other projects in our space,” Thusoo said. “We have a lot of options … to play with.”

This story was updated at 8:32 a.m. to clarify that Qubole can handle MapReduce and Pig jobs as well as Hive, and that its seed round came in late 2011, not late 2012.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Amazon S3 goes exponential, now stores 2 trillion objects

Derrick Harris — Thu, 18 Apr 2013 14:56:31 +0000

It took six years for Amazon Web Services’ Simple Storage Service, or S3, to grow to storing 1 trillion objects in June 2012. In a blog post on Thursday, however — less than a year later — the company announced the service is now housing more than 2 trillion objects.

I’m not certain what this exponential growth means. It probably means a lot of people are using Amazon Web Services. It probably means they’re building a lot of next-generation applications storing a whole lot of photos and other things (I’m thinking of Pinterest, Instagram, Dropbox, you name it). It probably also means big data is for real in the cloud, as Amazon S3 is the default storage system for AWS’s Elastic MapReduce Hadoop service.

Whatever it means, though, 2 trillion is a lot of anything and probably means Jeff Bezos was right to spend so much time talking AWS in discussing the company’s annual report. At an estimated $2 billion or so in revenue and storing probably a not insignificant amount of data for so many fast-growing web services, AWS looks to be around for the long haul.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Microsoft Azure storage ends the week with a bang — and not in a good way

Barb Darrow — Fri, 22 Feb 2013 23:00:03 +0000

That’s life, as Frank Sinatra once sang. Microsoft Azure Storage was named the world’s best public cloud storage service on Tuesday, then crashes and burns on Friday.

Here are a few of the posts to the Windows Azure status dashboard:

22-Feb-13 · 9:45 PM UTC

Access Control v2, Service Bus, WindowsAzure.com and WebSites services are impacted by Storage service degradation worldwide. We are actively validating the recovery steps to resolve it as soon as possible. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

22-Feb-13 · 8:44 PM UTC

We are experiencing an issue with Storage Worldwide and this is impacting all dependent services. We are actively investigating this issue and working to resolve it as soon as possible. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Folks on Twitter and elsewhere attributed the snafu to the lack of a new SSL certificate. If such a certificate does expire, users cannot authenticate against their various services: No authentication, no access.

Update: As of Saturday morning, this message was posted to the Azure status page — there was no timestamp so it is unclear when it posted. All of the storage areas affected on Friday still showed “service interruption” status.

On Friday, February 22 at 12:44 PM PST, Storage experienced a worldwide outage impacting HTTPS traffic due to an expired SSL certificate. This did not impact HTTP traffic. We have executed repair steps to update SSL certificate on the impacted clusters and have recovered to over 99% availability across all sub-regions. We will continue monitoring the health of the Storage service and SSL traffic for the next 24 hrs. Customers may experience intermittent failures during this period. We apologize for any inconvenience this causes our customers

I’ve asked Microsoft for comment and will update this when they do. Whatever the cause of the problem, it’s been an up-and-down week for Windows Azure. On Tuesday, Nasuni, a company that manages cloud storage for business customers, said Windows Azure storage outperformed all four other cloud services — including Amazon S3 — in rigorous performance testing. Despite Azure’s performance, Nasuni said it would stick to S3 as its primary supplier, citing its maturity. Looks like that may have been the right call.

Windows #Azure Blob Storage service is down due certificate issue. Check update at Windows Azure Service Dashboard:
windowsazure.com/en-us/support/…—
Avkash Chauhan (@avkashchauhan) February 22, 2013

Well, as Sinatra sang: “Riding high in April, shot down in May.” Web time just accelerates the process.

This story was updated February 23 at 6:25 a.m. PDT with a newer statement from the Microsoft Azure status page.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Who’s the best cloud storage provider? Microsoft, says Nasuni; but it still likes Amazon

Barb Darrow — Tue, 19 Feb 2013 23:33:53 +0000

So here’s an interesting tidbit. Nasuni, which manages cloud storage for businesses, ran a set of exhaustive tests to assess the performance, availability and scalability of five major cloud storage providers. And the winner? Microsoft Windows Azure. Yup. Not Amazon S3, but Azure Blob storage.

“We ran uptime tests and other tests and the long and short of it is that all the vendors got better, but Azure just leapfrogged. It was the fastest, had the best availability and uptime and was the only provider to never register an error,” Connor Fee, VP of marketing for Natick, Mass.-based Nasuni said in an interview.

This according to Nasuni’s new State of Cloud Storage 2013 Industry Report, which also evaluated Google, Hewlett-Packard, and Rackspace storage. Nasuni is a pretty good judge of cloud storage provider performance since it assesses the best of the services to use for its customers’ data. It views the various cloud storage players much as EMC or NetApp views hard drives — a piece of its overall service.

Azure’s no.1, but Nasuni still dubs Amazon S3 as its primary backend

Now before we get all wrapped around our axles about this glowing Azure endorsement, it’s important to note that Nasuni still counts on Amazon as its primary storage supplier and will continue to use Azure as a secondary supplier in some cases.

So if Azure is so great, why stick with Amazon? “One major thing we evaluate is maturity and experience in the market and Amazon still clearly has the most experience and is the most mature player in this space,” Fee said.

So how to explain Microsoft’s vast improvement? Fee, refers to this Microsoft blog post, which outlines a major upgrade of Azure’s storage layer, as a possible reason. Basically, Microsoft upgraded its storage layer, from a 1 gig to a 10 gig network and from a hierarchial to a flat network. That means it’s faster handling myriad small files.

Microsoft honed performance on handling lots of itty-bitty files

Think of it this way: Every time you want to store something to the cloud, you have alert the cloud that you’re about to write to it; then you write to it; then it acknowledges receipt of what you’ve written. “There’s a lot of back-and-forth there,” said Fee. “With very big files, if you have a very fast network connection that’s usually enough. But with small files, all of that chatter matters, so whatever Azure did, they got really, really good at handling small I/O,”

It’s also important to note that this year’s report differs from Nasuni’s 2102 testing so year over year comparisons aren’t all that useful, although Nasuni was impressed with Azure even then.

Feature photo courtesy of Flickr user Carlos Gutiérrez G.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Storage for the grand French cloud? Inktank partners with eNovance on Ceph

David Meyer — Wed, 30 Jan 2013 16:37:35 +0000

It’s been noted a few times that the Amazon of Europe will most likely be Amazon, but the French have something else in mind. With capital of €225 million ($305 million) — €75 million of which has come from the French government, no less – Cloudwatt is intended to be the Gallic and ultimately Europe-wide riposte to U.S.-based cloud platforms Amazon, Google and Microsoft. Orange and Thales are also involved, but the company that’s actually designing this OpenStack-based cloud is managed services provider eNovance.

Which is why it’s very interesting to see eNovance ink a deal with Inktank, a startup that’s trying to commercialize the open-source Ceph storage subsystem. Ceph is integrated with OpenStack (and CloudStack, and the Linux kernel), and is pitched at providers that want to use cheaper commodity hardware rather than more expensive proprietary storage hardware.

The deal will mostly see Paris-based eNovance resell Inktank’s professional support and consulting services. There seems to be more to it, though: Inktank sales VP Nigel Thomas told me at Cloud Expo Europe today that the two companies “have a joint customer” already that’s “going to implement an OpenStack solution with Ceph”. He was loath to name Cloudwatt as that customer, but eNovance’s involvement in that project makes the conclusion hard to avoid.

So when will we see Cloudwatt in action, presumably with Ceph powering the storage component? According to the website, it will go live in October.

In other Inktank news, parent company DreamHost just released its own Ceph-based cloud storage service, DreamObjects.

DreamObjects hit general availability on Tuesday following beta testing by thousands of users. It’s compatible with other object storage services including Amazon S3 and OpenStack’s Swift and, for those paying month-to-month, DreamHost is charging 7c/GB for both storage and transfer-out bandwidth. The company says those making longer-term commitments will get cheaper prices, and there are also introductory discounts for those signing up now.

“We believe that the current generation of innovators and content creators will thrive and prosper in an environment where feature-rich, reliable cloud storage can be utilized for pennies per gig,” DreamHost CEO Simon Anderson said in a statement.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

NoSQL startup Basho raises $11.1M and storms Japan

Derrick Harris — Tue, 17 Jul 2012 18:51:30 +0000

Basho Technologies, the company behind the Riak NoSQL database and the Riak CS cloud storage platform, has raised $11.1 million and has entered into a partnership to distribute its technology throughout Japan. Most of the new money, $6.1 million, came from IDC Frontier, a data center provider and Yahoo Japan subsidiary that will use Basho’s technology in its data centers to provide distributed data services to its customers. IDC Frontier’s investment comes on top of a $5 million Series F investment from Georgetown Partners last month.

The IDC Frontier partnership appears to focus on Riak CS, the Amazon S3-like cloud storage platform that Basho released in March. IDC Frontier offers cloud computing services as well as data center space, and Riak CS will provide it with a scalable, multi-tenant object storage system that actually works with the Amazon S3 APIs. That’s good for customers that are either used to S3 already or like the idea of not having to learn new APIs should they decide to leave a Riak CS-based service for S3.

Covering the Riak CS release, my colleague Barb Darrow wrote:

While RiakCS storage is based on the open-source version of the company’s Riak NoSQL database, it includes other proprietary software to handle multi-tenancy and support large object storage. … RiakCS’s ability to run on the cheapest commodity hardware and failover gracefully as needed, means that third party cloud providers really will be able to compete with Amazon on price.

Basho’s flagship software is still the core Riak key-value database, though. It’s that technology — open source, but developed by Basho rather than an open source project — that has attracted most of the company’s big-name users and new CEO Don Rippert, formerly CTO at Accenture. Basho’s total funding is something of a mystery, but the new money does bring its funding in the past year to $23.6 million.

Image courtesy of Shutterstock user brainpencil.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Scale out storage: Caringo says it’s all about the software

Stacey Higginbotham — Mon, 18 Jun 2012 23:08:05 +0000

Since the dawn of the digital age we’ve scattered more and more information about ourselves on the web, inside home computers and laptops and in online storage lockers. It’s the reason we love Facebook and use it as our personal scrapbooks. It’s the opportunity of a lifetime for marketers. And for Caringo it’s the reason it exists.

Caringo, an Austin Texas based company, was formed in 2005 with the idea of creating a new type ~~of file system for storing data~~ of storage software that uses a single namespace to scale to many petabytes of data. Its three founders had worked together at several startups including SequeLink and FileLink before forming Caringo. In 2006 before Amazon launched its Simple Storage Service Caringo released CAStor, software that created a scalable storage service. Amazon’s launch of cloud storage validated Caringo’s plans, and gave it something to sell against.

Caringo’s software is now used by hundreds of customers, who use it internally to create private storage clouds, and by resellers such as Vodafone which use it to offer scalable storage services in the cloud. The software sits on top of a cluster of storage boxes and abstracts out the hardware. It does the work of shifting data around to all the boxes, replicating it in case a disk fails and then restoring the data after a failure.

And as of Tuesday it will offer three new product extensions that allow companies a faster way to index the data stored on a Caringo cluster and queries that data. It also offers a way to dictate specific replication schemes for different types of content stored on boxes running Caringo’s software as well as a management layer that lets people bridge different hardware clusters in different data centers.

The product and the customers it has are a testament to the Caringo’s belief that storage is a hot area thanks to our massively growing digital footprints and a growing obsession with big data. Below I spoke with Jonathan Ring, co-founder and president of Caringo about his views on the future of scale-out storage and big data. His most controversial concept is probably a complete divorce of his software from the underlying hardware and his vision for the future of infrastructure. Check it out. Both are topics we’ll discuss at our Structure 2012 conference this week in San Francisco.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

AWS now stores 1 trillion objects in S3

Derrick Harris — Tue, 12 Jun 2012 13:12:18 +0000

Amazon Web Services’ Simple Storage Service (S3) now hosts more than a trillion — yes, a trillion — objects for its cloud computing customers, proving once again that AWS is the king of the cloud.

To get a concept of how big a trillion is, Amazon’s Jeff Barr in a blog post announcing the new peak calls on the following examples: “That’s 142 objects for every person on Planet Earth or 3.3 objects for every star in our Galaxy. If you could count one object per second it would take you 31,710 years to count them all.” I recently heard TED founder Richard Saul Wurman discuss the national debt by noting that in order to reach a trillion-dollar debt, you’d have to lose $1 million a day every day for about 2,739 years.

So, yeah, Amazon S3 is a wildly successful cloud storage service attached to a wildly successful cloud computing platform overall. As more big data services use S3 as the storage layer — off the top of my head, I can think of several Hadoop services alone that do – it’s just going to keep growing beyond the degree to which web applications alone would force it to scale. Although, S3 does

For a bit of perspective on its rate of growth, consider the following: As of October, it was hosting 566 billion objects, growing to 762 billion in January and 905 billion in April. According to Barr, “Lately, we’ve seen the object count grow by up to 3.5 billion objects in a single day (that’s over 40,000 new objects per second).”

How soon until we hit the quadrillion mark?

Feature image courtesy of Shutterstock user Slavoljub Pantelic.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

AWS Storage Gateway jolts cloud-storage ecosystem

Deni Connor — Tue, 10 Apr 2012 06:55:06 +0000

Amazon Web Services (AWS) Storage Gateway is the company’s first foray into the on-premises cloud-storage space. But it isn’t the only one: A number of vendors are attacking the on-premises cloud-storage gateway market. Do these new offerings, Amazon’s foremost, signal the death of the cloud gateway as an appliance, or are they simply validation of the market?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Basho arms would-be Amazon killers with AWS-compatible storage

Barb Darrow — Tue, 27 Mar 2012 17:07:59 +0000

For cloud providers that want to compete with Amazon’s storage product but don’t want to set up an S3 sandbox for customers, Basho’s latest product can help.

The Cambridge, Mass. startup is now offering RiakCS cloud-based storage services as an option for infrastructure-as-a-service companies and other cloud providers that want to be compatible with Amazon storage services without getting locked into the entire Amazon stack.

Taking on Amazon in cloud storage is a tough row to hoe given Amazon’s dominance. Still, there are many companies that want to move some storage to the cloud but can’t trust their digital stuff to a public cloud, like Amazon’s. To meet this need, an emerging group of cloud players Tier3, Opsource and others are building private cloud services to compete with Amazon Web Services in general and, in some cases, S3 storage in particular.

Building on Amazon to fight Amazon

Basho built its business with a close eye on Amazon. Its NoSQL database, launched in 2009, was based on an AWS white paper outlining the ideal distributed cloud database, according to Basho CMO Bobby Patrick.

RiakCS storage overlays multitenancy atop its Riak database and promises easy storage and access of simple objects. But most importantly, it pledges compatibility with Amazon’s storage API. By virtue of Amazon’s dominance, the AWS APIs are becoming the de facto standard for cloud computing.

Per Basho’s press release:

RiakCS is compatible with the S3 API, providing an easy-to-use interface and allowing enterprises to use existing S3 tools and frameworks or import and extract Amazon data.

Prices for RiakCS start at $10,000 per storage server, including multi-data center replication and support. A 24TB server would thus cost about 40 cents/GB.

While RiakCS storage is based on the open-source version of the company’s Riak NoSQL database, it includes other proprietary software to handle multi-tenancy and support large object storage.

RiakCS, Basho targets the aforementioned big cloud operators as well as big multi-national companies that typically require dedicated support.

Patrick said RiakCS’s ability to run on the cheapest commodity hardware and failover gracefully as needed, means that third party cloud providers really will be able to compete with Amazon on price.

Everyone into the cloud storage pool

The pool of available cloud-based storage options seems to get larger — and more confusing — by the day. Players include Nirvanix, Zettanet as well as Microsoft (with Azure storage), and Google (with Google Cloud Storage.)

Cloud storage is definitely gaining more credence among corporate IT departments wanting to cut costs. The question is how much money there really is in this opportunity as the various vendors appear to be in a race to the bottom, slicing prices on their cloud storage to win share. But it is also clear that there are many companies out there looking for a private, and more secure, cloud storage option. Basho could provide an important piece of that solution.

Feature photo courtesy of Flickr user breahn

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.