17 Comments

Summary:

Startups love the flexibility and pricing of AWS. But then again, no one cloud is right for everyone. Here are a few startups who decided to move at least some of their workloads off AWS.

Image credit: Wikimedia Commons

Amazon Web Services is the default, go-to IT infrastructure of choice for startups, right? Right. Er, well not always. Amazon Web Services

No one disputes that AWS provides an impressive and ever-growing array of compute, storage and networking resources. Update: it just launched AWS Activate, a program to help startups build and run scalable AWS implementations, for example.

Most nascent startups would be nuts to buy infrastructure when they can “rent” AWS especially when they don’t really know what resources they will need. But there are times when it behooves a startup to go elsewhere — to bring workloads in-house or maybe to another provider.

A lot depends on the type of workloads; whether they are stable or spiky; how much IT expertise and gear the company already has in-house; and how smart the customer is about leasing options that can slice hardware costs.

“If you’re constantly launching and destroying instances with no stable demand that fluctuates daily/weekly/monthly (e.g., traffic spikes, batch processing) then it makes sense and is cost effective to use Amazon. Startups are especially like that because they don’t know what they need,”  said David Mytton, CEO of Server Density, a server monitoring and cloud management vendor.

But when the load gets predictable and consistent, “it’ll always be cheaper to switch off the cloud,” he said. That’s a contention AWS would — and does dispute — if you read on.

When is AWS not the best option?

At Structure: Europe last month, Dell cloud exec Nnamdi Orakwue said companies often start looking at alternatives when their monthly AWS bill hits $50,000. Since Dell competes with AWS with its cloud, I took that with a grain of salt, but several startups later said that number actually makes sense.

“$50K a month is actually a good inflection point,” said Dave Anderson, manager of site reliability engineering for Integral Ad Science, a New York-based vendor. “The servers you get on AWS would probably cost you $1,500 to $3,000 to buy and, if you need $50,000 worth of resources, you could probably buy  [all the servers you need] outright for $200,000 or cheaper with a lease agreement.”

Integral Ad Science helps big companies make sure their ads run on the right sorts of sites — ensuring that ads for a children’s entertainment company, for example, don’t show up on porn sites. That work requires very fast interaction. “We sit in the middle — the end user browser hits us before the ad loads and we have to decide in that split second whether that ad should display on that site,” he said.

“Internet performance can be bad enough and AWS is even more inconsistent because of the shared infrastructure,” Anderson said, adding that it is fine for less latency-sensitive applications.

Another problem was that Integral needs to interface with third-party ad servers and that interaction generates a ton of data which must then be pulled into its on-premise analytics systems. Putting that data into AWS and then pumping it out all the time got prohibitively expensive given Amazon’s data egress charges, he said.

“If you can keep everything in AWS including your data processing, it would probably make sense, but with the amount of data processing we do and the amount of time we spend reporting on it, we have data analytics running 24 X 7, so it won’t work for us,” Anderson said.

What’s the cutover point?

Paris Georgallis, VP of platform operations for RMS, which builds catastrophe risk models for insurance companies, also put some credence in the $50,000 per month cut off, although he cautioned that every user’s needs vary. “$50K a month equates to $1.8M of capital spend over 36 months [and] an experienced IT team can work miracles with $1.8M in infrastructure, especially in a mid-to-large enterprise,” he said via email.

AWS: ReinventRMS started out with AWS because well, its developers loved AWS. That appeal is Amazon’s ace in the hole. AWS “is winning the developer war much like Microsoft did in the 90s — by creating simple tools and eliminating infrastructure as a concern for developers that attraction is high,” Georgallis said.

The problem with that is developers tend to treat AWS as an all-you-can-eat-buffet which is fine — to a point. But  poor management of virtual machines and storage means costs can and often do skyrocket.

In another scenario, a consultant said one client company is moving its Elastic Map Reduce (EMR) work off of AWS to save money. The customer, which he would not name, is spending about $80,000 a month on AWS — about 20 percent of which covers the cost of running the EMR cluster on large AWS CC2.8XL machines. The compute session runs briefly overnight, every night, and by moving it to a co-location facility, the company expects to save $250,000 to $300,000 over two years while also getting round-the-clock access to compute capacity as needed.

The allure of on-prem

Joe Emison thinks $50,000 a month  is too high a bar — and that companies spending $10,000 a month on Amazon services should start looking at alternatives — again depending on their workloads. “AWS says that on-prem never makes sense because cloud is always cheaper in the end. I think that’s BS,” said Emison, although he used stronger language.

Emison is CTO at BuildFax, an Asheville, N.C. company that collects and parses property information for the insurance industry. “We have an internal data processing pipeline that only our employees use. We’re fine with RTO/RPO of 4 to 5 hours on these, shooting for 99 percent business-hour uptime,” he said referring to recovery time objective and recovery point object which is, roughly translated, the amount of time a company expects any disrupted resources to be available again.

“If AWS made it really easy to ‘sleep’ servers at night and didn’t charge so much for storage — think 10 to 25 TB that we may need at any time — then maybe it could be cost competitive, but we don’t need AWS-type security or uptime or redundancy for these servers,” he said.

As a result, BuildFax now pays less than 50 percent of what it used to pay using AWS Reserved Instances for those workloads —  and gets more capacity and better performance, he said. It’s important to note that use of Reserved Instances because, typically, when Amazon talks price comparisons, it counsels that companies using RIs — which rent for a period of one or three years — and are cheaper than on-demand instances. The caveat is that if you end up not using those reserved instances, you pay for unused resources. (Amazon now lets people sell off fallow RIs on a marketplace,)

Amazon says AWS now, AWS later

Amazon clearly thinks AWS remains the best infrastructure source for most — if not all — startups. Asked for comment, a spokeswoman said that many fast-growing startups were born on AWS and remain on AWS. Such companies include Dropbox, AirBnB, Instagram, Pinterest, Foursquare, Mailbox, and Dropcam. She wrote:

“One of the reasons these start-ups overwhelmingly stay on AWS as they grow is that they actually save more money on AWS as they scale. In virtually every case we’ve seen, AWS offers significant cost savings – up to 80 percent – over equivalent on-premise options. We have a range of options that let customers optimize costs according to their usage patterns while still retaining the flexibility and scalability benefits of the cloud.”

For more on this topic, check out dueling blog posts from Blippex and from Codeship. One thing’s for sure, when or whether startups should leave AWS will be a continuing debate. Note: This story was updated at 5:38 a.m. PDT with information on AWS Activate program.

  1. excellent summary

    Share
  2. For my company, Server Density, we’ve just hit the monthly spend threshold where looking at running our own hardware in a colocation facilities could make sense. We’re with Softlayer right now and really like their service – pretty much no complaints, except for the long term cost. This is particularly the case because we have a very consistent workload with predictable growth.

    I just published a blog post talking about the experiment we’re running now with colocation in London and buying our own hardware from Dell: https://blog.serverdensity.com/hardware-experiment-london-colocation/

    Share
  3. Greg Knieriemen Thursday, October 10, 2013

    Another good post.

    Can’t believe that AWS still spins that they are cheaper at scale. If you have really inconsistent resource utilization (think web scale e-commerce or Netflix) then you might make the case that you can’t scale on-prem without costly excess resources. But for the other 95% of enterprise tech, AWS is absolutely not cheaper and this has been proven over and over again. Good to see startups are discovering this now too.

    Share
  4. If you’re spending too much on AWS it’s probably because you’re doing it wrong. There’s also a significant long-term Total Cost of Ownership issue for on-prem that’s completely missed here (and likely one that will rear it’s ugly head in a couple years for those going in that direction)

    Share
    1. thanks @Andrew, feel free to expound more on TCO here… i think these vendors i talked to took that into consideration but more data always welcome.

      Share
      1. WIthout human administration factored, your HVAC and power costs will almost always exceed the cost of your hardware fairly quickly. Then you have to add staff to administer and manage the hardware. How long before you have to start replacing that hardware? This replacement expense is one you’ll never incur on AWS. If you are looking at replacing your $50K per month AWS bill with $200K expense for on-prem hardware, you are clearly doing AWS completely wrong.

        There are usually a lot of optimizations that are left on the table when using AWS. For example, effective use of Autoscaling and wringing the most utilization out of your EC2 instances, use of S3 reduces redundancy storage with Glacier archives, spot market for asynchronous tasks, etc. There are use cases where AWS is expensive, but in my experience they are much fewer than those who haven’t properly leveraged the service. Look at what you’ve built and ask if you’re doing it right.

        While edge cases exist where AWS can be more expensive, if you’re not seeing a significant reduction of costs by using AWS then you need to look closely at your architecture and how you’re leveraging AWS before jumping to the conclusion that on-prem is a better option.

        Share
        1. @Andrew: I agree with re-examining your architecture to be sure you’re properly using AWS, but in a CoLo situation, the hosting company spreads those additional TCO costs across all customers, which effectively lower those costs per company. If you’re smart with hardware purchases, you can do quite well by going on-prem.

          Share
          1. @chris and negotiating good lease deals on hardware seems to be the key to success in really cutting costs

            Share
        2. We’ve found that AWS is about twice as expensive as other solutions. We did a very thorough investigation of using a 3 letter storage company product (begins with an “E” and not known for being cheap) compared them to AWS and found that even after factoring in replicating to a different datacenter, floorspace/HVAC (at two datacenters), employee time, acquisition cost, into the costs AWS was still more than 2x. This wasn’t edge, oddball condition it was simply storage of a few hundred terabytes of images for web to be retrieved additionally this product automatically provided replication to multiple datacenters, which Amazon doesn’t provide any method to do across availability zones and must be done programmatically. For our compute it got even worse.

          Share
  5. Robert Engstrom Thursday, October 10, 2013

    For my company AWS became about global reach. We have our own Co-Lo facilities and our own hardware in NY, Switzerland and Germany, but we needed to get into APAC quickly and without setting up a Co-Lo presence (yet). For that, AWS was the right fit.

    That said, we’ve hit what I think is our threshold of $5k/monthly which based on our needs, I think we can move into a Co-Lo, move some hardware from other facilities (so we’re saving on cost there) and reduce our monthly spend (or at least cap it, as we continue to grow this region).

    Essentially, AWS is going to meet the needs of people differently. It’s a great option and it has been incredibly stable for us (and our clients) so its a win-win right now.

    Share
  6. really interesting points here. our data here at Cloudyn show that over 50% of AWS customers are made up of companies spending less than $50k annually. This would seem to support Joe’s comments. Perhaps this pie chart of spend distribution, might add some insight.
    http://www.slideshare.net/Cloudyn/aws-customer-spend-patterns

    Share
  7. We’ve been doing some research with our customers and partners here at Scalr, and we’ve found that the type of workload is really a key issue to consider.

    Of course, some will always be better off on a Private Cloud anyway (e.g. low latency apps, apps that deal with regulated data).

    Bit then, then, there are those workloads that can run on a Private Cloud or a Public Cloud.

    That’s where things seem to get more interesting. This group is composed of two subgroups. There’s the Netflix-style very elastic workloads that will almost always be better off on a private cloud (and that also includes reporting, data crunching…), and there are those other workloads where Cloud is being used only for agility.

    Those non-elastic workloads seem to be those where Private Cloud is the most relevant, and where you can secure cost savings. Of course, we still have to take into account things such as global distribution, failover, and using AWS as a second Cloud / backup can shine in those situations.

    So it looks like Hybrid Cloud’s usually how things end up eventually — what’s you take? Has AWS made itself indispensable?

    Share
  8. What I miss in this article is the big question on how we got to the point that Cloud Computing is factors more expensive than on premise solutions (if workloads are stable and foreseeable). In my perspective this happened by Amazon only passing small parts of their cost savings they derive from the general degradation of hardware and datacenter cost. So their margin grew and grew over time and makes cloud computing much more expensive then it should/cloud be. And the whole industry kind of blindly followed Amazons lead.

    I am happy that now finally people are asking the right questions and are wondering if the perception that Amazons Web Services are cheap is really true. As Barb nicely described: in lots of cases it is not true.

    Cloud is overpriced!

    Sure the flexibility that comes with cloud has to come with a premium against bare metal hardware. But the premiums that are asked for by the majority of the industry are often obscenely high.

    Share
    1. I think the situation is mixed. Some Cloud services are really overpriced and others are really good value.

      In the AWS case for instance, DynamoDB seems at a fair price (at least since their massive price reductions). And AWS Glacier is cheap (any enterprise used to pay the “classical” price for data backups will agree).

      But some other services are *extremely* expensive. AWS/GCE/Azure network egress for instance is probably at least 10* more expensive than it needs to be if we compare prices with the prices we are used to in the hosting world.

      The margins must be huge in this area. Just remember how AWS instantly cut their prices from 0.10$ per Gb to 0.01$ per GB for inter-region traffic as soon as everybody realized that inter-region was key to proper high-availability.

      I think we can expect a lot of competition on the egress price space. Not from Google or Microsoft, but from other providers.

      Share
  9. “Total Cost of Ownership of AWS is lower” is 1996’s “We are Tier-1 backbone” is 1999’s “You don’t peer enough” is the “I’m a sales person that knows a few things but not really” statement.

    AWS is extremely expensive. It is extremely unreliable in a small number of instances and requires an enormous investment in people capable of working around AWS’s weirdness. If you are a software engineering/services company you better know how to create your own virtualization solutions or you won’t scale. Also the network performance of Amazon is absolutely and totally unreliable.

    Alex “I can’t wait to drop AWS costs to a couple of grand from a few hundred thousand a month”

    Share
  10. Dropbox was not born on aws, so that dis-credits anything amazon says. Dropbox started on one PC in their office. They also then only moved storage to S3. Dropbox architecture by their own engineers:

    http://techcrunch.com/2013/07/11/how-did-dropbox-scale-to-175m-users-a-former-engineer-details-the-early-days/

    Share

Comments have been disabled for this post