11 Comments

Summary:

OpenStack’s promise of an open-source cloud infrastructure free of vendor lock-in is big. But difficulties upgrading from one release to the next are a major kink that needs to be worked out before widespread adoption can begin.

Escher Relativity in Legos
photo: idigit_teddy

Technology updates are never a day at the beach, but customers trying to upgrade to the latest release of OpenStack from its predecessor are often encountering bigger-than-expected problems. That is definitely not good news for OpenStack as it tries to gain traction among big,  reference-type accounts at a time that Amazon Web Services is ramping up its enterprise push and VMware and Microsoft are tuning up their own enterprise-focused cloud efforts.

Even some in the OpenStack community admit that upgrades and migrations are more difficult than they should be, especially if the customer is not using a packaged version of OpenStack from one of the major vendors. Of course those vendors all talk up their own OpenStack implementation as the key to success. For this story I talked to several executives in the OpenStack ecosystem.

One major issue is that the Grizzly-to-Havana upgrade often requires that users shut down their applications and restart them when it’s over. One reason is that the major OpenStack modules — Nova compute, Cinder block storage, Quantum networking — have their own databases. “As you build new features, the database schema gets changed and that makes it extremely hard to upgrade in a rolling fashion. You have to wipe out your system and reinstall,”  said one executive with an OpenStack vendor. To be clear, the issue of rolling upgrades isn’t even solved in mainline versions of the Linux kernel.

scaffoldingStill, “wipe out and re-install?” That noise you heard is the scream from IT departments across the land.

One company that would not be named, also had to rejigger those applications that had run on Grizzly post-migrationto get them to run on Havana at all.

Easing migrations and management are major selling points behind  “enterprise grade” releases of OpenStack like Red Hat’s new Enterprise Linux OpenStack Platform 4.0. Tools like HP’s TripleO and Mirantis Fuel are also geared to ease migrations and deployments.

Upgrade problem is “unresolved”

But as things stand now, “the upgrade problem is real and unresolved,” said the OpenStack vendor  executive.

One issue is that OpenStack, at just over 3 years old, is still very young but is rushing to add features and functions to compete with AWS. “All of that upstream development to add features is part of a mentality that isn’t focused on basic things that enterprises need — like backwards compatibility,” he added.

Remember: Cloud is hard

Cloud computing is complicated and a technology built and deployed by multiple, often-competing vendors, gets more complicated. Things take longer to get set in stone and adopted. While the promise of open-source cloud infrastructure is that customers needn’t worry about vendor lock-in (to AWS, to VMware etc.) sometimes having one vendor set the requirements, makes deployment easier. That’s the whole promise behind Oracle’s “cloud in a box” strategy and one reason there have been calls among some cloud pundits for a “benevolent dictator” to take over OpenStack strategy.

To be fair, others acknowledge difficulties in Openstack deployments but that they can be mitigated by choosing deployment partners wisely. Jonathan Bryce, executive director of the OpenStack foundation said via email:

“OpenStack can be used and deployed in many different ways. Commercial and community support is available through a variety of distributions, products and services to meet different application and user requirements. With that choice and breadth of use cases, upgrade experiences can vary, but we have consistently heard that Havana has been the smoothest upgrade yet.”

Dreamhost did the Grizzly-to-Havana migration in a week, said Jonathan LaCour, VP of product and development for the Los Angeles-based web hosting company.

“We use continuous integration and deployment so it was a matter of rolling it out to smaller test and staging clusters first. We had some data migration issues that could be improved upon. If we’d used Ubuntu or Red Hat that would have been handled but we have a lot of specific stuff we need to handle so we went our own way,” Lacour said in an interview.

He recommends that less tech-savvy businesses go for a packaged release of OpenStack as opposed to rolling their own. “There is no shortage of fully supported OpenStack packages that make it easier on the enterprise,” LaCour said in an interview.

Randy Bias, co-founder and CTO of Cloudscaling, another OpenStack purveyer agreed. “Upgrades with OpenStack will always be hard. People have to use an actual product to get smooth upgrades — OpenStack is a big distributed software system with lots of interdependencies. Rolling your own OpenStack deployment using DIY is like making your own Linux distribution,” he said.

Probably so. But another software exec at another of the major OpenStack backers — an IT giant with tons of resources — admitted (privately for obvious reasons) that his own company was challenged by its Havana upgrade.

Too many cooks?

Why is this happening even to very tech-savvy companies? He blamed the proliferation of separate projects — like Ceilometer for billing systems, Marconi for queuing and notification, etc. — which he said are not aligned back to OpenStack core projects like Nova and Glance. Those multiplying projects means, in his opinion, not enough follow-through on promises to fix key issues of scale, manageability, performance and stability.

He also cited not-so-far-beheath the surface infighting between the various OpenStack groups and backers. “That looks horrible to the customers that are trying to run businesses on the platform that could care less about the soap opera drama,” he said via email.

Ouch. A cloud pundit who is watching developments carefully but also asked for anonymity because he advises a couple of the OpenStack players, said many of the current OpenStack cadre — HP, Cisco, etc.– have mixed agendas and aren’t all that hot on selling OpenStack just yet.

“Sure they talk up OpenStack but in reality they’re all trying to sell their own converged data center hardware where they can make money. In two or three years they’ll get their acts togehter and start selling OpenStakc and making it more attractive to run atop all those data center goodies they’ve sold,” he said.

For more on why OpenStack is a tough enterprise sale, check out Gartner analyst Alessandro Perilli’s take.

Meanwhile, there remain just a handful of really big OpenStack implemenations — CERN being the poster child. Meanwhile, rumors of big new customers — Salesforce.com, for example – surface not through the alleged customer itself but via OpenStack proponent — Rackspace. It smacks of desperation.

So we’re back to the worry that’s nagged OpenStack for more than a year. At the OpenStack Summit this fall, the foundation touted new users including Workday, Concur, Shutterstock and Fidelity Investments. But there is still concern about whether there are enough big companies deploying OpenStack at scale to push it into the mainstream. It sounds like that won’t happen until upgrades and migrations are proven to be much, much easier than they’ve been to date.

In other words, give it a few more years. God knows what Amazon Web Services can do in that time.

Feature photo courtesy of Flickr user idigit_teddy

  1. Upgrades are a difficult thing, especially for cloud platforms because they touch everything in a datacenter.

    Eucalyptus has warm upgrades:

    http://www.eucalyptus.com/eucalyptus-cloud/iaas/whats-new

    This means that you can upgrade Eucalyptus without taking down the workloads running on the cloud.

    Nokia with their big Euca cloud have 4 maintenance windows per year. Previously they had to wait for such a window to be able to upgrade Eucalyptus. Now, with Euca 3.4, they did the upgrade in 30 minutes without interruption to the operation of the cloud.

    The Nokia cloud has been described here: http://www.enterprisetech.com/2013/11/11/nokia-dev-cloud-will-swell-100k-cores/

    Marten Mickos

    Share
  2. Canonical demonstrated a live, zero downtime OpenStack update (not upgrade) as early as last april – see http://www.openstack.org/summit/portland-2013/session-videos/presentation/canonical-keynote-openstack-in-production

    This include kernel upgrades and Cloud reboots, deployed with no downtime to user workloads in guests.

    Share
  3. It may be true that upgrade is an issue for OpenStack, but I don’t think it is fair to single OpenStack out. Upgrades remain a problem when it comes to pretty much EVERY piece of software today.

    Linux has been around for over a decade and maybe with the exception of Oracle Linux, in-place upgrades of Linux kernel is still a problem today. Also, last I rememer, when I updated iOS I had to re-install all my apps…

    So, yes, it is not surprising that something as complex as OpenStack does not allow you to push one button and roll from Grizzly to Havanna. But, I am sorry to say, NOTHING does that.

    Is it an opportunity for the OpenStack community to Rally around to beter user experience? YES. Is it something that will preclude OpenStack from becoming the standard cloud operating system – DEFINITELY NOT.

    Share
    1. Agree, upgrades to Open Stack will always be hard. That is a key advantage AWS has. Customers don’t have to worry about upgrades. All handled under the covers by AWS. Easy breezy beautiful.

      Share
    2. +1, Boris.

      Share
  4. OpenStack is too young to manage enterprise grade upgrades, but Red Hat, Canonical, IBM and other commercial supported OpenStack distributions will provide the desired warm upgrade support in the future. They’ve to learn from their grand father “Euca” how to do that :-)

    Share
  5. OpenStack Swift, the Object Storage project, has always had a very smooth upgrade path. This is available both in the pure open source project and with commercial products.

    With OpenStack Swift (blog): http://swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/

    With an OpenStack Swift Product, SwiftStack (video): http://www.youtube.com/watch?v=xYfHES0zka8

    Share
  6. How to Upgrade an OpenStack Swift Cluster With No Downtime.

    http://swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/
    @gigabarb Not sure if you tried playing with it as its not mentioned in your blog.

    cheers,

    @koolhead17

    Share
  7. Not an issue with Metacloud: “OpenStack as a Service”. OpenStack features + enterprise extensions and zero upgrade headaches.

    Share
  8. Ignacio M. Llorente Saturday, December 21, 2013

    OpenNebula provides automatic live update (minor version) and upgrade (major version) for production systems since version 3.0 (October 2011) and this is part of the fully open-source distribution.

    http://docs.opennebula.org/stable/design_and_installation/building_your_cloud/upgrade.html

    There are very-large production deployments that have reported a complete upgrade of their infrastructure to OpenNebula 4.4 in only 30 minutes.

    Ignacio M. Llorente

    Share
  9. Technology and/or open source aside, choice of partner is still a determinant of OpenStack migration success. Decision involves more than the technology itself.

    Share

Comments have been disabled for this post