Blog Post

Cloud is complex—deal with it

Stay on Top of Emerging Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

If you are looking to cloud computing to simplify your IT environment, I’m afraid I have bad news for you.

Yeah, you might find yourself having to worry less about infrastructure, less about how storage systems work or what networking to use to connect a virtualized resource pool, or even what middleware settings are optimal for your applications. However, for every problem eliminated by choosing cloud, you’ll find it just creates more of the problems you remain accountable for—and may even create some new problems that you never had to face before.

Which is as it should be. Let me explain.

When I describe cloud computing as an application-centric operations model, one of the first questions that should come to mind is “operations of what, exactly?” Just because the cloud is focused on the application, it by no means implies that the application is all that is being operated. In fact, just as in any computing technology since the earliest electronic computers, the application can’t exist without myriad things supporting it.

And the world doesn’t consist of a single applications, but, in fact, millions of applications. Most of these are interconnected in some way, and the matrix of code, data, infrastructure, people, policies, requirements and so on that makes up modern IT is ultimately a very interconnected, complex system. Cloud computing is just one (very effective) way of dealing with that complexity.

Cloud as a complex system

What’s interesting is that it turns out science has a whole body of work around complex systems. A complex system, according to Wikipedia, is “a system composed of interconnected parts that as a whole exhibit one or more properties (behavior among the possible properties) not obvious from the properties of the individual parts.”

That’s certainly true of the modern interconnected IT environment. Just look at automated trading systems and the famous “flash crash” for an example—systems designed for increasing market returns reacted to each other in a way that temporarily crashed that very market. Other examples abound, and I’m sure your own IT environment often behaves in ways that no single application or other element was designed to do explicitly.

What science teaches us about complex systems is that they are made up of many individual agents, each of which effect and are affected by agents around them. The feedback loops of events created by agents affecting each other both directly and indirectly, combined with the mechanisms that choose behaviors to in response to those events, combine to create the systemic behavior that is so unpredictable.

Cloud as an adaptive system

The thing is, however, a certain class of complex systems, complex adaptive systems, have the additional trait that they can change their behavior in response to the success or failure of previous behaviors when a given event occurs—or when a certain series of events occurs. This ability to “learn” and adapt to the surrounding system environment creates amazing outcomes, including many of the most rich, enduring and powerful systems in our universe.

Think biology. Think economics. Think ecosystems.

IT is adaptive, in that winning functionality survives and thrives, while losing functionality dies out and disappears. Thus, those investing in building IT technologies are constantly seeking ways for their technology to survive in a changing, often hostile environment.

If an application, or function or even just a line of code fails to add value to the environment—or worse, negatively disrupts the value of the environment—it will be removed or changed, one way or another. Those that rely on IT are constantly seeking ways to optimize applications, data and technologies to take the most advantage of their systems environments.

The result is constant innovation, and constant adjustment to our needs as businesses and individuals. It ain’t always pretty, as they say, but so far it has been quite effective. (I should note that this even applies to infrequently modified “legacy” applications; there is an ongoing decision to not modify such an application, and thus it continues to survive.)

The developer as DNA

I want to leave you with one last thought, however. One of the things about complex adaptive systems is the learning or adapting traits of the agents in the system. In the world of evolution, the main agent of learning or change is DNA. In the world of IT, the agent of learning or change is the engineer or software developer.

If something goes wrong with an application, developers are on the hook to fix it, change it or kill it. If existing hardware fails to create new opportunities to innovate, engineers find new approaches to introduce into the ecosystem to shake things up.

However, developers and engineers can only make those changes one, or a few, components at a time. Nobody can configure the “system” to work an expected way. All you can do is constantly monitor the success and effectiveness of the technologies you deploy into the cloud, and constantly tweak them to make them as useful as they can be in that environment.

It’s up to people to make technologies that survive cloud as a complex system—one component at a time. That’s, well, how you deal with it.

Image courtesy of Flickr user gruntzooki.

13 Responses to “Cloud is complex—deal with it”

  1. Chris Haddad

    James, what laws do you see controlling the complex, adaptive IT system? Are their corollaries with Darwin’s Law, Relativity, Economic Random Walk, or trading patterns (i.e. cup with saucer)? Maybe most IT environments follow the Expanding Universe theory….

  2. Becky Swain

    @jamesurquhart I appreciate how you illustrate the importance of having an acute awareness of the complexities of a cloud ecosystem. I would want to add the role of the architect as another element of the DNA. Without that systems view encompassing of data architecture and flows, information security (and compliance) risks rapidly start to emerge. Therefore, having that architectural perspective becomes essential.

  3. markhahn

    Cloud doesn’t just introduce multi-tenancy, since the design of systems built on the cloud are different from, say, server(s) in your own machineroom. with Cloud, you’ll be doing things like dynamically scaling or nosql that you probably wouldn’t have, pre-Cloud.

    but the main article starts out with a weird hook: Cloud to simplify your IT? that makes no sense: Cloud is outsourcing, which means that you’re simply dependent on different people. most importantly, IaaS does not reduce your need for sysadmins, since a your random VMs in the cloud still needs to be taken care of. (sure, you’ve outsourced the monkey work of racking and replacing, but that was never a big part of your IT overhead.)

    the more you outsource, the more you introduce new agents and complicate your accountability.

  4. JP Morgenthal

    Clearly, the adage “the more moving parts the more likely it is to break” applies to the general theme of your blog. The problem with adaptive systems is still unintended consequences. I’m not totally clear if this was your point about humans being an integral element of adaptive systems DNA, that is the need for humans to qualify the proposed adaptations before they are imposed. If so, then I agree that the iterative loop becomes faster and more effective with the machines determining possible improvements and qualified by humans. After all we don’t want a Skynet situation.

    What I don’t necessarily agree with is that cloud computing increases the overall complexity. Indeed, the only real new complexity that comes to mind is multitenancy. Multitenancy changes the dynamic for the ways applications will behave, but only because those applications were designed with single tenancy in mind. Being mindful of resource consumption shouldn’t make the system more complex, but does require more architecture and design prior to implementation than a single tenancy environment where deficiencies can be accounted for through the application of hardware scaling.

  5. Great article on cloud cloud computing and the possible hurdles that adopters of this model might face in their quest for moving to the cloud .Just viewed an excellent video presentation Adoption Roadmap of cloud computing focusing on adoption strategies @

  6. As someone who’s been in Support, Development, Q/A & now Implementation, this article really hits home. Few people seem to remember these days how connected and important each of the connected systems are.

    Engineers & Dev = DNA
    Support = anti-bodies / defense & repair
    Q/A = trial & error / high speed evolution
    Implementation = birth of the system

    There are more analogies of course, but unless all parts work together, you have a system that’s lacking. Thanks for helping remind us!