Getting started

Before you do anything else, if you haven’t already read The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, by Gene Kim, Kevin Behr and George Spafford. If you don’t absolutely identify with the pain felt by the characters at the beginning of the book, or with the wisdom of the approach introduced by the end, then this concept probably won’t click with you. However, if you’ve spent any time involved in enterprise IT at all, I’m betting this book will hit home, both intellectually and emotionally.

After that, the previous posts in this series provide some good background, as well:

• Devops, complexity and anti-fragility in IT: An introduction
• Devops, complexity and anti-fragility in IT: Risk and anti-fragility
• Devops, complexity and anti-fragility in IT: Stability and resilience
• Is your PaaS composable or contextual? (Hint: the answer matters)

Complexity and anti-fragility

Although I don’t love everything about Nasem Taleb’s Anti-Fragile: Things that Gain from Disorder, it is undeniably one of the most important books I’ve read in a while. The reason for this is that it articulates a key concept that is often missed by those of us that seek resiliency in systems: that there is a class of systems that show a behavior that actually gains from randomness. In other words, they tend to move toward a “better” state over the course of both positive and negative variation in their environments. The post on risk and anti-fragility that I link to above covers this concept in more depth, but the book explores the concept in many different contexts.

The best book on complex systems that I’ve read to date remains Complexity: The Emerging Science at the Edge of Order and Chaos, by M. Mitchell Waldrop. The telling for the story behind the founding of the Santa Fe Institute, still considered the hub of complex systems science, Waldrop’s book covers much of both the concepts and the methods of exploring complex systems (and its critical subset, complex adaptive systems). It is a little out of date now, however.

If you prefer to learn by doing, Margaret Mitchell’s ”Introduction to Complexity” course through the Santa Fe Institute is an excellent 101- level course on the subject, though tilted heavily toward the academic study of the subject,. The only focus on practical applications comes via interviews with famous complex systems scientists.

Devops and continuous integration

For devops, in particular, there are a lot of great sources available online, as well:

• A decent overview (with a decent list of both cultural and technical elements of devops) is “Devops: An Introduction,” a slide presentation from Patrick Dubois.
• Anything written by John Willis and Gene Kim at ITRevolution, John Allspaw and his crew at Etsy, the team at Netflix, and a host of others are worth reading as well.
• My favorite source for devops learning, however, is the DevOps Weekly newsletter, a very well-curated list of reading material each weekend. Definitely a must if you want to understand devops in depth and in real time.

I hope everyone has gained something from these posts. I certainly believe this shift in focus — from risk avoidance to anti-fragility, from a focus on stability to a focus on resilience, and from a focus on large-grained contextual systems to small-grained composable alternatives — will and is opening a whole new world of agility, experimentation and execution for enterprise IT. It’s a critical subject for every IT practitioner to understand.

This is, of course, only a partial list of the many amazing books, web sites, blogs and events that I’ve used to explore this topic. I encourage you to add your favorites to the comments below, or share them with me on Twitter, where I am @jamesurquhart.

James Urquhart is vice president of products at enStratius and a regular GigaOM contributor.

Feature image courtesy of Shutterstock user Linda Parton.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Continuous delivery and the world of devops
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: Cloud and big data woo enterprises

These two patterns are described well in this recent post from Neal Ford, self-described “Director, Software Architect, and Meme Wrangler” at systems integrator ThoughtWorks:

In my keynote, I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive…The knowledge and effort required for a seemingly trivial change prevents the change from occurring, leaving the developer with a perpetually dull tool. Contextual tools aren’t bad things at all – Eclipse and IntelliJ wouldn’t exist without that approach. Contextual tools provide a huge amount of infrastructure that developers don’t have to build. Once mastered, the intricacies of Eclipse’s API provide access to enormous encapsulated power…and there’s the rub: how encapsulated?

In the late 1990’s, 4GLs were all the rage, and they exemplified the contextual approach. The built the context into the language itself: dBASE, FoxPro, Clipper, Paradox, PowerBuilder, Microsoft Access, and similar ilk all had database-inspired facilities directly in the language and tooling. Ultimately, 4GLs fell from grace because of Dietzler’s Law, which I defined in my book Productive Programmer, based on experiences by my colleague Terry Dietzler, who ran the Access projects for my employer at the time:

---

Dietzler’s Law for Access

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.

---

Ultimately Dietzler’s Law killed the market for 4GLs. While they made it easy to build simple things fast, they didn’t scale to meet the demands of the real world. We all returned to general purpose languages.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things. A famous story from 1992 illustrates just how powerful these abstractions are. Donald Knuth was asked to write a program to solve this text handling problem: read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. He wrote a program consisting of more than ten pages of Pascal, designing (and documenting) a new algorithm along the way. Then, Doug McIlroy demonstrated a shell script that would easily fit within a Twitter post that solved the problem more simply, elegantly, and understandably (if you understand shell commands):

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q

I suspect that even the designers of Unix shells are often surprised at the inventive uses developers have wrought with their simple but powerfully composable abstractions.

Ford goes on to describe the pros and cons of each approach in much more detail, but the key conclusion he reaches is, I think, critical to understanding how one should develop the tools and tool chains that drive new IT models:

These abstractions apply to tools and frameworks as well, particularly tools that must scale in their power and sophistication along with projects, like build tools. By hard-won lesson,composable build tools scale (in time, complexity, and usefulness) better than contextual ones. Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible, Dietzler’s Law Redux. This is especially true in tools where critical parts of how they function, like the ordering of tasks, is inaccessible without hacking.

Ford’s distinction is one that finally helps me articulate a key concern I’ve had with respect to Platform-as-a-Service tools for some time now. In my mind, there are primarily two classes of PaaS systems on the market today (now articulated in Ford’s terms). One class is contextual PaaS systems, in which a coding framework is provided, and code built to that framework will gain all of the benefits of the PaaS with little or no special configuration or custom automation. The other is composable PaaS, in which the majority of benefits of the PaaS are delivered as components (including operational automation) that can be assembled as needed to support different applications.

Contextual PaaS

Examples of contextual PaaS include the original releases of Google App Engine, Heroku and other “first-generation” PaaS systems that asked the developer to adhere to specific architecture and consume PaaS-specific classes in the application itself. These systems were incredibly powerful for building applications that were variations of what these frameworks were designed to do, but began to fail quickly for applications that fell outside of that domain.

The classic example is Google App Engine’s limit of 30 seconds for any backend request to complete. Great if you were building a Facebook game, but a requirement that eliminated its use for many multi-step transactional applications. Of course, there were ways to deal with those situations, as well, but they were mostly complicated and added risk to the system.

There is a parallel here with the 4GLs of the late 1990s that Ford talks about in his post. At that time, I worked for Forte Software (acquired by Sun Microsystems in 1999), which built a 4GL development and operations environment for distributed application development. We had a business model where we relied heavily on systems integrator partners to help our customers deliver these often sophisticated applications, and every one of those SIs eventually built a framework environment to make building complex applications “easier.”

The problem? Almost every customer that used one of these frameworks had a requirement (or many) that the framework didn’t handle well. This resulted in either the SIs scrambling to modify their frameworks to support these requirements — inevitably resulting in the framework being much less “easy” to use — or the customer bypassing the framework all together for those needs, resulting in an application that was harder to debug and operate.

Composable PaaS

Composable PaaS systems, on the other had, do much less to anticipate the architecture or functionality of the application built on it, and do much more to simplify the assembly of services, including underlying infrastructure, automation, data sources, specialized data tools, etc. I think the classic example of a composable PaaS is Cloud Foundry, the open source PaaS effort from VMware that’s now part of its Pivotal Initiative spinoff. Modern versions of Heroku, Engine Yard, CloudBees and other also exhibit more of this approach than “first-generation” PaaS systems.

An old, but illustrative, Cloud Foundry diagram.

Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).

Now, take the concept a step further. Imagine a deployment environment that delivers a wide variety of these individual tools and components and simplifies the process of creating tool chains on demand from them. Imagine that environment would let each development team choose from known tool chain “patterns,” but modify them as they see fit for each project. This, I believe, will be the ultimate general purpose PaaS success, not some hard-and-fast framework-based PaaS.

The concept of composable and contextual applies to a lot more than PaaS and cloud, of course. And it is important to note that it’s not an either/or choice, much like stability and resiliency. Parts of an IT environment should be composable, but there will always be elements where the relative stability of contextual extension makes more sense. And composable systems can leverage API-driven systems that themselves are designed primarily for extensibility via contextual approaches.

The key is to think about each system from the perspective of how it will be used, and to target its extensibility mechanism based on needs. Just remember, however, that choosing a contextual path will dictate a lot more about how your system could be used in the future than a composable approach would.

I’d love to hear your thoughts, either in the comments below, or on Twitter, where I am @jamesurquhart.

Feature image courtesy of Shutterstock user Nenov Brothers Photography.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• PaaS market accelerators, 2012–2013
• Cloud computing infrastructure: 2012 and beyond
• Platform as a Service in 2012

Phil is a friend and he has an extensive background in systems and data center administration. His history is one of surviving the IT operations battles of the last decade or so. He cringes every time I mention concepts like continuous deployment, devops and — especially — the stability-resiliency tradeoff.

What is the stability-resiliency tradeoff?

The best description of the stability-resiliency tradeoff I can find is from C.S. Holling in a 1996 paper entitled “Command and Control and the Pathology of Natural Resource Management”:

“We call the result ‘the pathology of natural resource management’ (Holling 1986; Holling 1995), a simple but far-reaching observation defined here as follows: when the range of natural variation in a system is reduced, the system loses resilience. That is, a system in which natural levels of variation have been reduced through command-and-control activities will be less resilient than an unaltered system when subsequently faced with external perturbations, either of a natural (storms, fires, floods) or human-induced (social or institutional) origin. We believe this principle applies beyond ecosystems and is particularly relevant at the intersection of ecological, social, and economic systems.”

In other words, the more you try to control — or stabilize — a system by reducing variance in the system, the less resilient you make the system to unexpected events. Command-and-control, top-down approaches to complex systems operations actually make those systems more fragile in those cases.

Now, to be sure, for this principle to apply you must be talking about a complex system made up of many independent agents. Distributed applications are good examples of this. In fact, the more instances make up a distributed application, the more the principle applies.

And, of course, the entire portfolio of business and operational applications in a typical enterprise tends to create a higher-level complex system: Every agent (e.g., application, service or device) is connected to every other agent by just a few degrees of separation. (Just sharing a data center provides one common point of dependency for disparate software applications, for instance.)

Is there really a tradeoff?

Now, back to Phil. For him, stability is specific concept, and he thinks many have lost their way trying to repudiate it:

[It's not an] AND versus OR argument. There’s a lot of folks who have gone completely overboard with this idea that if you don’t do continuous deployment, you’re doing it wrong. And the simple fact of the matter is that they’re wrong. IT is not a zero sum game, nor is it strictly OR operations. Most organizations don’t want or need continuous deployment. And many organizations (e.g. Google who likes to break their infrastructure at the expense of paying customers and products) are doing it completely wrong.

Phil goes on to explore the definitions of “stable” and “resilient,” and to argue that what you want is stability and resiliency, as they are critical to running a business. He argues that these new models are detrimental to achieving both.

Read the post, and think about what he says.

Stability and resiliency in different contexts

If you have read the other posts in the series, especially the last one, you know that I disagree with Phil on his stances that “most organizations don’t want or need continuous deployment.” But as I read Phil’s post a second time, I began to see a distinction that I hadn’t considered before.

The stability being described in Holling’s paper is command-and-control approaches to squeezing variability out of a system. We see this in traditional approaches to architecture, ship building and other disciplines, where the belief that absolute control over all design parameters of a complex structure is both possible and mandatory for safety.

The stability that Phil introduces is also about “efforts to reduce the number of shocks the process or system is subjected to,” but he’s looking at it from the perspective of ending up with something that works after a failure. Which is also what resilience is all about, right?

So why would Phil argue so adamantly that stability and resiliency are interrelated? Perhaps because, from a hardware perspective, it’s obvious they are. At some level, stability is required for a CPU or — at the very least — a transistor. Higher-order systems may build resiliency around many such stable (e.g., unchanging) components, but some level of stability is desirable at some level within every complex system.

It’s just that trying to arbitrarily limit variance in the complex system is detrimental to the system. The system becomes less resilient as a result.

How should IT consider stability and resiliency?

Ultimately, perhaps the design decision is as simple as:

• If the system being designed is best operated as a complex adaptive system, with highly independent agents and a dynamic structure, then the focus should be on resiliency and variance. The various teams that own the agents, the relationships between agents and the goals the business has for that system should drive this.
• If the system is instead fairly static and/or made of few agents, choose a more prescriptive, static design approach. Limit what agents can participate in the system, and hardwire the relationships between agents so change is less dynamic (or not dynamic at all).
• Since complex adaptive systems can be made of static components, the IT organization should be prepared for the full range of approaches needed to operate everything in the system.

(Keep in mind that the IT portfolio, if large enough, is itself a complex system, so resiliency and variance should dominate at that scale.)

While I disagree with Phil about the validity of continuous integration and continuous delivery approaches for enterprise software projects, I do agree that IT itself doesn’t have a stability/resiliency tradeoff. The tradeoff exists solely for each system structure, and not necessarily for the agents of those systems themselves.

My next post will turn its attention to practical matters, exploring sources of information to explore these topics further, and suggesting a few steps that enterprises can consider today with respect to devops adoption, resiliency and anti-fragility.

Thoughts? Comment below or find me on Twitter at @jamesurquhart.

James Urquhart is vice president of products at enStratus and a regular GigaOM contributor.

Feature image courtesy of Shutterstock user Olivier Le Moal.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Continuous delivery and the world of devops
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

Enterprise IT organizations have spent decades trying to create systematic approaches to control and (hopefully) eliminate disruption in computing operations. The standard approach to date has been to strictly control change. Now, concepts like continuous integration and deployment, modularized application systems, and “fail fast” agile processes encourage continuous change.

Embracing anti-fragility

So why would anyone want to promote an approach that encourages constant change, when failure in the form of outages or breaches or large-scale processing errors exact such a heavy toll on businesses? The short answer is because some application domains require it, but that’s also a bit glib. Instead, let me bring in the concept of “anti-fragility,” as coined by Nassim Nicholas Taleb in his book “Anti-fragile: Things That Gain From Disorder.”

I explained the gist last week:

“Anti-fragility is the opposite of fragility: as Taleb notes, where a fragile package would be stamped with ‘do not mishandle,’ an anti-fragile package would be stamped ‘please mishandle.’ Anti-fragile things get better with each (non-fatal) failure.”

Anti-fragile systems benefit from variability and can take advantage of differences from the “normal” to ultimately gain value. Anti-fragile systems behave in such a way that failures due to change exact a small cost, but successful change drives exponentially higher value, so the system gains overall. Taleb argues this is only achieved by keeping the scope of each activity small enough that the downside risk is manageable (and results in strengthening the system), and that any gains can be maintained ongoing.

Jez Humble, co-author of the book “Continuous Delivery,” did a very good job of analyzing anti-fragility from the perspective of software delivery:

Taleb shows why the traditional approach of operations – making change hard, since change is risky — is flawed: ‘the problem with artificially suppressed volatility is not just that the system tends to become extremely fragile; it is that, at the same time, it exhibits no visible risks. . . . These artificially constrained systems become prone to Black Swans. Such environments eventually experience massive blowups. . . . catching everyone off guard and undoing years of stability or, in almost all cases, ending up far worse than they were in their initial volatile state’ . . .

This is a great explanation of how many attempts to manage risk actually result in risk management theatre — giving the appearance of effective risk management while actually making the system (and the organization) extremely fragile to unexpected events. It also explains why continuous delivery works. The most important heuristic we describe in the book is ‘if it hurts, do it more often, and bring the pain forward.’ The effect of following this principle is to exert a constant stress on your delivery and deployment process to reduce its fragility so that releasing becomes a boring, low-risk activity.

Today’s IT models don’t demonstrate that behavior, at least at the project level. As Humble noted, most IT projects are highly fragile — a few relatively small errors during development or operations can send the entire project crashing down at an inopportune time. IT projects (and individual project releases, for that matter) tend to:

• have giant scopes of hundreds or thousands of requirements.
• be managed through a series of organizational siloes with weak feedback loops between the silos.
• introduce new operations vulnerabilities with each release, due to dependence upon manual process steps, and highly context-specific, fragile “scripting.”

Change, therefore, is artificially suppressed, or at least intensely controlled. This just makes projects more fragile in the long term, especially from the perspective of meeting constantly changing business needs.

Approaching anti-fragility through devops

It doesn’t have to be this way.

One solution to that problem is highlighted today in the form of devops or “noops”-driven software organizations like Netflix  and Etsy. The software approach these organizations take is one of releasing small changes as often as possible, with heavy reliance on automation, and — this is very important — measuring the resulting effect on dynamics important to the business stakeholders.

Oh, and they can quickly reverse or replace stuff that doesn’t work out as expected. Which happens fairly often. Which leaves them no worse off then they were before they tried the change. See the anti-fragility yet?

However, in order to get to this state of low-risk, constant experimentation, these organizations have had to employ skills, tools, processes and practices that are significantly different than the change-management techniques of the past. The most obvious qualities of their devops systems are:

• Automation enforces certain practices, such as running various tests with every build or build environment (e.g., running regression tests before moving from dev to staging).
• Culture enforces practices, such as Etsy’s practice of allowing developers to own their mistakes without fear of reprisal, which encourages tribal knowledge of how to avoid such mistakes in the future. Culture also dicates that dev, ops, security, business and other stakeholders all work together over the entirety of the application lifecycle.
• Prudent measurement of all elements of the processes, tools and applications provides the key feedback necessary to continually strive for improvement.

Devops and anti-fragility are by no means synonymous, however. Devops can be implemented in such a way that it doesn’t exhibit the trait of being anti-fragile —  like when high developer turnover results from always putting the “best people” on the “next great thing,” and knowledge or culture is lost as a result.

Anti-fragility can also be acheived without devops, though I’m not aware of another consistent methodology for doing so. Nonetheless, anti-fragility is a trait to be strived for, not a methodology itself, so there are options for achieving that trait.

No silver bullet

And, lest you think we’ve hit upon yet another “drop everything and change the way you do things” approach to enterprise IT, I would caution against applying anti-fragility religion where the investment wouldn’t pay off.

Given the difference between devops and most “construction-method” approaches to IT that we see today, for example, I would argue that enterprises should adopt devops and address anti-fragility first by using it for those IT projects that would benefit from continuous change. Ones like marketing applications, business process automation and so on. Less critical would be systems like core ERP databases and infrastructure that don’t often need to undergo change.

You can’t crowbar a change-averse technology into a change-driven methodology. However, over time you might be able to adopt a few of the benefits to lessen risk when change is necessary in those systems.

Here’s a heuristic I’m experimenting with: the more that differentiation and adaptation are important to the solution at hand, the more that anti-fragility should be strived for. Undifferentiated activities (such as running data center facilities or core SAP packages) should strive for resiliency, but perhaps adapt automation, etc., over time as part of a more traditional approach to software project control.

This heuristic is backed up, somewhat, by the work of my friend Simon Wardley, who has one of the most-comprehensive theories of the evolution of enterprise activities from inovation through commoditization. Activities at different stages of that spectrum benefit from different practices, and IT is no exception to that rule.

In my next post, I’ll go into more detail about this spectrum of practices, define the “stability-resiliency” tradeoff and explain how enterprise IT can navigate it. In the meantime, this is an opportunity for you to express your thoughts on the subject of new IT models and old, either here in the comments or via Twitter, where my handle is @jamesurquhart.

For more discussion on devops and next-generation systems management, check out this panel discussion (that includes James Urquhart) from Structure 2012:

Feature image courtesy of Shutterstock user Claudio Divizia; name tag image courtesy of Shutterstock user alexmillos.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Continuous delivery and the world of devops
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

However, as I started trying to outline a response, I realized that there was a lot of ground to cover. The core of Phil’s argument comes from his background as a hardware and systems administration expert in traditional IT organizations. And with that in mind, what he articulates in the post is a reasonable way to see the world.

However, cloud computing is changing things greatly for software developers, and these new models don’t take kindly to strict control models. From an application down perspective, Phil’s views are highly suspect, given the immense success of companies like Etsy and Netflix (despite their recent problems) have had with continuous deployment and engineering for resiliency.

Reconciling the two views of the world means exploring three core concepts required to understand why a new IT model is emerging, but not necessarily replacing everything about the old model.

The first of these concepts is devops, which earned its own three-part series from me a few years ago, and has since spawned off its own IT subculture. The short, short version of the devops story is simple: modern applications (especially on webscale, or so-called “big data” apps) require developers and operators to work together to create software that will work consistently at scale. Operations specifications have to be integrated into the application specifications, and automation delivered as part of the deployment.

In this model, development and operations co-develop and very often even cooperate, thus the term devops.

The second concept is one that I spoke about often in 2012: complex adaptive systems. I’ve defined that broad concept in earlier posts, but the stability-resiliency tradeoff is a concept that is derived from the study of complex adaptive systems. Understanding that tradeoff is critical to understanding why software development and operations practices are changing.

The third concept is that of anti-fragility, a term introduced by Nassim Nicholas Taleb in his recent book Anti-fragile: Things That Gain from Disorder. Anti-fragility is the opposite of fragility: as Taleb notes, where a fragile package would be stamped with “do not mishandle,” an anti-fragile package would be stamped “please mishandle.” Anti-fragile things get better with each (non-fatal) failure.

Although there are elements of Taleb’s commentary that I don’t love (the New York Times review linked to above covers the issues pretty well), the core concept of the book is a critical eye-opener for those trying to understand what cloud computing, build automation, configuration management automation and a variety of other technologies are enabling software engineers to do today that were prohibitively expensive even 10 years ago.

So, over the next few weeks, I will try to explore these concepts in greater detail. Along the way, I will endeavor to address Jaenke’s concerns about the ways in which these concepts can be misapplied to some IT activites.

Please join me for this exploration. Use the comments section to push back when you think I am off-base, acknowledge when what I say matches what you have experienced, and, above all, how you think about how your organization and career will change one way or another.

As always, I can be found on Twitter as @jamesurquhart.

Feature image courtesy of Shutterstock user Sinisa Botas.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Continuous delivery and the world of devops
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

Nick Carr famously wrote about the lack of differentiation that IT brings to the business in his 2004 book Does IT Matter?. His argument was essentially that as computing is more and more expected in business, the things IT does for the business provide less and less differentiation. That means that each business owning its own information technology makes less and less sense.

If you believe Carr’s vision, cloud computing is one logical outcome. In fact, Carr himself made the argument in his subsequent 2008 book, The Big Switch, where he argued that the increasingly commoditized nature of computing would drive it toward a utility model, like electricity or water. It’s a compelling vision, and one that helped to ignite the cloud computing market we know today.

So,  is it possible IT departments will fade away completely as more IT services are available from third parties online and there are fewer legal hurdles to creating, analyzing and taking action on data outside of an organization’s own figurative four walls? I don’t think so.

I think there is a critical role for a central IT unit in organizations of any significant size (beyond help desks and device management). To understand what that role is, however, we have to explore the nature of the applications being created, acquired and operated in those organizations. We also have to explore what IT has been doing, and why that’s so disassociated from what they would be asked to do in a cloud-centric organization.

The IT we know and love

One of the problems of generalizing about IT is that any given organization probably operates somewhat differently from any other. However, there are some general trends that evolved from the client-server computing model that I would argue apply to most mature IT organizations. The most-critical of these trends is what I would call “server centricity,” which might be better called “infrastructure centricity.”

Think about how computing got its start. Before you could do anything, you needed a computer. Once you had a computer, you needed an operating system, which acted as an interface between the human and the machine. With those things in place, you could now decide how you wanted to apply that computer to some form of problem (or some set of problems). That’s where application software came in.

The computer (or the switch or the storage system) was critical to this model. Without the hardware, nothing else happened.

So, IT evolved to take on running infrastructure (and, almost always, operating systems, middleware and databases) to support the applications that business units required to do their jobs. This function grew in complexity until companies spent significant budgets on data centers, infrastructure availability, inter-networking and so on. This was the most critical role IT could possibly play for the business.

IT and cloud computing

Developers — the ones who ultimately applied computing to business problems — were frustrated with the understandably limited capacity that IT had for addressing software opportunities. Standing up infrastructure is work — often expensive work — and the time and money needed to deliver it could never keep up with the so-called long tail of developer demand.

Now, however, the game has changed significantly. General infrastructure is available on a cashflow-friendly basis to anyone who wants it. Add to that the variety of innovative software tools and services that have evolved thanks to the internet, open source and the new economics of cloud computing, and developers are finding utility services a much more palatable option than internal IT for many classes of application development and deployment.

Provisioning virtual servers on Amazon Web Services.

When developers think about operations, they are most definitely focused on the applications themselves, not the infrastructure the applications are running on.

So IT is getting cut out of the loop in many organizations? Not “officially,” and often in very stealthy ways. However, it is happening, and increasingly in unexpected industries and companies. And while most of this happens with the two critical software classes that cloud enables – web applications at scale, and data collection and analysis —  some of it is just developer frustration with IT in general.

So what is an IT department to do?

I think the answer comes in recognizing what “application-centricity” really means in a complex business. No business runs on one application. No business has only one deployment that they manage, only one executable that must meet the breadth of its computing demand. Every company runs on a system of applications: a collection of  highly interconnected, interdependent software components, services and data that must all work as required in order for the company as a whole to survive and thrive.

In the era of cloud computing, what the business requires of a central IT department is coordination of the application system — aiding the various application owners with what has to happen for their software to be a “good citizen” within the computing environment as a whole.

Here are just a few key questions that IT must answer with respect to the new application system:

1. How does the company handle identity, authentication, authorization, data management, and other central security and compliance-related operations functions that must be coordinated across all of its independent operating entities.

2. How does the company troubleshoot issues that happen when applications interact with each other across operating entity or even development team boundaries?

3. Is there anything that can be done independent of the individual appliations to improve the heath of the system as a whole?

4. Who knows the system as a whole well enough to give the appropriate advice on how to best integrate new application ideas and components?

Thus, the primary role of IT moves from running infrastructure to operating software — or, more accurately, assisting developers in operating their software in a larger software system. It’s a consultative role with a number of tools and services that have to be in place and — this is very important — relevant to the developers that IT is supporting. These could be tools to visualize how applications are interconnected and the resources they’re consuming, or services that add intelligence to operations.

The core idea is that IT has to let go of trying to control everything and focus on coordinating and enhancing things that other people control. It can be done. Several online companies, including Netflix and Amazon do it today. The result is significantly better agility, experimentation and innovation, with the trade-off that cooperation, communication and measurement are increasingly critical to success.

The scary part is that most IT organizations are still infrastructure-centric, or at least “context”-centric if you include so-called enterprise software packages. The move to application-centricity and developer self-service is going to be hard, and require some change of skills and culture. It also means that private cloud is not the most important cloud initiative that IT can take on.

I wonder how long it will be until most IT organizations figure that out.

Feature image courtesy of Shutterstock user Arjuna Kodisinghe.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Near-term strategies for IT managers
• New challenges for the IT organization
• A near-term outlook for big data

To understand why are they beginning to lap the field, it is important to understand what has been successful in cloud computing to date, and what hasn’t been successful (or at least not yet).

Market dynamics create the opportunity

Despite several years of basic cloud services being available, and a small explosion of different services and business models, success in the cloud has been somewhat limited to a few areas. Software as a service has been surprisingly (to some) successful, but mostly for task-specific consumer applications, or for contextual (i.e., non-differentiated) business applications.

The other successful service market, infrastructure as a service, only has two application classes that are really successful: large-scale web applications and data analysis/processing. The mass migration of legacy transactional applications to IaaS has yet to really take hold, if it ever will. For the most part, the economic advantages for IaaS really still belong to apps that have dynamic natures — either fixed execution timeframes (e.g., data processing) or variable load (e.g., web apps).

Platform as a service, while showing real promise, is not being used at nearly the scale of these other two categories. I would contend, as well, that the most successful platforms will either run the types of applications that are already successful on IaaS, or be “attached” to SaaS applications in order to extend, enhance and integrate those offerings.

Thus, when it comes to PaaS, I think the biggest beneficiaries of its eventual success — which is likely, but not guaranteed — are the same two companies I believe are pulling away today.

Why Amazon Web Services?

I almost feel silly calling out Amazon today, as today its dominance seems so obvious. But 18 months ago, while it was definitely the visionary among the IaaS offerings (as I noted in a follow-up post), it hadn’t really stepped into the era of offering services that competitors couldn’t match within 12 to 18 months (assuming those competitors had the vision to do so).

What is so disappointing, however, is that none of Amazon’s competitors really took the challenge. While many focused on EC2 (compute), some on S3 (object storage) and a few on RDS (relational database) or SimpleDB (key/value storage), none understood the total package that Amazon was providing. What sets Amazon apart are things like reserved instances, a spot market for unused capacity, an integrated management console, and an overall focus on what customers need today to build and run the applications that make cloud valuable.

Yes, I believe Microsoft has some competitive capability, but it is, for now, focused on  features and services it can sell to developers piecemeal. Amazon started this way, but I believe James Hamilton and others helped Amazon determine early on that the real financial market in cloud happens in operations, not development. Amazon’s spot and reserved instance markets are examples. Microsoft doesn’t seem to have grasped it … yet.

Google, too, has the scale to compete with Amazon, but I don’t see it scrambling to address operational needs. Where is the unified console for operating and monitoring all the Google services an IT team is using? How about additional services around application deployment, configuration and monitoring? Google has promise, but it also has a way to go.

All in all, nobody has put the package together that Amazon has for the new computer utility market (as my friend Simon Wardley would call it).

Why Salesforce.com?

The thesis of my earlier post was the idea that the eventual cloud winners would offer emerging businesses completely integrated, but “mixable,” sets of IT capability on demand from a single service interface. While I expected Microsoft and Google to be the leaders in that space (and they still could be in eight years), the current leader appears to be Salesforce.com.

Sadagopan Singam has a great post outlining his thoughts about the recent Dreamforce conference put on by the SaaS leader. He calls the company an “enterprise nerve center” that can let all parts of an organization work in tandem to respond to business needs.

What Salesforce is doing so well is combining the core functions of business and the social interactions with customers, partners, and within the organization itself. So, as work gets created, moves through the company, and results in deliverables and/or revenue, Salesforce can automate key elements, coordinate the human aspects, and measure and analyze it all.

This is disruptive stuff. Most existing enterprise software companies drive business by moving documents around and relying on humans to handle their own communications around those documents. This is slowly changing (and is not, frankly, my area of expertise), but you can see a distinct difference in the language used by Salesforce, and that of Microsoft or Google.

Microsoft can certainly give Salesforce a run for its money here, assuming Redmond can overcome the internal inertia and politics involved in realigning products in such a fundamental way. The same is true of Oracle. Google is largely a no-show in this respect, in my opinion, as anything other than productivity apps seems outside its comfort zone.

But Salesforce still has a long way to go in subverting the existing relationships, technologies, processes and cultures that legacy vendors have established. It’s not just that its software has to be better, it also has the challenge of convincing more IT organizations that SaaS is the way to go when upgrading or replacing legacy applications. However, Salesforce has met that challenge often in the last decade.

Is it game over in the cloud?

I am unwilling to say these companies have locked down the future of cloud computing, but both have engaged in what economist Brian Arthur once called “The Law of Increasing Returns.” Now that they’ve established some advantage, they are going to start attracting more and more of the IT ecosystem in a positive feedback loop that drives more market share back to them.

As to companies that can disrupt (or at least blunt) their bright futures, keep an eye on Microsoft. If it can complete the work it’s doing on changing its internal mindset and stop thinking as a product company first and a service company second, Microsoft will begin to make great strides in integrating its formidable portfolio.

The rest of the cloud computing market lines up in one category or another, but no company has yet to show signs of significant disruption except in niche areas. There are also myriad disruptive startups in the cloud computing space, although the pace of that disruption will continue to be slow and steady. Many established IT companies are only now realizing the time and effort it to address this opportunity before it’s too late.

In the meantime, I want to leave you with one last scary thought about Amazon. In the last few weeks, it has announced its Glacier archiving service and a marketplace for selling unused reserved instances. It also has its first major conference coming up in November. If it announced those two important services now, I wonder what Amazon is waiting to announce at the conference …

Feature image courtesy of Shutterstock user Neale Cousland.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Amazon’s DynamoDB: rattling the cloud market
• A field guide to cloud computing: current trends, future opportunities

In this case, however, I’m not talking simply about creating and controlling interoperability from the developer level. Tools and services like Dell’s Boomi or IBM’s CastIron have existed for years, and have some success in delivering more flexibility to integration between applications and services. However, these services are focused on solving the developer’s key issues with integration –how to make sure messages move between components based on a process definition and one or more translations, if needed.

The interoperability challenges facing IT operations

But today application operators see a tangental set of problems, and these problems are increasingly becoming difficult to deal with. For the operators, the problem of interoperability has several parts:

• Maintaining interoperability with dependencies.For the developer, the problem of managing dependencies is one of logic—finding the right configuration of code and file dependencies to allow the application to execute successfully. This is largely a static problem, though one that increasingly requires devs to design for resiliency; if one dependency disappears, an alternative method of achieving the task at hand should be attempted instead. For operations, however, the problem is ongoing, as operations has to deal with the reality of why a dependency failed the component or components that depended on it.
• Maintaining interoperability for dependents. The rapid growth of cloud services and APIs, on the other hand, make it operations’ job to deliver availability, performance and consistency of the software systems they operate to those that depend on that software. If you plan on earning business via services delivered via APIs, your operations team has to ensure that those services are there when your customers need them, without fail. Even if you simply provide data via batch files to a partner or customer, that mechanism has to run as the customer expects it to, every time.
• Maintaining interoperability with things operations controls. The other key aspect of operations focus on interoperability has to do with control. There is a variety of responsibility that is inherent in operating systems that interact with one another. The goal of operations, in this case should be to optimize how these systems work together once deployed. Some of that is going back to developers and asking for changes to the applications or data themselves, but often much of that optimization has to do with network and storage configuration, tuning virtualization platforms, ensuring security systems and practices are in place, and so on.
• Maintaining interoperability with things operations doesn’t control. Perhaps the most interesting aspect of application operations in the cloud computing era is the increased need to maintain control of one’s applications in the face of losing control over key elements on which those applications depend. Dealing with upgrades of third party services, handling changes to network availability (or billing, for that matter), or even ensuring that data is shipped to the correct location, on the correct media, by the right delivery service, are all tasks in which operations can only effect one side of the equation, and had to trust and/or respond to changes on the other side.

Complexity makes interoperability difficult

None of this is a shock to most IT operators, but there is one other element that I’ve hinted at before that is creating the rapid expansion of complexity facing operations today, and that is the sheer volume of integrations between software and data elements both within and across organizational boundaries. It’s no longer a good idea to think of individual applications in isolation, or to assume a data element has one customer, or even one set of customers with a common purpose for using that data.

Today we live in a world where almost everything that matters in business is connected by a finite number of degrees of separation from just about everything else in that category. Cloud computing is one driver, but the success of REST APIs is another, as is the explosion of so-called “big data” and analytics across businesses and industries.

We, in business software, exist in large part to automate the economy, in my opinion. The economy is a massive, highly integrated complex adaptive system. Our software is rapidly coming to mimic it.

We need standard operations interoperability

All of this brings me to the opportunity that this interoperability explosion brings to operators and vendors of operations tools alike. If we are going to manage software and data that interoperates as a system at such a massive scale, we need tools that interoperate in support of that system. We need to begin to implement much of what my friend, Chris Hoff, called for five years ago from the security software community:

We all know that what we need is robust protocols, strong mutual authentication, encryption, resilient operating systems and applications that don’t suck.

But because we can’t wait until the Sun explodes to get this, we need a way for these individual security components to securely communicate and interoperate using a common protocol based upon open standards.

We need to push for an approach to an ecosystem that allows devices that have visibility to our data and the network that interconnects them to tap this messaging bus and either enact a disposition, describe how to, and communicate appropriately when we do so.

We have the technology, we have the ability, we have the need.  Now all we need is the vendor gene pool to get off their duff and work together to get it done. The only thing preventing this is GREED.

Amen, Chris. That remains as true today as it was then, as far as I can tell. Only now the scope has exploded to include all of application and infrastructure operations, not just security software. While everyone is looking for standards that allow one tool to talk to another, we are missing the bigger picture. We need standards that allow every component in the operations arsenal to exchange events with any other component, within understood guidelines. That may be as simple as setting the expectations that any operations software will have both an execution and a notification API set.

Another option is a formal event taxonomy and protocol, but that option doesn’t interest me very much. Those standards tend to become outdated quickly and are far too restrictive.

One last thing: John Palfrey and Urs Gasser have written a book on interoperability which I am in the middle of reading. So far, the most interesting aspect of the model they describe is a multi-tiered view of interoperability that supplements data and software interoperability with human and institutional interoperability. The latter two concepts are incredibly important in the new cloud-based systems world.

It’s not good enough to focus on software, protocols and APIs. We have to begin to work together as an ecosystem to overcome the human and institutional barriers to better IT interoperability. Unfortunately, lack of interoperability often benefits software vendors, and as Hoff noted above, the only thing preventing this is greed.

Photo credit: Cory Doctorow

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Quality of the cloud: best practices for ISVs
• A clouded view of Google Music
• What Enterprise Software Vendors Could Learn from the Consumer Space

The Google Developers Blog post announcing the service broke down three key “offers” of GCE,  which I interpret as the three key differentiators from Google’s perspective of its service over the competition (not necessarily just AWS):

• Scale. At Google we tackle huge computing tasks all the time, like indexing the web, or handling billions of search queries a day. Using Google’s data centers, Google Compute Engine reduces the time to scale up for tasks that require large amounts of computing power. You can launch enormous compute clusters – tens of thousands of cores or more.
• Performance. Many of you have learned to live with erratic performance in the cloud. We have built our systems to offer strong and consistent performance even at massive scale. For example, we have sophisticated network connections that ensure consistency. Even in a shared cloud you don’t see interruptions; you can tune your app and rely on it not degrading.
• Value. Computing in the cloud is getting even more appealing from a cost perspective. The economy of scale and efficiency of our data centers allows Google Compute Engine to give you 50% more compute for your money than with other leading cloud providers. You can see pricing details here.

Scale and price matter, but …

A demo of genome analysis on Compute Engine.

All of the coverage I’ve read to date has focused on the scale and value elements of Google’s story. And these are critically important. When it comes to scale, few can match a launch that includes roughly 100,000 servers and 770,000 cores of available capacity (though I doubt you’ll be able to grab half that for yourself in a few weeks). Google does scale for a living, and with a reported 1 million servers in operation across the company, nobody comes close — except maybe Amazon.

When it comes to value, Google fired a shot across AWS’s bow with roughly equivalent pricing, though Google argues that app-for-app, their service will be cheaper—or, as it says, “50% more compute for your money than with other leading cloud providers.” Other analysis questions this, and certainly any advantage Google has today will be challenged quickly by Amazon, given its own history of reducing prices.

Performance is Google’s ace in the hole

However, I think the key potential differentiator for Google, if it wants to take market share from AWS, is the performance of the cloud infrastructure itself. If Google can deliver a service that eliminates most of the I/O and network performance inconsistencies that AWS customers currently experience, I can guarantee you there are many major compute customers of AWS that will want to give Compute Engine a test run.

The I/O experience alone has been a real thorn in the side of many technologists, who — despite having designed applications to account for performance inconsistency — have real concerns about whether their applications can run as efficiently as possible in such an environment. If Google’s performance claims are confirmed, you will see one or two large-scale AWS customers begin to spread their compute loads between the two services by the middle of 2013. Heck, at least one may actually move off of EC2 altogether.

Still, if performance turns out to be not significantly better than AWS, or if there are other major limitations to the Compute Engine service, those customers will quietly put to rest their experiments and return to the “tried and true” AWS service that they know. And others will no doubt use Compute Engine, even if it’s not the best thing since sliced bread, turning it into a real contender in the cloud computing space but nothing near an AWS killer. The scale and value stories are legitimate, they’re just not compelling enough by themselves to drive too many customers to change how they do cloud computing.

Feature image courtesy of Shutterstock user David Castillo Dominici; 

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Understanding and managing the cost of the cloud
• Cloud computing infrastructure: 2012 and beyond

Defined in Wikipedia as “the process of understanding how things influence one another within a whole,” systems thinking represents a modeling, analysis and design discipline that carefully explores “macro” aspects of highly interdependent systems. Systems thinking is heavily utilized in such fields as the social sciences, organizational dynamics, and industrial engineering to evaluate, model, and/or design how systems are composed and how they behave.

Systems thinking is difficult for those that have been educated to always apply reductionist thinking to problem solving. The idea in systems thinking is not to drill down to a root cause or a fundamental principle, but instead to continuously expand your knowledge about the system as a whole.

The problem of cloud boundaries

It’s one of the fascinating questions that faces anyone trying to model a system: What are the system’s boundaries? When everything is so highly interdependent (economies are linked to governments are linked to societies are linked to individual people, etc), how do you know where to start modeling, and where to stop?

In her classic book on systems thinking, the late Donella H. Meadows articulated brilliantly the challenge that “systems thinkers” are faced when scoping the problem they need to address:

“The lesson of boundaries is hard even for systems thinkers to get. There is no single, legitimate boundary to draw around a system. We have to invent boundaries for clarity and sanity; and boundaries can produce problems when we forget that we’ve artificially created them…

…There are no separate systems. The world is a continuum. Where to draw a boundary around a system depends on the purpose of the discussion—the questions we want to ask.”

She goes on to say:

“The right boundary for thinking about a problem rarely coincides with the boundary of an academic discipline, or with a political boundary. Rivers make handy borders between countries, but the worst possible borders for managing the quantity and quality of the water. Air is worse than water in its insistence on crossing political borders. National boundaries mean nothing when it comes to ozone depletion in the stratosphere, or greenhouse gases in the atmosphere, or ocean dumping.”

This, I think, is a critical observation for people building large scale cloud-computing applications and services that integrate with other applications and services in the cloud. Understanding where the boundaries of source code and data models lie is relatively straightforward, but understanding the boundaries of operations — monitoring, compliance, decision making, liability and so on in cloud-based applications — is not so straightforward.

The nature of cloud systems boundaries

In fact, I would argue that the nature of cloud-systems boundaries will themselves be highly dynamic, in part because of the comings and goings of technologies and services (not to mention politics, economics, and so on). However, it is also true that it will take time to discover the full extent of those systems for each application or service you operate, as everything is so interconnected.

This is different from what we experienced with so-called “traditional IT”, as we could typically maintain control of all but a few elements of our application systems, and the applications were generally quite isolated. We strived for stability, and that included stable boundaries. It is clear to me that this is becoming increasingly impossible.

There is also an interesting corollary to the problem of boundaries that must be considered when planning any application or service that might be consumed by outside parties. If you do not necessarily know which third-party services affect your “system”, it stands to reason that you also do not know which external systems or applications your offering affects.

In other words, how do you know the application systems that you may ultimately impact if anyone can consume your service at any time? Are you making it easy for them to design “around” you for their own resilience?

They key is transparency

All of this leads me to what I think is the key conclusion that has to be reached about the future architecture of our shared cloud computing “system”: transparency is essential. Without a steady stream of feedback data from whatever sources we determine — over time — have a significant impact on the operation of our applications, we are doomed to be unable to properly find the right “boundaries” for those applications.

Information about the functioning state of infrastructure (like compute nodes and networks), services (like data stores and platform services) or even other applications (like SaaS or your partners’ applications) will be critical to evolving the automation that successfully enables resiliency. And, as I noted in my keynote at last month’s excellent Gluecon in Colorado, one key goal in these systems is resiliency.

Will such transparency happen? I believe it already is. Just this week, Amazon Web Services announced a method for downloading billing information for their services. At one point in time, it was postulated that Amazon would never do this. However, customers have spoken, and the need to access real time costs of Amazon’s services programmatically has forced transparency.

Regardless, it is important to start thinking about your applications and services in the cloud as systems, not just stand-alone components. The challenge before you is to determine what the boundaries of those systems are, and how to design, build and operate your software to thrive within those boundaries.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang
• From car to cloud: the future of the in-vehicle app landscape