19 Comments

Summary:

In his latest post on next-generation systems design, James Urquhart discusses the different types of PaaS offerings and why it matters that some are composable and others are contextual.

I want to touch base on a topic that is subtle, but has a profound impact on the way anti-fragile IT systems will evolve and in what Platform-as-a-Service offerings companies will choose to use: the difference between two types of extensibility and programmability in systems, contextual and composable. This topic is an important part of my continued exploration of how the concepts of devops, complex adaptive system and anti-fragility apply to software development and IT operations in the era of cloud computing.

These two patterns are described well in this recent post from Neal Ford, self-described “Director, Software Architect, and Meme Wrangler” at systems integrator ThoughtWorks:

In my keynote, I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive…The knowledge and effort required for a seemingly trivial change prevents the change from occurring, leaving the developer with a perpetually dull tool. Contextual tools aren’t bad things at all – Eclipse and IntelliJ wouldn’t exist without that approach. Contextual tools provide a huge amount of infrastructure that developers don’t have to build. Once mastered, the intricacies of Eclipse’s API provide access to enormous encapsulated power…and there’s the rub: how encapsulated?

In the late 1990’s, 4GLs were all the rage, and they exemplified the contextual approach. The built the context into the language itself: dBASE, FoxPro, Clipper, Paradox, PowerBuilder, Microsoft Access, and similar ilk all had database-inspired facilities directly in the language and tooling. Ultimately, 4GLs fell from grace because of Dietzler’s Law, which I defined in my book Productive Programmer, based on experiences by my colleague Terry Dietzler, who ran the Access projects for my employer at the time:


Dietzler’s Law for Access

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.


Ultimately Dietzler’s Law killed the market for 4GLs. While they made it easy to build simple things fast, they didn’t scale to meet the demands of the real world. We all returned to general purpose languages.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things. A famous story from 1992 illustrates just how powerful these abstractions are. Donald Knuth was asked to write a program to solve this text handling problem: read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. He wrote a program consisting of more than ten pages of Pascal, designing (and documenting) a new algorithm along the way. Then, Doug McIlroy demonstrated a shell script that would easily fit within a Twitter post that solved the problem more simply, elegantly, and understandably (if you understand shell commands):

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q

I suspect that even the designers of Unix shells are often surprised at the inventive uses developers have wrought with their simple but powerfully composable abstractions.

Ford goes on to describe the pros and cons of each approach in much more detail, but the key conclusion he reaches is, I think, critical to understanding how one should develop the tools and tool chains that drive new IT models:

These abstractions apply to tools and frameworks as well, particularly tools that must scale in their power and sophistication along with projects, like build tools. By hard-won lesson,composable build tools scale (in time, complexity, and usefulness) better than contextual ones. Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible, Dietzler’s Law Redux. This is especially true in tools where critical parts of how they function, like the ordering of tasks, is inaccessible without hacking.

Ford’s distinction is one that finally helps me articulate a key concern I’ve had with respect to Platform-as-a-Service tools for some time now. In my mind, there are primarily two classes of PaaS systems on the market today (now articulated in Ford’s terms). One class is contextual PaaS systems, in which a coding framework is provided, and code built to that framework will gain all of the benefits of the PaaS with little or no special configuration or custom automation. The other is composable PaaS, in which the majority of benefits of the PaaS are delivered as components (including operational automation) that can be assembled as needed to support different applications.

Contextual PaaS

Examples of contextual PaaS include the original releases of Google App Engine, Heroku and other “first-generation” PaaS systems that asked the developer to adhere to specific architecture and consume PaaS-specific classes in the application itself. These systems were incredibly powerful for building applications that were variations of what these frameworks were designed to do, but began to fail quickly for applications that fell outside of that domain.

The classic example is Google App Engine’s limit of 30 seconds for any backend request to complete. Great if you were building a Facebook game, but a requirement that eliminated its use for many multi-step transactional applications. Of course, there were ways to deal with those situations, as well, but they were mostly complicated and added risk to the system.

There is a parallel here with the 4GLs of the late 1990s that Ford talks about in his post. At that time, I worked for Forte Software (acquired by Sun Microsystems in 1999), which built a 4GL development and operations environment for distributed application development. We had a business model where we relied heavily on systems integrator partners to help our customers deliver these often sophisticated applications, and every one of those SIs eventually built a framework environment to make building complex applications “easier.”

The problem? Almost every customer that used one of these frameworks had a requirement (or many) that the framework didn’t handle well. This resulted in either the SIs scrambling to modify their frameworks to support these requirements — inevitably resulting in the framework being much less “easy” to use — or the customer bypassing the framework all together for those needs, resulting in an application that was harder to debug and operate.

Composable PaaS

Composable PaaS systems, on the other had, do much less to anticipate the architecture or functionality of the application built on it, and do much more to simplify the assembly of services, including underlying infrastructure, automation, data sources, specialized data tools, etc. I think the classic example of a composable PaaS is Cloud Foundry, the open source PaaS effort from VMware that’s now part of its Pivotal Initiative spinoff. Modern versions of Heroku, Engine Yard, CloudBees and other also exhibit more of this approach than “first-generation” PaaS systems.

An old, but illustrative, Cloud Foundry diagram.

An old, but illustrative, Cloud Foundry diagram.

Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).

Now, take the concept a step further. Imagine a deployment environment that delivers a wide variety of these individual tools and components and simplifies the process of creating tool chains on demand from them. Imagine that environment would let each development team choose from known tool chain “patterns,” but modify them as they see fit for each project. This, I believe, will be the ultimate general purpose PaaS success, not some hard-and-fast framework-based PaaS.

The concept of composable and contextual applies to a lot more than PaaS and cloud, of course. And it is important to note that it’s not an either/or choice, much like stability and resiliency. Parts of an IT environment should be composable, but there will always be elements where the relative stability of contextual extension makes more sense. And composable systems can leverage API-driven systems that themselves are designed primarily for extensibility via contextual approaches.

The key is to think about each system from the perspective of how it will be used, and to target its extensibility mechanism based on needs. Just remember, however, that choosing a contextual path will dictate a lot more about how your system could be used in the future than a composable approach would.

I’d love to hear your thoughts, either in the comments below, or on Twitter, where I am @jamesurquhart.

Feature image courtesy of Shutterstock user Nenov Brothers Photography.

  1. Great post. Your description of the practical difference between top-down contextual, framework-based solutions and solutions that are based on bottom-up, composable components is spot on.

    That said, I think the best designs often embrace and harmonize both approaches. In my experience it is possible to build top-down frameworks that are, underneath, comprised of re-usable components that the user can gain access to if needed. I call it the “onion” approach. With this approach the system is designed in layers, like an onion. The outer layers can be used contextually, like a framework. However, the system is designed such that any layer can be peeled away to reveal the one underneath. As layers are peeled back, the end-user is able to transition to using composition, allowing even the most unanticipated use-cases to be addressed. At no point is the user unable to dig underneath the abstractions they are presented with.

    It is hard to design systems like this, but it can be done. When done properly these systems offer the get-going-quickly-dont-bother-me-with-the-details usage model of the contextual approach with the flexibility and power of the compositional approach if needed.

    Will

    Share
  2. Delete yourself, hervey papele.

    Share
  3. Good article. What’s your take on the role of declarative approaches to system configuration, e.g. http://projects.puppetlabs.com/projects/1/wiki/Big_Picture ?

    Are there examples of OSS composition techniques which play a role equivalent to I/O pipes in the Unix shell, e.g. message queuing (RestMS, AMQP, RabbitMQ, XMPP, PSYC, etc)?

    Share
  4. Hey James, in your list of composable PaaS you forgot the original: dotCloud (http://dotcloud.com) has been advocating “mix-and-match paas” since late 2010 (http://techcrunch.com/2010/12/22/yc-funded-dotcloud-an-application-platform-that-lets-you-mix-and-match/).

    You’re right, service composition is the only way to architect an application that won’t collapse under its own weight as it grows. As more people figure this out I think the tooling available will only get more awesome.

    It’s a great time to be a developer!

    Share
  5. Is this what Apcera planning to do… Did u reveal their secret

    Share
  6. Sound like but not exactly what we used to call Utility and Functional sub-routines 40 years ago. Good algorith amd program design practices never go out of style – improve and age fine with time

    Share
  7. Contextual designs are not a failing technology, they are a tradeoff just like anything else….

    It’s all about knowing your market. Not eveyone is a saavy developer who wanting to compose everything from scratch otherwise there wouldn’t be a large and thriving CMS market.

    MS Access didn’t “fail” by any stretch. It was widely successful for many years and eventualy got replaced by other things like Sharepoint, Salesforce, etc.

    Share
  8. Derick Winkworth Sunday, February 17, 2013

    Great article! These concepts… contextual vs composable… apply to other aspects of the IT world too, think.

    It seems to me that the folks who make an art of duct taping components together to get stuff done are the ones that will succeed. Some components more on the composable end, some components on the contextual end… pick what’s appropriate for your application. If you don’t need to fiddle and tune, then don’t.. throw it into the blackhole of a contextual component.

    Share
  9. pradeepprabhu1 Sunday, February 17, 2013

    Hey James, loved your thoughts on open source “build” tool chains…this is exactly what CloudMunch delivers (www.cloudmunch.com). We call it DevOps PaaS. Integrating with GitHub or any code repo, with Jenkins CI, Selenium for Automated Tests, AWS Beanstalk, CloudFormation and monitoring tools like New Relic, Nagios + ALM tools like Jira. We have implemented an App model where the integrations or for that matter Build/Deploy/Run are Apps that a developer can choose depending on what they see fit for each project. As developers rely on a bevy of cloud services, a composable DevOps approach means you can orchestrate the entire application life cycle with utmost flexibility for maximum effectiveness.

    Pradeep (@cloudmunch.com)

    Share
  10. I read, and read this post, and finally I think I got it…. it was all about making the developers life easier.

    My desire is a world not of developer centric architectures but a join of data and user centric architectures, where usability, usefulness, trust and collaboration are the watch words.

    Simply put; I want my Apps to use my data, not their data.

    The current world of Apps does some clever things but I find my self having to mold / recreate / copy / re-enter my data to meet the various Apps needs.

    On the surface it looks like a composable approach will make the delivery of my user/data centric world easier to deliver. (I suspect as in all life a hybrid approach will be the most effective) The problem is that making the world I want will be harder for the developer than maintaining their Developer Centric view. It will take thinking architecture from the data and users perspective, rather than an architecture perspective. Architects in the real world get that subtle point, the relatively immature world of Systems Architecture has yet to grok the point.

    My desire is that Apple gets it’s mojo back and takes us to the next level of Useability and Usefullness. Frankly I don’t mind if a new start up called Pear makes the shift. I will very quickly become a Pear bigot! Because at the end of the day I want Systems and Systems Architectures to make MY WORLD easier, I don’t mind if that is going to mean some hard work for the developers.

    I believe the developers will get more reward and wealth from doing some hard work, if a 3 year old can develop “Compostable” programs they will be just that : Throw Away!

    (Methinks that Will Shuman appears to have grok’d it!)

    Share

Comments have been disabled for this post