Is your PaaS composable or contextual? (Hint: the answer matters)

19 Comments

I want to touch base on a topic that is subtle, but has a profound impact on the way anti-fragile IT systems will evolve and in what Platform-as-a-Service offerings companies will choose to use: the difference between two types of extensibility and programmability in systems, contextual and composable. This topic is an important part of my continued exploration of how the concepts of devops, complex adaptive system and anti-fragility apply to software development and IT operations in the era of cloud computing.

These two patterns are described well in this recent post from Neal Ford, self-described “Director, Software Architect, and Meme Wrangler” at systems integrator ThoughtWorks:

In my keynote, I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive…The knowledge and effort required for a seemingly trivial change prevents the change from occurring, leaving the developer with a perpetually dull tool. Contextual tools aren’t bad things at all – Eclipse and IntelliJ wouldn’t exist without that approach. Contextual tools provide a huge amount of infrastructure that developers don’t have to build. Once mastered, the intricacies of Eclipse’s API provide access to enormous encapsulated power…and there’s the rub: how encapsulated?

In the late 1990’s, 4GLs were all the rage, and they exemplified the contextual approach. The built the context into the language itself: dBASE, FoxPro, Clipper, Paradox, PowerBuilder, Microsoft Access, and similar ilk all had database-inspired facilities directly in the language and tooling. Ultimately, 4GLs fell from grace because of Dietzler’s Law, which I defined in my book Productive Programmer, based on experiences by my colleague Terry Dietzler, who ran the Access projects for my employer at the time:


Dietzler’s Law for Access

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.


Ultimately Dietzler’s Law killed the market for 4GLs. While they made it easy to build simple things fast, they didn’t scale to meet the demands of the real world. We all returned to general purpose languages.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things. A famous story from 1992 illustrates just how powerful these abstractions are. Donald Knuth was asked to write a program to solve this text handling problem: read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. He wrote a program consisting of more than ten pages of Pascal, designing (and documenting) a new algorithm along the way. Then, Doug McIlroy demonstrated a shell script that would easily fit within a Twitter post that solved the problem more simply, elegantly, and understandably (if you understand shell commands):

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q

I suspect that even the designers of Unix shells are often surprised at the inventive uses developers have wrought with their simple but powerfully composable abstractions.

Ford goes on to describe the pros and cons of each approach in much more detail, but the key conclusion he reaches is, I think, critical to understanding how one should develop the tools and tool chains that drive new IT models:

These abstractions apply to tools and frameworks as well, particularly tools that must scale in their power and sophistication along with projects, like build tools. By hard-won lesson,composable build tools scale (in time, complexity, and usefulness) better than contextual ones. Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible, Dietzler’s Law Redux. This is especially true in tools where critical parts of how they function, like the ordering of tasks, is inaccessible without hacking.

Ford’s distinction is one that finally helps me articulate a key concern I’ve had with respect to Platform-as-a-Service tools for some time now. In my mind, there are primarily two classes of PaaS systems on the market today (now articulated in Ford’s terms). One class is contextual PaaS systems, in which a coding framework is provided, and code built to that framework will gain all of the benefits of the PaaS with little or no special configuration or custom automation. The other is composable PaaS, in which the majority of benefits of the PaaS are delivered as components (including operational automation) that can be assembled as needed to support different applications.

Contextual PaaS

Examples of contextual PaaS include the original releases of Google App Engine, Heroku and other “first-generation” PaaS systems that asked the developer to adhere to specific architecture and consume PaaS-specific classes in the application itself. These systems were incredibly powerful for building applications that were variations of what these frameworks were designed to do, but began to fail quickly for applications that fell outside of that domain.

The classic example is Google App Engine’s limit of 30 seconds for any backend request to complete. Great if you were building a Facebook game, but a requirement that eliminated its use for many multi-step transactional applications. Of course, there were ways to deal with those situations, as well, but they were mostly complicated and added risk to the system.

There is a parallel here with the 4GLs of the late 1990s that Ford talks about in his post. At that time, I worked for Forte Software (acquired by Sun Microsystems in 1999), which built a 4GL development and operations environment for distributed application development. We had a business model where we relied heavily on systems integrator partners to help our customers deliver these often sophisticated applications, and every one of those SIs eventually built a framework environment to make building complex applications “easier.”

The problem? Almost every customer that used one of these frameworks had a requirement (or many) that the framework didn’t handle well. This resulted in either the SIs scrambling to modify their frameworks to support these requirements — inevitably resulting in the framework being much less “easy” to use — or the customer bypassing the framework all together for those needs, resulting in an application that was harder to debug and operate.

Composable PaaS

Composable PaaS systems, on the other had, do much less to anticipate the architecture or functionality of the application built on it, and do much more to simplify the assembly of services, including underlying infrastructure, automation, data sources, specialized data tools, etc. I think the classic example of a composable PaaS is Cloud Foundry, the open source PaaS effort from VMware (s vmw) that’s now part of its Pivotal Initiative spinoff. Modern versions of Heroku, Engine Yard, CloudBees and other also exhibit more of this approach than “first-generation” PaaS systems.

An old, but illustrative, Cloud Foundry diagram.

An old, but illustrative, Cloud Foundry diagram.

Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).

Now, take the concept a step further. Imagine a deployment environment that delivers a wide variety of these individual tools and components and simplifies the process of creating tool chains on demand from them. Imagine that environment would let each development team choose from known tool chain “patterns,” but modify them as they see fit for each project. This, I believe, will be the ultimate general purpose PaaS success, not some hard-and-fast framework-based PaaS.

The concept of composable and contextual applies to a lot more than PaaS and cloud, of course. And it is important to note that it’s not an either/or choice, much like stability and resiliency. Parts of an IT environment should be composable, but there will always be elements where the relative stability of contextual extension makes more sense. And composable systems can leverage API-driven systems that themselves are designed primarily for extensibility via contextual approaches.

The key is to think about each system from the perspective of how it will be used, and to target its extensibility mechanism based on needs. Just remember, however, that choosing a contextual path will dictate a lot more about how your system could be used in the future than a composable approach would.

I’d love to hear your thoughts, either in the comments below, or on Twitter, where I am @jamesurquhart.

Feature image courtesy of Shutterstock user Nenov Brothers Photography.

19 Comments

Chris Taylor

James, great writeup. For cloud to be as revolutionary as its proponents (and hypsters) claim, it has to add composable tools delivered through the cloud version of an eclipse plug in. We wrote up our thoughts here:

http://successfulworkplace.com/2013/04/27/cloud-makes-anyone-and-everyone-a-robber-baron/

The idea that the web will follow the path of previous technology breakthroughs is very interesting. I’ve watched at least thirty years of technology move through cycles of contextual and composable and why should we think we’re done now?

Dave Duggal

Good post, but this might be false binary – you can bake your cake and eat it two!

http:www.enterpriseweb.com – composable and contextual

Happy to discuss.
Dave

Steve Wood

I think I disagree with the main sentiment of the article. Mainly because the point is taken from such a limited point of view. Just because you can code more granular low level stuff using contextual approaches, doesn’t mean the business gets 100% of what they want. Most of what the business wants is related to UI/UX and business process. Most of the reason they don’t get it is because developers normally suck at that. Contextual tools focus much more on UI/UX and business process for precisely this reason. Often, the business is happy with 80% as long as that 80% really does support their existing processes and help them automate difficult or manual processes. Often 100% is very elusive – using low level frameworks it simply gets to expensive and complex to give the full end-to-end solution and using contextual frameworks can make is too clunky to hit the edge use-cases – so those must remain manual or out of scope. In either scenario, there’s no 100%. Equally, if you spend a lot of time working on business IT projects, things are never finished. Business projects are agile, iterative and never ending – because business is always evolving. IT projects are done with a build to last mentality – business projects should be embarked upon with a build for change approach. Access does very well at that – with immediacy and iteration. So does Lotus Notes and most recently force.com from salesforce.

I’m not sure I get this article. The listing of 4GL players is just odd – not sure how Heroku can be compared with MS Access. Sounds like an old C or binary programmer puffing their chest about these new fangled languages that don’t require you to manage memory – so you can’t possibly be a REAL developer! ;)

tkunze

Great article, James, and a thoughtful reminder that Amazon might just turn out to be “The PaaS” after all. While I agree with your and some of the commenters’ observation that one developer’s composable Legoland might well be the next guy’s contextual walled garden, I still believe the essence of composability at any level can be broken down into three characteristics:

1. Components must be elementary enough to not matter to my application
2. Components must be flexible enough so I don’t have to modify them
3. Connections must be trivially simple

Today’s crop of PaaS has a number of challenges in that regard. First, they are in the business to provide other people’s “contexts”, so to speak: JEE, PHP, MySQL, MongoDB, etc. But even beyond that, composability is limited by a number of factors, such as the specific versions of languages/frameworks/middleware/etc. on offer (“sorry, we don’t yet support PHP 5.4”), architectural restrictions (“must have at least one app container”), development and operational processes supported, and, of course, by rather minimal support for business requirements.

Amazon, being off-premise, certainly has its own set of challenges and isn’t quite as trivial to wire up as Unix pipes in a shell, but it dwarfs PaaS offerings when it comes to being flexible and elementary. True, there is value in being contextual as there is in being composable. But the question is: can a composable system minimize the added value of a contextual system? And, if yes, how close is Amazon to getting there?

Treff LaPlante

As the product driver behind WorkXpress (www.workxpress.com) for over 10 years now, and as I’ve watched the PaaS market from its beginning, Dietzler’s Law is the sort of thing that I’ve bumped up against many many times.

I believe the law as written has one fundamental flaw though that makes it fall apart; cost. People don’t “want what they want” as the law proscribes, but rather, they live with what they can afford. Cost is the ultimate driver in determining the set of features that a business user will implement.

For most 4GL style PaaS, the actual numbers are closer to 60% visual tools, 39% coding required, and 1% simply can’t do.

The art of a successful code-less PaaS is to get to closer 99% visual tools, 1% coding or can’t do.

The key thing here is that the visual tools cut deployment times and cost so much (let’s say 90%), that the business user will always want to forego the small percentage of things they can’t have for whatever reason.

And this is then is the fundamental misnomer of Dietzler’s Law: Do you want 95% at a 90% time and cost reduction, or 100% for 100% of the price? Are you willing to forego 5% of your wish list for a 90% reduction in cost?

PaaS customers are emphatically saying “yes”.

Raj

Thanks for bringing up this topic and well written post. We do believe that composable platforms provide the flexibility that many applications require, especially in the enterprises. Amazon web services is way well adopted than any other PaaS and exactly for the reasons that services can be orchestrated for designing complex architectures which is a major limitation on many PaaS offerings today. Cumulogic platform provides the ease of use of contextual PaaS as well as cloud services that can ve orchestrated to build a robust backend-as-a-service.

natishalom

James great writeup as usual – thanks!
In the context of compassable PaaS i think that it is important to have a measure of how compassable your PaaS is really is as i see many names that provides some degree of flexibility jumping into that category.

The ultimate measure IMO is 1)How fast can you deploy a given or desired application blue-print, the second measure is to measure the degree of diversity i.e. how many kind of application blue-prints can you support into your platform.

You’ll find that CloudFoundry and other opensource platform comes pretty high on that list but would still require changes to the core codebase to introduce new stacks.

This is where it would makes most sense to marry the concepts from DevOps into the PaaS world. By that i mean that the ability to compose the application stack and define new blue-print would be a first class citizen feature in the platform and not a hack.

A DSL that would enable to define and extend the application stack leads to a higher degree of composability by the measures that i outlined above.

As Krishnan pointed out new generation PaaS framework such as Cloudify and Brooklyn fits into that second category. For more details on this approach see one of my recent posts: Putting DevOps and PaaS together: http://natishalom.typepad.com/nati_shaloms_blog/2012/08/putting-devops-and-paas-together-with-cloudify.html

Adrius42

I read, and read this post, and finally I think I got it…. it was all about making the developers life easier.

My desire is a world not of developer centric architectures but a join of data and user centric architectures, where usability, usefulness, trust and collaboration are the watch words.

Simply put; I want my Apps to use my data, not their data.

The current world of Apps does some clever things but I find my self having to mold / recreate / copy / re-enter my data to meet the various Apps needs.

On the surface it looks like a composable approach will make the delivery of my user/data centric world easier to deliver. (I suspect as in all life a hybrid approach will be the most effective) The problem is that making the world I want will be harder for the developer than maintaining their Developer Centric view. It will take thinking architecture from the data and users perspective, rather than an architecture perspective. Architects in the real world get that subtle point, the relatively immature world of Systems Architecture has yet to grok the point.

My desire is that Apple gets it’s mojo back and takes us to the next level of Useability and Usefullness. Frankly I don’t mind if a new start up called Pear makes the shift. I will very quickly become a Pear bigot! Because at the end of the day I want Systems and Systems Architectures to make MY WORLD easier, I don’t mind if that is going to mean some hard work for the developers.

I believe the developers will get more reward and wealth from doing some hard work, if a 3 year old can develop “Compostable” programs they will be just that : Throw Away!

(Methinks that Will Shuman appears to have grok’d it!)

pradeepprabhu1

Hey James, loved your thoughts on open source “build” tool chains…this is exactly what CloudMunch delivers (www.cloudmunch.com). We call it DevOps PaaS. Integrating with GitHub or any code repo, with Jenkins CI, Selenium for Automated Tests, AWS Beanstalk, CloudFormation and monitoring tools like New Relic, Nagios + ALM tools like Jira. We have implemented an App model where the integrations or for that matter Build/Deploy/Run are Apps that a developer can choose depending on what they see fit for each project. As developers rely on a bevy of cloud services, a composable DevOps approach means you can orchestrate the entire application life cycle with utmost flexibility for maximum effectiveness.

Pradeep (@cloudmunch.com)

Derick Winkworth

Great article! These concepts… contextual vs composable… apply to other aspects of the IT world too, think.

It seems to me that the folks who make an art of duct taping components together to get stuff done are the ones that will succeed. Some components more on the composable end, some components on the contextual end… pick what’s appropriate for your application. If you don’t need to fiddle and tune, then don’t.. throw it into the blackhole of a contextual component.

oolong2

Contextual designs are not a failing technology, they are a tradeoff just like anything else….

It’s all about knowing your market. Not eveyone is a saavy developer who wanting to compose everything from scratch otherwise there wouldn’t be a large and thriving CMS market.

MS Access didn’t “fail” by any stretch. It was widely successful for many years and eventualy got replaced by other things like Sharepoint, Salesforce, etc.

Sri K

Sound like but not exactly what we used to call Utility and Functional sub-routines 40 years ago. Good algorith amd program design practices never go out of style – improve and age fine with time

Solomon Hykes

Hey James, in your list of composable PaaS you forgot the original: dotCloud (http://dotcloud.com) has been advocating “mix-and-match paas” since late 2010 (http://techcrunch.com/2010/12/22/yc-funded-dotcloud-an-application-platform-that-lets-you-mix-and-match/).

You’re right, service composition is the only way to architect an application that won’t collapse under its own weight as it grows. As more people figure this out I think the tooling available will only get more awesome.

It’s a great time to be a developer!

Will Shulman

Great post. Your description of the practical difference between top-down contextual, framework-based solutions and solutions that are based on bottom-up, composable components is spot on.

That said, I think the best designs often embrace and harmonize both approaches. In my experience it is possible to build top-down frameworks that are, underneath, comprised of re-usable components that the user can gain access to if needed. I call it the “onion” approach. With this approach the system is designed in layers, like an onion. The outer layers can be used contextually, like a framework. However, the system is designed such that any layer can be peeled away to reveal the one underneath. As layers are peeled back, the end-user is able to transition to using composition, allowing even the most unanticipated use-cases to be addressed. At no point is the user unable to dig underneath the abstractions they are presented with.

It is hard to design systems like this, but it can be done. When done properly these systems offer the get-going-quickly-dont-bother-me-with-the-details usage model of the contextual approach with the flexibility and power of the compositional approach if needed.

Will

Comments are closed.