For instance, application design principles changed dramatically when the PC, x86 architecture, and client/server paradigm were born in the ’80s. And  it happened again with the advent of the web and open-source technology in the mid ’90s. Whenever such abrupt changes arise, application developers are forced to rethink how they build and deploy their software.

Today, we’re seeing a huge leap in infrastructure capability, this time pioneered by Amazon Web Services. It’s clear that to take full advantage of the new cloud infrastructure, applications that run successfully on AWS must be inherently different than applications that were built to run successfully on a corporate server – even a virtualized one. But there are a number of other particular ways in which today’s (and tomorrow’s) cloud applications will need to be designed differently than in the past. Here are the most crucial ones, and how the ways of the old world have been changed in the new one :

Scaling 

In the old world, scaling was accomplished by scaling up – to accommodate more users or data, you simply bought a bigger server.

In the new world, scaling is typically done by scaling out. You don’t add a bigger machine, you add multiple machines of the same sort. In the cloud world, those machines are virtual machines, and their instantiations in the cloud are instances.

Resilience 

Before, software was seen as unreliable, and resilience was built into the hardware layer.

Today, the underlying infrastructure – the hardware – is seen as the weak link, and it is up to applications to accommodate for this. There is no guarantee that a virtual machine instance will always function. It can disappear at any moment and the application must be prepared for this.

By way of example, Netflix, arguably the most advanced user of the cloud today, has gone the farthest in adopting this new paradigm. They have a process called ChaosMonkey that randomly kills virtual machine instances from underneath the application workloads. Why on earth do they do this on purpose? Because they are ensuring uptime and resilience: By exposing their applications to random loss of instances, they force application developers to build more resilient apps. Brilliant.

Bursting

In the old world – think accounting and payroll applications – the application workload was reasonably stable and predictable. It was known how many users a system had, and how many records they were likely to process at any given moment.

In the new world, we see variable and unpredictable workloads. Today’s software systems have to reach farther out in the world, to consumers and devices that demand services at unpredictable moments and unpredictable loads. To accommodate such unforeseen fluctuations in individual application workloads required a new software architecture. We now have it in the cloud, but clearly it is still in its infancy.

Software variety

In the past we didn’t have much software variety. Each application was written in one language and used one database. Companies standardized on a single, or at least very few operating systems. The software stack was boringly simple and uniform (at least now in retrospect).

In the new world of cloud, the opposite is happening. Within a single application, many different languages can be used, many different libraries and toolkits can be employed, and many different database products can be used. And because in a cloud you can create and spin up your own image, tailored to your and your application’s specific needs, applications within one company must be able to operate under a spectrum of configurations.

From VM to cloud 

Even between the relatively new technology of hypervisors and the modern cloud thinking, there are differences. VMware, the pioneer and leader in virtualization, built its hypervisors to essentially behave the way physical machines did before.

But in the cloud world, the virtual machine is not a representation of a physical server; it’s a representation of units of compute. (Steve Bradshaw wrote about this topic in depth.)

User patience

In the old world, users were taught to be patient. The system may have needed a long time to respond to simple retrieval or update requests, and new features were added slowly to the application (if at all).

In the new cloud world, users have no patience. They hardly tolerate latency or wait times, and they look for improvements in the service every week, if not every day. Evidence of this can be found in self-service IT. Rather than file a ticket with IT and wait for a response several days later, users of IT can self-provision the resources they need.

Do these observations rhyme with what you are experiencing and taking action on in your organization? I look forward to comments and debate on this topic.

Marten Mickos is the CEO of Eucalyptus Systems. He previously served as CEO of MySQL AB, which was acquired by Sun Microsystems. He is a member of the board of directors of Nokia.

Have an idea for a post you’d like to contribute to GigaOm? Click here for our guidelines and contact info.

Photo courtesy of Mike Flippo/Shutterstock.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• What Enterprise Software Vendors Could Learn from the Consumer Space
• Continuous delivery and the world of devops

These two patterns are described well in this recent post from Neal Ford, self-described “Director, Software Architect, and Meme Wrangler” at systems integrator ThoughtWorks:

In my keynote, I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive…The knowledge and effort required for a seemingly trivial change prevents the change from occurring, leaving the developer with a perpetually dull tool. Contextual tools aren’t bad things at all – Eclipse and IntelliJ wouldn’t exist without that approach. Contextual tools provide a huge amount of infrastructure that developers don’t have to build. Once mastered, the intricacies of Eclipse’s API provide access to enormous encapsulated power…and there’s the rub: how encapsulated?

In the late 1990’s, 4GLs were all the rage, and they exemplified the contextual approach. The built the context into the language itself: dBASE, FoxPro, Clipper, Paradox, PowerBuilder, Microsoft Access, and similar ilk all had database-inspired facilities directly in the language and tooling. Ultimately, 4GLs fell from grace because of Dietzler’s Law, which I defined in my book Productive Programmer, based on experiences by my colleague Terry Dietzler, who ran the Access projects for my employer at the time:

---

Dietzler’s Law for Access

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.

---

Ultimately Dietzler’s Law killed the market for 4GLs. While they made it easy to build simple things fast, they didn’t scale to meet the demands of the real world. We all returned to general purpose languages.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things. A famous story from 1992 illustrates just how powerful these abstractions are. Donald Knuth was asked to write a program to solve this text handling problem: read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. He wrote a program consisting of more than ten pages of Pascal, designing (and documenting) a new algorithm along the way. Then, Doug McIlroy demonstrated a shell script that would easily fit within a Twitter post that solved the problem more simply, elegantly, and understandably (if you understand shell commands):

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q

I suspect that even the designers of Unix shells are often surprised at the inventive uses developers have wrought with their simple but powerfully composable abstractions.

Ford goes on to describe the pros and cons of each approach in much more detail, but the key conclusion he reaches is, I think, critical to understanding how one should develop the tools and tool chains that drive new IT models:

These abstractions apply to tools and frameworks as well, particularly tools that must scale in their power and sophistication along with projects, like build tools. By hard-won lesson,composable build tools scale (in time, complexity, and usefulness) better than contextual ones. Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible, Dietzler’s Law Redux. This is especially true in tools where critical parts of how they function, like the ordering of tasks, is inaccessible without hacking.

Ford’s distinction is one that finally helps me articulate a key concern I’ve had with respect to Platform-as-a-Service tools for some time now. In my mind, there are primarily two classes of PaaS systems on the market today (now articulated in Ford’s terms). One class is contextual PaaS systems, in which a coding framework is provided, and code built to that framework will gain all of the benefits of the PaaS with little or no special configuration or custom automation. The other is composable PaaS, in which the majority of benefits of the PaaS are delivered as components (including operational automation) that can be assembled as needed to support different applications.

Contextual PaaS

Examples of contextual PaaS include the original releases of Google App Engine, Heroku and other “first-generation” PaaS systems that asked the developer to adhere to specific architecture and consume PaaS-specific classes in the application itself. These systems were incredibly powerful for building applications that were variations of what these frameworks were designed to do, but began to fail quickly for applications that fell outside of that domain.

The classic example is Google App Engine’s limit of 30 seconds for any backend request to complete. Great if you were building a Facebook game, but a requirement that eliminated its use for many multi-step transactional applications. Of course, there were ways to deal with those situations, as well, but they were mostly complicated and added risk to the system.

There is a parallel here with the 4GLs of the late 1990s that Ford talks about in his post. At that time, I worked for Forte Software (acquired by Sun Microsystems in 1999), which built a 4GL development and operations environment for distributed application development. We had a business model where we relied heavily on systems integrator partners to help our customers deliver these often sophisticated applications, and every one of those SIs eventually built a framework environment to make building complex applications “easier.”

The problem? Almost every customer that used one of these frameworks had a requirement (or many) that the framework didn’t handle well. This resulted in either the SIs scrambling to modify their frameworks to support these requirements — inevitably resulting in the framework being much less “easy” to use — or the customer bypassing the framework all together for those needs, resulting in an application that was harder to debug and operate.

Composable PaaS

Composable PaaS systems, on the other had, do much less to anticipate the architecture or functionality of the application built on it, and do much more to simplify the assembly of services, including underlying infrastructure, automation, data sources, specialized data tools, etc. I think the classic example of a composable PaaS is Cloud Foundry, the open source PaaS effort from VMware that’s now part of its Pivotal Initiative spinoff. Modern versions of Heroku, Engine Yard, CloudBees and other also exhibit more of this approach than “first-generation” PaaS systems.

An old, but illustrative, Cloud Foundry diagram.

Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).

Now, take the concept a step further. Imagine a deployment environment that delivers a wide variety of these individual tools and components and simplifies the process of creating tool chains on demand from them. Imagine that environment would let each development team choose from known tool chain “patterns,” but modify them as they see fit for each project. This, I believe, will be the ultimate general purpose PaaS success, not some hard-and-fast framework-based PaaS.

The concept of composable and contextual applies to a lot more than PaaS and cloud, of course. And it is important to note that it’s not an either/or choice, much like stability and resiliency. Parts of an IT environment should be composable, but there will always be elements where the relative stability of contextual extension makes more sense. And composable systems can leverage API-driven systems that themselves are designed primarily for extensibility via contextual approaches.

The key is to think about each system from the perspective of how it will be used, and to target its extensibility mechanism based on needs. Just remember, however, that choosing a contextual path will dictate a lot more about how your system could be used in the future than a composable approach would.

I’d love to hear your thoughts, either in the comments below, or on Twitter, where I am @jamesurquhart.

Feature image courtesy of Shutterstock user Nenov Brothers Photography.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• PaaS market accelerators, 2012–2013
• Cloud computing infrastructure: 2012 and beyond
• Platform as a Service in 2012

BitDeli is targeting small to medium-sized businesses that need something between the easy-but-limited Google Analytics on one end of the spectrum, and a full-fledged Hadoop cluster on the other. According to Co-founder and CEO Ville Tuulos, many companies start small with some simple web analytics, but eventually realize they need something more. Especially if that something more involves tracking custom metrics unique to their business (e.g., how many Mac users are buying T-shirts or how many people are buying products within social games), he said, the natural instinct is to do what they see everyone else doing and try deploying Hadoop or maybe a MongoDB database.

The problems with that approach are pretty easy to spot: Deploying and managing those types of systems can be difficult, and they’re not particularly helpful if you’re trying to track your data in real time. BitDeli fills the gap, Tuulos explained, by letting developers easily set up custom analytic functions using Python scripts. They can be as simple or complex as necesary — even incorporating techniques such as machine learning — and they’re consumable by the entire company via a simple collection of dashboards called “cards.”

The service, which is hosted on the Amazon Web Services cloud, also keeps a database for each user so they can analyze historical data as well as the streaming data that BitDeli really focuses on.

Tuulos compares what BitDeli is trying to do to how frameworks such as Rails democratized web development earlier this century. Development used to be a heavy and expensive process, but open, lightweight frameworks made it so “10 times more developers can do things,” he said. He analogizes Hadoop and MongoDB to those first-generation development methods, while BitDeli is the lightweight new tool for the masses.

Sure, Tuulos acknowledges, there are workloads for which Hadoop is definitely still the right solution, but if you were to start thinking today about what most companies need to fulfill their analytic requirements, “you don’t come up with Hadoop,” he said. Tuulos ought to know — his previous gig was working on Nokia’s data team alongside its petabyte-scale Hadoop cluster. But, he said, when the data team are the only guys who know how to interact with the system in order to run jobs, that team becomes a bottleneck.

With BiDeli, he explained, scripts can be written and analyzing data in production in minutes, and developers can easily iterate on them when they need to be tweaked.

Whatever tools people choose to use, though, Tuulos’s analogy to web development is hard to argue against. Other companies, including Structure 2012 Launchpad winner Keen.io, are doing very similar things for mobile analytics, and even developer-focused Hadoop frameworks such as Continuuity and Mortar Data are trying to make it fast and easy to analyze big data. Analytics has to become more of a continuous development situation than a waterfall situation, Tuulos said, and “you have to increase the number of people capable of doing these things themselves.”

Feature image courtesy of Flickr user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

When I sat down with Amazon CTO Werner Vogels at the Amazon Web Services re:Invent conference on Wednesday, we began the discussion by talking about applications designed to take advantage of everything the cloud has to offer in terms of control, resiliency and programmability — what Vogels calls 21st-century architectures. It’s great in theory but, I asked, “Who’s actually building these apps?”

Mainly, I was concerned with whether AWS has been able to prod the enterprise customers it so desperately desires into adopting Vogels’s design principles for their applications. Often times, any discussion about “enterprise cloud computing” begins and ends with whether they can run their legacy SAP applications on cloud servers.

Vogels acknowledged there are plenty of reasons to focus on those legacy applications, including the decreased costs and increased flexibility the cloud can bring. He pointed to human resources applications as one of the first that moved to the cloud, citing Amazon’s own internal employee-review process that requires some 60,000 employees to file performance reviews on March 1 every year. He calls these “IT life events” that get companies thinking seriously about whether it’s worth their time and money to invest in more hardware.

Scaling from 1 server to 200 servers for just a week or so is a way to achieve serious return on investment “without radically innovating in terms of software,” Vogels said. However, while that’s a topic rife with financial implications, it doesn’t address the spirit of cloud-enabled innovation that companies such as AWS tout so loudly.

Werner Vogels (left) and Om Malik at Structure 2012
(c)2012 Pinar Ozger pinar@pinarozger.com

Just consider what’s possible. Anyone who follows cloud computing is familiar with how Netflix has a built a service-oriented, highly dynamic cloud infrastructure atop AWS that’s built to withstand whatever service disruptions come its way. Vogels and I also discussed Pinterest and its cloud system, which is designed to maximize cost savings by relying heavily on Reserved Instances for the base load and then programmatically favoring Spot Instances over On-Demand Instances when more resources are required. (“These guys have turned [cloud pricing] completely on its head,” Vogels said.)

However, he added, although AWS uses startups to illustrate cool architectures, “we also could invite Samsung on stage.” Samsung’s smart TVs get their intelligence from software, and all that runs in the AWS cloud. “That architecture is not an 19th-century architecture,” Vogels said. “That architecture is a 21st-century architecture.”

Or take media companies, where Vogels said many traditionally print media companies are looking for new ways to monetize their content. The United Kingdom’s Telegraph used AWS to power an application that makes it easy for readers to buy what they see on the website’s fashion section, he explained. That attracted more advertising dollars, he said, and could be expanded into coverage about cars, music and other consumer-centric content.

ABC’s mobile app for watching live content splits uploads into parallel streams, does live transcoding and automatically inserts advertisements based on where viewers are geographically located at any given time. Other media companies are using advanced analytics running in AWS’s cloud to figure out what content viewers want to read and watch on their mobile devices, and how they want to consume it.

Shell Oil is using the cloud for continuous deployment and testing of applications, and pharmaceutical companies are for the first time ever doing true collaborative research thanks to the cloud. All sorts of smart devices use the cloud as the backend for storing and processing data.

“Are these old-style IT architectures?” Vogels asked. “No.”

They are, however, new applications that aren’t tied to lucrative enterprise software markets built over decades. Rather than the mission-critical applications that many cloud providers — including AWS — are trying to convince enterprise users to migrate to the cloud, these new applications are relatively inexpensive to build and run. And, Vogels, noted, if these applications don’t work, companies can pull the plug on them with relatively minimal business impact.

He sees more companies coming around on this type of innovation through applications. “Maybe sometimes experimentation is a hard word, it’s like doing research or science,” Vogels said, but when companies are facing increasing consumer choices and decreasing customer loyalty, they have to be agile and figure out what works in order to keep their businesses growing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The fourth quarter of 2012 in cloud
• How direct-access solutions can speed up cloud adoption

The goal of the framework, says Mortar Data co-founder and CEO K Young, is not just to make application development easier, but also to make it easier to share cool datasets. Inspired by Rails, Mortar is a way to write jobs that process data with Hadoop using pipelines, similar in theory to Cascading. Developers can do all their work within a command line interface, including testing jobs before they run. Mortar supports programming in Python and Ruby, and still uses Apache Pig as the workflow language.

The real beauty, though, might be in how easy Mortar makes it to share datasets. Once people find cool public datasets from cities, governments or other sources, they can they analyze them using the Mortar Data service and then share the code on Github. As long as the dataset is now stored in Amazon Web Services — in either S3 or, now, in MongoDB on EC2 instances — anyone cloning the code from Github will automatically be able to connect to it.

“We’re trying to take lessons from app development … and bring that to working with data,” Young said. He wants Mortar to make working with Hadoop a repeatable, collaborative experience.

Of course, Young noted, this has utility beyond just weekend hackers working on public datasets. Corporate development teams that might have silos even within AWS can easily share their data and workflows with each other, too. As he did when I first spoke with him in April, Young acknowledged Mortar Data might have to extend its service and the new framework to work with on-premises infrastructure and data, but that’s still not in the works right now.

Apart from potentially moving into the data center, Young said the Mortar framework will likely expand beyond batch processing and MapReduce as alternative Hadoop use cases begin to emerge. “Who really knows what Hadoop is?” he asked, referencing YARN, Impala, graph databases and other alternative processing methods built atop Hadoop (some of which have frameworks of their own). Now, however, he thinks there’s a still enough business in batch processing with MapReduce around which to build a business.

Although it’s a bit premature to begin making predictions about which Hadoop tools developers will flock to — we’ve covered platforms such as Infochimps and Continuuity before, and more certainly will emerge — but Young is confident Mortar can grow a big community around it a la SpringSource in the Java application space. After all, despite spending the better part of its existence thus far intentionally holding back on marketing and support in order to focus on engineering, Young said developer uptake has nonetheless been good.

Mortar Data just became an advanced tier AWS partner, he noted, “which means we’re spending a crapload of money on Amazon.”

Feature image courtesy of Shutterstock user isak55.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Netflix Engineer Ben Christensen explained Hystrix thusly in a blog post on Monday:

Hystrix is a library designed to control the interactions between these distributed services providing greater tolerance of latency and failure. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve the system’s overall resiliency.

Hystrix actually stems from earlier work to add resilience to the Netflix API, the means by which many customer-facing applications access the services they need to run. As Christensen explained in a February 2012 blog post, services are distributed across thousands of instances in AWS, and if there are problems with those services — such as high latency or failed connections between them — it can wreak havoc on the Netflix API and seriously affect the performance of all the applications that depend on it.

Source: Netflix (https://speakerdeck.com/benjchristensen/performance-and-fault-tolerance-for-the-netflix-api-august-2012)

And, as he notes in that February post, “Intermittent failure is guaranteed with this many variables, even if every dependency itself has excellent availability and uptime … Thus, it is a requirement of high volume, high availability applications to build fault tolerance into their architecture and not expect infrastructure to solve it for them.”

Distributed systems are hard work to build and manage — ask anyone at Yahoo, Google or Facebook — and building distributed, service-oriented applications on top of those systems is probably no less difficult. Netflix has an even more-novel challenge because it opted to host all of its applications and services in the cloud, which provides some great tools for maximizing uptime but also some new layers of complexity in application architecture. The company’s focus on building resilient apps has been core to its ability to survive most of AWS’s cloud outages with little or no significant downtime.

In fact, Obama for America CTO Harper Reed told me during a post-election interview that Netflix tools and techniques helped the president’s AWS-hosted applications stay up and running even during three outages between late June and Nov. 6. Netflix has also open-sourced its Eureka load-balancing technology, its Edda dynamic querying tool, its Asgard management console and its lauded Chaos Monkey for testing application resilience.

It’s no surprise then that Netflix is something of a shining star at the AWS: Reinvent user conference in Las Vegas this week (CEO Reed Hastings will take the stage long with numerous engineers), prompting some to refer to it jokingly as a Netflix technology conference. Not that it’s an insult to anybody — Amazon and other infrastructure-as-a-service providers rent virtual servers, networks and management tools, but it takes cutting-edge users to engineer apps that can make the most of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud computing infrastructure: 2012 and beyond
• Amazon’s DynamoDB: rattling the cloud market

The idea behind the company is that developers and data scientists without huge budgets and/or mad Java skills should be able to take advantage of Hadoop both cheaply and easily. Mortar Data’s trick is a custom blend of the Pig and Python programming languages that makes job creation a snap, coupled with a cloud computing backend that means users don’t even have to own a server.

However, because it’s hosted on the Amazon Web Services cloud, Mortar Data is still best-suited to customers already storing data in Amazon’s S3 storage service. The good news for the company at this early stage is there are plenty of developers doing just that. It’s also good news for the developer community that Mortar recognizes this limitation and is working to expand the number of data sources and repositories it supports, as well as the means of visualizing the results of users’ Hadoop jobs.

In the meantime, you can see some of what Mortar is up to on its Github page, which includes the development framework (so developers can write jobs locally), a MongoDB adapter, an API client for Ruby and a couple of sample datasets. And, Co-Founder and CEO K Young has promised me that we can expect to see more news out of Mortar Data later this month as the company begins ramping up its efforts publicly.

Here’s a video of demonstrating the service in action:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The importance of putting the U and I in visualization
• A near-term outlook for big data

As WibiData product manager Devjit Chakravarti told me during a recent call, KijiSchema essentially “takes the ‘No’ out of NoSQL.” What he means is that although NoSQL databases such as HBase are lauded in part because they can store unstructured data and don’t require rigid rules for data formatting like relational databases do, having some structure is actually necessary once you want to do meaningful analysis on it. That’s why some commercial products, such as Drawn to Scale’s Spire and Splice Machine’s Splice SQL Engine, already have built functional SQL databases on top of HBase.

Kimball speaking at Structure: Data in 2012
(c) 2012 Pinar Ozger. pinar@pinarozger.com

“If you can’t store data in an organized way, you can’t analyze it effectively,” WibiData Co-Founder and CTO Aaron Kimball explained. KijiSchema isn’t part of WibiData’s secret sauce around predictive analytics for user data, he added, but nothing gets done without it.

Here’s how Kimball describes how KijiSchema manages data in a blog post announcing the project:

“KijiSchema gives developers the ability to easily store both structured and unstructured data within HBase using Avro serialization. It supports a variety of rich schema features, including complex, compound data types, HBase column key and time-series indexing, as well cell-level evolving schemas that dynamically encode version information.

“KijiSchema promotes the use of entity-centric data modeling, where all information about a given entity (user, mobile device, ad, product, etc.), including dimensional and transaction data, is encoded within the same row. This approach is particularly valuable for user-based analytics such as targeting, recommendations, and personalization.”

Kiji resides in the lower left section

The coolest part for HBase developers or prospective HBase developers, however, might be that KijiSchema isn’t just code but is already pre-packaged any ready to deploy. WibiData has created what it calls the Kiji BentoBox — “a fully-functional HBase mini-cluster with KijiSchema on your machine with minimal configuration in under 15 minutes” — that’s available for download on Github.

KijiSchema is also part of a broader Kiji framework for HBase that WibiData plans to open source over the next year or so. People perceive HBase as being complicated to set up and having a steep learning curve, Kimball said, and his teams wants to make it more accessible and lower the barrier for getting started. The ultimate goal is to make the types of HBase applications that folks at Facebook, eBay and other large web shops are building something that any developer can do.

WibiData’s Omer Trajman, formerly VP of technology solutions at Cloudera, describes the ultimate Kiji framework as being akin what the Spring framework if for Java. Despite its complexity, “there are also tens of thousands of developers who have been able to figure [HBase] out,” he said, but learning it might take weeks of intensive training on learning the low-level guts of the Hadoop Distributed File System and other stuff. Why learn to build an enterprise Java application from scratch, Trajman asked, when you can just use Spring?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• How search can unlock the power of big data

A presidential campaign is not a tech startup; it has to innovate on tight deadlines and in an environment where failure really is not an option. So although it had to move fast, for example, Reed’s team couldn’t afford to re-invent the wheel because it could maybe shave 5 milliseconds off of page-load time for a web page.

Or, as Reed put it a phone call with me on Monday morning: “Our goal [was] to be the force multiplier, not to be a technology experiment.”

It was there to make sure the president’s foot soldiers — the folks who really do affect the results — could execute their ground-game without having to worry about technology failing them. Reed’s team just had to take the tools at its disposal and use them to their fullest extent so that old-world and potentially time-consuming techniques such as calling phones and knocking on doors were done as efficiently as possible.

For example, Reed explained, his team didn’t invent the tool for making and handling phone calls, but it gave that tool legs. “We made it so it could stand up and take all the calls,” he said. “Innovation was scale. Innovation was not falling down.”

Here’s how Reed and company fulfilled their duty.

Reed (center rear) and team on election day.

It made the cloud work for it …

Reed said the the vast majority of the campaign’s infrastructure was hosted on Amazon Web Services, with only the analytics platform and some “nominal” other pieces residing on physical gear. This was a fairly big change from 2008, when Obama for America was running only a few minor tasks in the cloud. A reliable cloud service is a game-changer, he said, because smart campaigns no longer have to worry about buying servers, engaging contractors or negotiating software licenses.

AWS, he said, was particularly helpful because of all the services it offers. In order to ensure visitors to President Obama’s website had the same experience wherever they were located, the team decided to “light up boxes all over [the country]” and use the CloudFront content-delivery network feature to route data from the closest virtual server. When it wanted a key-value store, it went to DynamoDB.

As for that phone-call tool and the millions of calls it has to support, Auto-Scaling, a method for adding additional resources in mere minutes, saved the day. “We used the hell out of it,” Reed said. When volume was higher than expected, he added, “we could just turn that to 11.”

… even when the cloud crashed

But relying so heavily on the cloud wasn’t always easy, because the cloud isn’t always reliable. Amazon’s cloud went down a few times between June 2012 and election day — including as late as Oct. 22 — and brought a lot of web properties down with it. “Both times we survived,” Reed said, “but it was hard.”

Obama for America stayed online because it embraced devops (the tight alignment between application development, engineering and operations that cloud computing enables) and the smart architectural strategies employed by cloud computing pioneers such as Netflix. (Paraphrasing a discussion thread from popular programming site Hacker News, Reed joked that if a developer can still use Netflix while AWS is down, then the developer screwed up. If Netflix is down, then AWS is really down.) That means spending a little more money to replicate databases and applications across geographic regions and generally being smarter about where the various application components run and how they’re connected to each other.

AWS gives you tools to do amazing things, he explained, “but you’re responsible for screwing it up.” At the beginning, he added, everything might be chill and people are working on how to creatively architect a reliable cloud application, but eventually “you start valuing uptime over experimentation.” At that point, you just have to accept the gravity of your mission and say “we’re going all in.”

Reed thinks keeping applications up in the cloud will get easier thanks to some of the innovative companies and technologies working in that space. He pointed to cloud-management company RightScale, the Tungsten Replicator engine for MySQL and the general platform-as-a-service space as things he thinks are really cool but weren’t quite ready for primetime in 2012 — at least for something as deadline- and mission-critical as a presidential campaign.

It understood how the web works

Aside the from finer points of cloud computing infrastructure, Reed said Obama’s tech team also really understood how the web has evolved since 2008. Take, for example, the advent of Twitter and Facebook as forces to be reckoned with in terms of voter engagement. Then-candidate Obama got a lot of attention for his 5 millionish social media connections, but President Obama now has roughly 10 times that many across Facebook and Twitter. That’s a lot of people from which to spark a network effect, and a lot of data to analyze.

“Being able to work at that scale is amazing,” Reed said.

And it’s not the just the president’s follower counts that have grown — social media platforms have evolved significantly and people are far more familiar with how they work. Facebook Pages, for example, were created after 2008 in part because having profile pages for candidates and companies didn’t work out too well, and Facebook Connect didn’t exist either. Twitter has also grown into a widely popular platform, while MySpace is all but gone. Reed said it was critical that team Obama take the platforms built in the four years between elections and use them the way they are being used today.

Or take the details of how people access those sites to begin with. The president’s technology team knew (no doubt thanks in part to its analytics team that has received so much attention post-election for its data mining efforts) that voters in urban areas where Obama was counting on high voter turnout were more likely to use a mobile phone rather than a laptop as their primary means of internet access. They’re also more likely to use Android devices than iOS devices, as are many potential swing voters who generally don’t care too much about technology. So, Reed said, the team designed apps to run on multiple operating systems and used responsive design to ensure apps ran well on whatever devices voters were using.

It disrupted with data

Reed’s team worked closely with that vaunted analytics team, he said, and what the Obama for America team really did better than Mitt Romney’s team was disrupt the status quo with regard to how it used data. He’s not too keen on jumping on the bandwagon calling out the failings of Romney’s Project Orca (“I never would wish technology failing on any sort of opponent or enemy,” Reed said) but he will acknowledge that the analytics team Obama had been putting together for more than a year leading into the election was a major differnentiator.

He compares the advanced modeling and analytic techniques of his comrades and guys like Nate Silver to MP3s, thus making traditional pollsters and political “experts” akin to the music industry. They both had been going about their business for decades without competition, and they both reacted violently when their worlds were disrupted. However, these are smart people, and Reed expects they’ll come around in the next election cycles.

But for now, he said, “That’s a hard position to be in when you’re no longer relevant.”

It had a great leader

In the end, though, Reed thinks the success of Obama’s campaign all boiled down to having a good leader. It wasn’t just Obama’s good fortune of running in during the era of Twitter and Amazon Web Services that let him forever change the way campaigns are run. The president is the one who set the tone about how the campaign would function and what it would focus on, Reed said, and the president put in place the campaign leaders who followed through on his vision by hiring the right people down the chain.

If Reed does come back to politics, it will take another candidate with Obama’s vision and ability to generate excitement among the populace to get him back in the saddle. But for now, Reed said, “I’m definitely ready for a little break.”

Feature and Obama for America photo courtesy of Harper Reed.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• How direct-access solutions can speed up cloud adoption
• Takeaways from the second quarter in cloud and data

MongoLab appears to have figured out the formula. According to Founder and CEO Will Shulman, the company, a cloud computing startup offering fully managed versions of the MongoDB NoSQL database, has tens of thousands of customers since launching in 2011 and is growing between 10 percent and 30 percent a month. On Wednesday, it became the first MongoDB service available in Windows Azure Store, a marketplace for add-ons that complement what Microsoft itself offers on the Windows Azure cloud platform.

MongoLab also just closed a Series B funding round of $5 million, bringing its two-round total to $8 million. The money came from existing investors Foundry Group and Baseline Ventures, as well as GRP Partners, Freestyle Capital and David Cohen (of TechStars).

Go where your users are

The secret to the company’s early success is camping out where the programming tidal waves of MongoDB and Platforms as a Service converge, and then managing the impact in the name of developers. “We really take away a ton of headache for teams just wanting to develop their app,” Shulman said.

The way MongoLab works is simple enough. Through its website, developers can choose the architecture they want — from a small free offering on shared infrastructure to a clustered database on dedicated infrastructure — as well as the cloud on which they want it to run. Alternatively, developers can deploy a MongoLab database via their favorite PaaS providers (including Heroku, AppFog and AppHarbor). Once deployed, the idea behind MongoLab is like all things PaaS: developers worry about applications, and MongoLab worries about keeping the database up and running.

The rationale for building a PaaS is easy enough to understand, but there are some who’d argue MongoDB is a tougher sell. Although it’s wildly popular among developers (just look at the pile of venture capital MongoDB creator — and close MongoLab ally — 10gen has raised), the product does have its share of detractors. But whether those criticisms are fair or not, Shulman thinks the key to understanding when and how to use MongoDB, and why so many people choose to do so, is understanding how MongoDB fits into most programmers’ tool belts.

It’s all about objects

Although the NoSQL movement “rode in under this banner of scalability,” he explained, choosing a NoSQL option based on how it scales can actually be a false choice. MongoLab operates under a thesis that MongoDB is pulling away as the world’s most-popular NoSQL database not because it scales the best — it does scale, Shulman said, but he’d actually choose Cassandra if he just needed a multi-petabyte data store without much concern over queries or data structure — but because web developers are moving away from the relational format to an object-oriented format.

He explains the idea more fully in an August blog post on the topic, but the general idea is that document databases such as MongoDB let users store JSON objects and other objects without forcing them to conform to a relational table format. These objects, or other things, actually look very much like the objects in your code, Shulman said, and “[developers] have an emotional love of MongoDB because of this.”

If it works, developers will come

After picking the right business (MongoDB) and the right delivery model (PaaS), the next step to winning developers hearts, minds and dollars is packaging it all in a way that appeals to them. That’s why MongoLab plans to spend its new venture funding like it generally spends its cash — on good engineers and building a good product. Shulman said the company doesn’t do a lot of marketing because, to its audience, good tools and good documentation will make sure word spreads just fine.

And that’s good, because like other recent developer-centric startups, he also doesn’t want to hire a sales team: “Our audience doesn’t like that too much.”

Feature image courtesy of Flickr user junyaogura.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How devops can reduce cycle times
• Takeaways from the second quarter in cloud and data