They were app developers; the transformation into systems engineers was just a necessity of growing the business. A website that looked pretty and worked smoothly was little more than veneer if the real product — the AWS virtual servers running Heroku’s customers’ applications under the covers of the hip samurai-themed web service — didn’t work. At one point, Lindenbaum joked during a recent call, Heroku had to do a UX refresh “and we literally didn’t have anyone in the company who could build web apps anymore.”

After leaving Heroku and ultimately its parent company Salesforce.com , Lindenbaum is back in the public eye with a new effort called Heavybit Industries that aims to save other startups from Heroku’s early growing pains. The idea came after so many fledgling companies came to Heroku looking for help, and after Lindenbaum got personally involved with some as an adviser or investor. He eventually realized that there’s a lot of institutional knowledge out there about how to build business that serve developers, but there’s also a lot of duplicated effort because the people starting these businesses often don’t know their peers exist, much less what they’re up to.

How Heavybit works

Heavybit is like a co-working-space-meets-incubator-meets fraternity, and Lindenbaum has recruited some of the biggest names in venture capital, application development and developer-focused startups to make sure Heavybit delivers on its promise. It works like this: Companies that have raised some money, gained some traction among developers, and now have to deal with the difficult problems of scaling their code or monetizing their business come to Heavybit. Once accepted, they’re in the “active period” for nine months, which includes a curriculum of weekly talks on technology or entrepreneurship; office hours with experts and investors; and meeting/work space in a building in San Francisco’s SOMA neighborhood.

James Lindenbaum at Structure 2009. (c) Pinar Ozger

Membership in Heavybit is in exchange for equity, and the community has space for between 10 to 15 companies at a time, Lindenbaum said (although it’s really a “membership for life” situation). The first batch of startups includes some more-established ones — Zencoder, Stripe, PagerDuty and Meteor — that will serve as mentors as well as receive mentorship. Other inaugural Heavybit members include Treasure Data, Kodowa, Iron.io, CircleCi, CloudConnect, Keen IO, Codenvy and Backlift.

Early Heavybit expert advisers include Derek Collison (Apcera/VMware/Google), Adam Gross (CloudConnect/Salesforce.com), Jesse Robbins (Opscode/Amazon), Javier Soltero (Acompli/VMware/Hyperic), and Lindenbaum’s Heroku co-founders Adam Wiggins and Orion Henry. Among  Heavybit’s early investor partners are Ping Li (Accel Partners), Chris Sacca (Lowercase Capital), John Connors (Ignition Partners) and Matt Ocko (Data Collective).

Bringing “bad-ass” engineers back from mobile apps

One of Lindenbaum’s goals when putting together the advisers — and one of his continuing goals with Heavybit — is to to put member companies in touch with people who really understand the business and architectural complexities of distributed, multitenant applications. “The reason [these developer-focused] products are so great is because they’re built … by app developers for app developers,” Lindenbaum said. But, like the Heroku team early on, the founders aren’t always skilled in building systems designed to scale.

“The scale curve is much steeper for these companies,” he explained, because the way it usually works is these businesses attract customers who also have their own customers to serve. So rather than handling data for one company, they might be handling data for 20 of that company’s clients, as well.

In order to put his companies in touch with the best of the best of distributed systems engineers, though, Lindenbaum first has to walk those guys back from the Instagram edge. Once you prove yourself at a place like Google or Facebook, Lindenbaum said, “[E]veryone thinks you’re a bad ass. As soon as you say you’re going to build a thing, VCs line up to give you money.”

Unfortunately, he added, many of these people are chasing the past rather than future, trying to cash in their lottery tickets on building the next photo-sharing app rather than on the hairy enterprise-grade systems problems where their skills would really be valuable.

But he has a plan to bring them back to the enterprise side. He intends to push the message of how hard these problems are and how much the cloud services and developer-focused products industries are becoming analogous to traditional heavy industries in terms of the complex but mature supply chains involved.

Essentially, he wants to do the “Got Milk?” ads for the cloud services industry, educating the market so the individual companies don’t have to. It’ll be like a trade association, he joked, “only the non-evil version of that.” Once you shine a light on the difficulty of the problems, Lindenbaum said, the really good engineers come running back to solve them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How direct-access solutions can speed up cloud adoption
• Cloud and data third-quarter 2012
• Examining open hybrid cloud options for the enterprise

Google announced the new service on its Android Developers and Cloud Platform blogs, describing it as “a one-click deployable, complete mobile backend that allows you to reap the benefits of a cloud backend with none of the headaches. It provides a ready-to-deploy, general purpose cloud backend and a general purpose client-side framework for Android.”

Google’s new service is hardly the first on the market, though, and it’s nowhere near the broadest. The market for providing what some call “mobile backend as a service” has been growing in popularity over the past couple years as smartphone use has proliferated. The classic infrastructure-as-a-service and even platform-as-a-service clouds have been a godsend to many web and early mobile startups, but many mobile developers just want something even simpler, it appears.

Mobile Backend Starter joins the aforementioned Parse, as well as other offerings from startups such as Appcelerator, Kinvey, Flurry and StackMob. All of these services offer broader feature sets and support for numerous mobile platforms beyond Android. Google’s prevailing thinking seems to be, though, that if you’re developing apps for Android, there’s no better place to do it than with the company who created the operating system.

In fact, Google is just the latest of bigger, badder vendors getting into the MbaaS space: Rackspace and Salesforce.com did so earlier this year and, if Amazon Web Services job postings are to be believed, the king of public cloud is also eyeing entry into this space.

Technically, Google unveiled Mobile Backend Starter during a talk at Google I/O last month, but apparently word didn’t spread much beyond the walls of that room. David Chandler, one of the Google developers who gave the presentation, has collected a number of additional resources here, on his personal blog.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• PaaS market accelerators, 2012–2013
• Cloud computing infrastructure: 2012 and beyond
• Survey: How apps can solve photo management

For instance, application design principles changed dramatically when the PC, x86 architecture, and client/server paradigm were born in the ’80s. And  it happened again with the advent of the web and open-source technology in the mid ’90s. Whenever such abrupt changes arise, application developers are forced to rethink how they build and deploy their software.

Today, we’re seeing a huge leap in infrastructure capability, this time pioneered by Amazon Web Services. It’s clear that to take full advantage of the new cloud infrastructure, applications that run successfully on AWS must be inherently different than applications that were built to run successfully on a corporate server – even a virtualized one. But there are a number of other particular ways in which today’s (and tomorrow’s) cloud applications will need to be designed differently than in the past. Here are the most crucial ones, and how the ways of the old world have been changed in the new one :

Scaling 

In the old world, scaling was accomplished by scaling up – to accommodate more users or data, you simply bought a bigger server.

In the new world, scaling is typically done by scaling out. You don’t add a bigger machine, you add multiple machines of the same sort. In the cloud world, those machines are virtual machines, and their instantiations in the cloud are instances.

Resilience 

Before, software was seen as unreliable, and resilience was built into the hardware layer.

Today, the underlying infrastructure – the hardware – is seen as the weak link, and it is up to applications to accommodate for this. There is no guarantee that a virtual machine instance will always function. It can disappear at any moment and the application must be prepared for this.

By way of example, Netflix, arguably the most advanced user of the cloud today, has gone the farthest in adopting this new paradigm. They have a process called ChaosMonkey that randomly kills virtual machine instances from underneath the application workloads. Why on earth do they do this on purpose? Because they are ensuring uptime and resilience: By exposing their applications to random loss of instances, they force application developers to build more resilient apps. Brilliant.

Bursting

In the old world – think accounting and payroll applications – the application workload was reasonably stable and predictable. It was known how many users a system had, and how many records they were likely to process at any given moment.

In the new world, we see variable and unpredictable workloads. Today’s software systems have to reach farther out in the world, to consumers and devices that demand services at unpredictable moments and unpredictable loads. To accommodate such unforeseen fluctuations in individual application workloads required a new software architecture. We now have it in the cloud, but clearly it is still in its infancy.

Software variety

In the past we didn’t have much software variety. Each application was written in one language and used one database. Companies standardized on a single, or at least very few operating systems. The software stack was boringly simple and uniform (at least now in retrospect).

In the new world of cloud, the opposite is happening. Within a single application, many different languages can be used, many different libraries and toolkits can be employed, and many different database products can be used. And because in a cloud you can create and spin up your own image, tailored to your and your application’s specific needs, applications within one company must be able to operate under a spectrum of configurations.

From VM to cloud 

Even between the relatively new technology of hypervisors and the modern cloud thinking, there are differences. VMware, the pioneer and leader in virtualization, built its hypervisors to essentially behave the way physical machines did before.

But in the cloud world, the virtual machine is not a representation of a physical server; it’s a representation of units of compute. (Steve Bradshaw wrote about this topic in depth.)

User patience

In the old world, users were taught to be patient. The system may have needed a long time to respond to simple retrieval or update requests, and new features were added slowly to the application (if at all).

In the new cloud world, users have no patience. They hardly tolerate latency or wait times, and they look for improvements in the service every week, if not every day. Evidence of this can be found in self-service IT. Rather than file a ticket with IT and wait for a response several days later, users of IT can self-provision the resources they need.

Do these observations rhyme with what you are experiencing and taking action on in your organization? I look forward to comments and debate on this topic.

Marten Mickos is the CEO of Eucalyptus Systems. He previously served as CEO of MySQL AB, which was acquired by Sun Microsystems. He is a member of the board of directors of Nokia.

Have an idea for a post you’d like to contribute to GigaOm? Click here for our guidelines and contact info.

Photo courtesy of Mike Flippo/Shutterstock.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• What Enterprise Software Vendors Could Learn from the Consumer Space
• Sector RoadMap: Social customer service in 2013

These two patterns are described well in this recent post from Neal Ford, self-described “Director, Software Architect, and Meme Wrangler” at systems integrator ThoughtWorks:

In my keynote, I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive…The knowledge and effort required for a seemingly trivial change prevents the change from occurring, leaving the developer with a perpetually dull tool. Contextual tools aren’t bad things at all – Eclipse and IntelliJ wouldn’t exist without that approach. Contextual tools provide a huge amount of infrastructure that developers don’t have to build. Once mastered, the intricacies of Eclipse’s API provide access to enormous encapsulated power…and there’s the rub: how encapsulated?

In the late 1990’s, 4GLs were all the rage, and they exemplified the contextual approach. The built the context into the language itself: dBASE, FoxPro, Clipper, Paradox, PowerBuilder, Microsoft Access, and similar ilk all had database-inspired facilities directly in the language and tooling. Ultimately, 4GLs fell from grace because of Dietzler’s Law, which I defined in my book Productive Programmer, based on experiences by my colleague Terry Dietzler, who ran the Access projects for my employer at the time:

---

Dietzler’s Law for Access

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.

---

Ultimately Dietzler’s Law killed the market for 4GLs. While they made it easy to build simple things fast, they didn’t scale to meet the demands of the real world. We all returned to general purpose languages.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things. A famous story from 1992 illustrates just how powerful these abstractions are. Donald Knuth was asked to write a program to solve this text handling problem: read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. He wrote a program consisting of more than ten pages of Pascal, designing (and documenting) a new algorithm along the way. Then, Doug McIlroy demonstrated a shell script that would easily fit within a Twitter post that solved the problem more simply, elegantly, and understandably (if you understand shell commands):

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q

I suspect that even the designers of Unix shells are often surprised at the inventive uses developers have wrought with their simple but powerfully composable abstractions.

Ford goes on to describe the pros and cons of each approach in much more detail, but the key conclusion he reaches is, I think, critical to understanding how one should develop the tools and tool chains that drive new IT models:

These abstractions apply to tools and frameworks as well, particularly tools that must scale in their power and sophistication along with projects, like build tools. By hard-won lesson,composable build tools scale (in time, complexity, and usefulness) better than contextual ones. Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible, Dietzler’s Law Redux. This is especially true in tools where critical parts of how they function, like the ordering of tasks, is inaccessible without hacking.

Ford’s distinction is one that finally helps me articulate a key concern I’ve had with respect to Platform-as-a-Service tools for some time now. In my mind, there are primarily two classes of PaaS systems on the market today (now articulated in Ford’s terms). One class is contextual PaaS systems, in which a coding framework is provided, and code built to that framework will gain all of the benefits of the PaaS with little or no special configuration or custom automation. The other is composable PaaS, in which the majority of benefits of the PaaS are delivered as components (including operational automation) that can be assembled as needed to support different applications.

Contextual PaaS

Examples of contextual PaaS include the original releases of Google App Engine, Heroku and other “first-generation” PaaS systems that asked the developer to adhere to specific architecture and consume PaaS-specific classes in the application itself. These systems were incredibly powerful for building applications that were variations of what these frameworks were designed to do, but began to fail quickly for applications that fell outside of that domain.

The classic example is Google App Engine’s limit of 30 seconds for any backend request to complete. Great if you were building a Facebook game, but a requirement that eliminated its use for many multi-step transactional applications. Of course, there were ways to deal with those situations, as well, but they were mostly complicated and added risk to the system.

There is a parallel here with the 4GLs of the late 1990s that Ford talks about in his post. At that time, I worked for Forte Software (acquired by Sun Microsystems in 1999), which built a 4GL development and operations environment for distributed application development. We had a business model where we relied heavily on systems integrator partners to help our customers deliver these often sophisticated applications, and every one of those SIs eventually built a framework environment to make building complex applications “easier.”

The problem? Almost every customer that used one of these frameworks had a requirement (or many) that the framework didn’t handle well. This resulted in either the SIs scrambling to modify their frameworks to support these requirements — inevitably resulting in the framework being much less “easy” to use — or the customer bypassing the framework all together for those needs, resulting in an application that was harder to debug and operate.

Composable PaaS

Composable PaaS systems, on the other had, do much less to anticipate the architecture or functionality of the application built on it, and do much more to simplify the assembly of services, including underlying infrastructure, automation, data sources, specialized data tools, etc. I think the classic example of a composable PaaS is Cloud Foundry, the open source PaaS effort from VMware that’s now part of its Pivotal Initiative spinoff. Modern versions of Heroku, Engine Yard, CloudBees and other also exhibit more of this approach than “first-generation” PaaS systems.

An old, but illustrative, Cloud Foundry diagram.

Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).

Now, take the concept a step further. Imagine a deployment environment that delivers a wide variety of these individual tools and components and simplifies the process of creating tool chains on demand from them. Imagine that environment would let each development team choose from known tool chain “patterns,” but modify them as they see fit for each project. This, I believe, will be the ultimate general purpose PaaS success, not some hard-and-fast framework-based PaaS.

The concept of composable and contextual applies to a lot more than PaaS and cloud, of course. And it is important to note that it’s not an either/or choice, much like stability and resiliency. Parts of an IT environment should be composable, but there will always be elements where the relative stability of contextual extension makes more sense. And composable systems can leverage API-driven systems that themselves are designed primarily for extensibility via contextual approaches.

The key is to think about each system from the perspective of how it will be used, and to target its extensibility mechanism based on needs. Just remember, however, that choosing a contextual path will dictate a lot more about how your system could be used in the future than a composable approach would.

I’d love to hear your thoughts, either in the comments below, or on Twitter, where I am @jamesurquhart.

Feature image courtesy of Shutterstock user Nenov Brothers Photography.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

BitDeli is targeting small to medium-sized businesses that need something between the easy-but-limited Google Analytics on one end of the spectrum, and a full-fledged Hadoop cluster on the other. According to Co-founder and CEO Ville Tuulos, many companies start small with some simple web analytics, but eventually realize they need something more. Especially if that something more involves tracking custom metrics unique to their business (e.g., how many Mac users are buying T-shirts or how many people are buying products within social games), he said, the natural instinct is to do what they see everyone else doing and try deploying Hadoop or maybe a MongoDB database.

The problems with that approach are pretty easy to spot: Deploying and managing those types of systems can be difficult, and they’re not particularly helpful if you’re trying to track your data in real time. BitDeli fills the gap, Tuulos explained, by letting developers easily set up custom analytic functions using Python scripts. They can be as simple or complex as necesary — even incorporating techniques such as machine learning — and they’re consumable by the entire company via a simple collection of dashboards called “cards.”

The service, which is hosted on the Amazon Web Services cloud, also keeps a database for each user so they can analyze historical data as well as the streaming data that BitDeli really focuses on.

Tuulos compares what BitDeli is trying to do to how frameworks such as Rails democratized web development earlier this century. Development used to be a heavy and expensive process, but open, lightweight frameworks made it so “10 times more developers can do things,” he said. He analogizes Hadoop and MongoDB to those first-generation development methods, while BitDeli is the lightweight new tool for the masses.

Sure, Tuulos acknowledges, there are workloads for which Hadoop is definitely still the right solution, but if you were to start thinking today about what most companies need to fulfill their analytic requirements, “you don’t come up with Hadoop,” he said. Tuulos ought to know — his previous gig was working on Nokia’s data team alongside its petabyte-scale Hadoop cluster. But, he said, when the data team are the only guys who know how to interact with the system in order to run jobs, that team becomes a bottleneck.

With BiDeli, he explained, scripts can be written and analyzing data in production in minutes, and developers can easily iterate on them when they need to be tweaked.

Whatever tools people choose to use, though, Tuulos’s analogy to web development is hard to argue against. Other companies, including Structure 2012 Launchpad winner Keen.io, are doing very similar things for mobile analytics, and even developer-focused Hadoop frameworks such as Continuuity and Mortar Data are trying to make it fast and easy to analyze big data. Analytics has to become more of a continuous development situation than a waterfall situation, Tuulos said, and “you have to increase the number of people capable of doing these things themselves.”

Feature image courtesy of Flickr user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• Dissecting the data: 5 issues for our digital future
• How to manage big data without breaking the bank

When I sat down with Amazon CTO Werner Vogels at the Amazon Web Services re:Invent conference on Wednesday, we began the discussion by talking about applications designed to take advantage of everything the cloud has to offer in terms of control, resiliency and programmability — what Vogels calls 21st-century architectures. It’s great in theory but, I asked, “Who’s actually building these apps?”

Mainly, I was concerned with whether AWS has been able to prod the enterprise customers it so desperately desires into adopting Vogels’s design principles for their applications. Often times, any discussion about “enterprise cloud computing” begins and ends with whether they can run their legacy SAP applications on cloud servers.

Vogels acknowledged there are plenty of reasons to focus on those legacy applications, including the decreased costs and increased flexibility the cloud can bring. He pointed to human resources applications as one of the first that moved to the cloud, citing Amazon’s own internal employee-review process that requires some 60,000 employees to file performance reviews on March 1 every year. He calls these “IT life events” that get companies thinking seriously about whether it’s worth their time and money to invest in more hardware.

Scaling from 1 server to 200 servers for just a week or so is a way to achieve serious return on investment “without radically innovating in terms of software,” Vogels said. However, while that’s a topic rife with financial implications, it doesn’t address the spirit of cloud-enabled innovation that companies such as AWS tout so loudly.

Werner Vogels (left) and Om Malik at Structure 2012
(c)2012 Pinar Ozger pinar@pinarozger.com

Just consider what’s possible. Anyone who follows cloud computing is familiar with how Netflix has a built a service-oriented, highly dynamic cloud infrastructure atop AWS that’s built to withstand whatever service disruptions come its way. Vogels and I also discussed Pinterest and its cloud system, which is designed to maximize cost savings by relying heavily on Reserved Instances for the base load and then programmatically favoring Spot Instances over On-Demand Instances when more resources are required. (“These guys have turned [cloud pricing] completely on its head,” Vogels said.)

However, he added, although AWS uses startups to illustrate cool architectures, “we also could invite Samsung on stage.” Samsung’s smart TVs get their intelligence from software, and all that runs in the AWS cloud. “That architecture is not an 19th-century architecture,” Vogels said. “That architecture is a 21st-century architecture.”

Or take media companies, where Vogels said many traditionally print media companies are looking for new ways to monetize their content. The United Kingdom’s Telegraph used AWS to power an application that makes it easy for readers to buy what they see on the website’s fashion section, he explained. That attracted more advertising dollars, he said, and could be expanded into coverage about cars, music and other consumer-centric content.

ABC’s mobile app for watching live content splits uploads into parallel streams, does live transcoding and automatically inserts advertisements based on where viewers are geographically located at any given time. Other media companies are using advanced analytics running in AWS’s cloud to figure out what content viewers want to read and watch on their mobile devices, and how they want to consume it.

Shell Oil is using the cloud for continuous deployment and testing of applications, and pharmaceutical companies are for the first time ever doing true collaborative research thanks to the cloud. All sorts of smart devices use the cloud as the backend for storing and processing data.

“Are these old-style IT architectures?” Vogels asked. “No.”

They are, however, new applications that aren’t tied to lucrative enterprise software markets built over decades. Rather than the mission-critical applications that many cloud providers — including AWS — are trying to convince enterprise users to migrate to the cloud, these new applications are relatively inexpensive to build and run. And, Vogels, noted, if these applications don’t work, companies can pull the plug on them with relatively minimal business impact.

He sees more companies coming around on this type of innovation through applications. “Maybe sometimes experimentation is a hard word, it’s like doing research or science,” Vogels said, but when companies are facing increasing consumer choices and decreasing customer loyalty, they have to be agile and figure out what works in order to keep their businesses growing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The fourth quarter of 2012 in cloud
• How direct-access solutions can speed up cloud adoption

The goal of the framework, says Mortar Data co-founder and CEO K Young, is not just to make application development easier, but also to make it easier to share cool datasets. Inspired by Rails, Mortar is a way to write jobs that process data with Hadoop using pipelines, similar in theory to Cascading. Developers can do all their work within a command line interface, including testing jobs before they run. Mortar supports programming in Python and Ruby, and still uses Apache Pig as the workflow language.

The real beauty, though, might be in how easy Mortar makes it to share datasets. Once people find cool public datasets from cities, governments or other sources, they can they analyze them using the Mortar Data service and then share the code on Github. As long as the dataset is now stored in Amazon Web Services — in either S3 or, now, in MongoDB on EC2 instances — anyone cloning the code from Github will automatically be able to connect to it.

“We’re trying to take lessons from app development … and bring that to working with data,” Young said. He wants Mortar to make working with Hadoop a repeatable, collaborative experience.

Of course, Young noted, this has utility beyond just weekend hackers working on public datasets. Corporate development teams that might have silos even within AWS can easily share their data and workflows with each other, too. As he did when I first spoke with him in April, Young acknowledged Mortar Data might have to extend its service and the new framework to work with on-premises infrastructure and data, but that’s still not in the works right now.

Apart from potentially moving into the data center, Young said the Mortar framework will likely expand beyond batch processing and MapReduce as alternative Hadoop use cases begin to emerge. “Who really knows what Hadoop is?” he asked, referencing YARN, Impala, graph databases and other alternative processing methods built atop Hadoop (some of which have frameworks of their own). Now, however, he thinks there’s a still enough business in batch processing with MapReduce around which to build a business.

Although it’s a bit premature to begin making predictions about which Hadoop tools developers will flock to — we’ve covered platforms such as Infochimps and Continuuity before, and more certainly will emerge — but Young is confident Mortar can grow a big community around it a la SpringSource in the Java application space. After all, despite spending the better part of its existence thus far intentionally holding back on marketing and support in order to focus on engineering, Young said developer uptake has nonetheless been good.

Mortar Data just became an advanced tier AWS partner, he noted, “which means we’re spending a crapload of money on Amazon.”

Feature image courtesy of Shutterstock user isak55.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data first-quarter 2013: analysis and outlook
• The importance of putting the U and I in visualization

Netflix Engineer Ben Christensen explained Hystrix thusly in a blog post on Monday:

Hystrix is a library designed to control the interactions between these distributed services providing greater tolerance of latency and failure. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve the system’s overall resiliency.

Hystrix actually stems from earlier work to add resilience to the Netflix API, the means by which many customer-facing applications access the services they need to run. As Christensen explained in a February 2012 blog post, services are distributed across thousands of instances in AWS, and if there are problems with those services — such as high latency or failed connections between them — it can wreak havoc on the Netflix API and seriously affect the performance of all the applications that depend on it.

Source: Netflix (https://speakerdeck.com/benjchristensen/performance-and-fault-tolerance-for-the-netflix-api-august-2012)

And, as he notes in that February post, “Intermittent failure is guaranteed with this many variables, even if every dependency itself has excellent availability and uptime … Thus, it is a requirement of high volume, high availability applications to build fault tolerance into their architecture and not expect infrastructure to solve it for them.”

Distributed systems are hard work to build and manage — ask anyone at Yahoo, Google or Facebook — and building distributed, service-oriented applications on top of those systems is probably no less difficult. Netflix has an even more-novel challenge because it opted to host all of its applications and services in the cloud, which provides some great tools for maximizing uptime but also some new layers of complexity in application architecture. The company’s focus on building resilient apps has been core to its ability to survive most of AWS’s cloud outages with little or no significant downtime.

In fact, Obama for America CTO Harper Reed told me during a post-election interview that Netflix tools and techniques helped the president’s AWS-hosted applications stay up and running even during three outages between late June and Nov. 6. Netflix has also open-sourced its Eureka load-balancing technology, its Edda dynamic querying tool, its Asgard management console and its lauded Chaos Monkey for testing application resilience.

It’s no surprise then that Netflix is something of a shining star at the AWS: Reinvent user conference in Las Vegas this week (CEO Reed Hastings will take the stage long with numerous engineers), prompting some to refer to it jokingly as a Netflix technology conference. Not that it’s an insult to anybody — Amazon and other infrastructure-as-a-service providers rent virtual servers, networks and management tools, but it takes cutting-edge users to engineer apps that can make the most of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud computing infrastructure: 2012 and beyond
• Amazon’s DynamoDB: rattling the cloud market

The idea behind the company is that developers and data scientists without huge budgets and/or mad Java skills should be able to take advantage of Hadoop both cheaply and easily. Mortar Data’s trick is a custom blend of the Pig and Python programming languages that makes job creation a snap, coupled with a cloud computing backend that means users don’t even have to own a server.

However, because it’s hosted on the Amazon Web Services cloud, Mortar Data is still best-suited to customers already storing data in Amazon’s S3 storage service. The good news for the company at this early stage is there are plenty of developers doing just that. It’s also good news for the developer community that Mortar recognizes this limitation and is working to expand the number of data sources and repositories it supports, as well as the means of visualizing the results of users’ Hadoop jobs.

In the meantime, you can see some of what Mortar is up to on its Github page, which includes the development framework (so developers can write jobs locally), a MongoDB adapter, an API client for Ruby and a couple of sample datasets. And, Co-Founder and CEO K Young has promised me that we can expect to see more news out of Mortar Data later this month as the company begins ramping up its efforts publicly.

Here’s a video of demonstrating the service in action:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The importance of putting the U and I in visualization
• A near-term outlook for big data

As WibiData product manager Devjit Chakravarti told me during a recent call, KijiSchema essentially “takes the ‘No’ out of NoSQL.” What he means is that although NoSQL databases such as HBase are lauded in part because they can store unstructured data and don’t require rigid rules for data formatting like relational databases do, having some structure is actually necessary once you want to do meaningful analysis on it. That’s why some commercial products, such as Drawn to Scale’s Spire and Splice Machine’s Splice SQL Engine, already have built functional SQL databases on top of HBase.

Kimball speaking at Structure: Data in 2012
(c) 2012 Pinar Ozger. pinar@pinarozger.com

“If you can’t store data in an organized way, you can’t analyze it effectively,” WibiData Co-Founder and CTO Aaron Kimball explained. KijiSchema isn’t part of WibiData’s secret sauce around predictive analytics for user data, he added, but nothing gets done without it.

Here’s how Kimball describes how KijiSchema manages data in a blog post announcing the project:

“KijiSchema gives developers the ability to easily store both structured and unstructured data within HBase using Avro serialization. It supports a variety of rich schema features, including complex, compound data types, HBase column key and time-series indexing, as well cell-level evolving schemas that dynamically encode version information.

“KijiSchema promotes the use of entity-centric data modeling, where all information about a given entity (user, mobile device, ad, product, etc.), including dimensional and transaction data, is encoded within the same row. This approach is particularly valuable for user-based analytics such as targeting, recommendations, and personalization.”

Kiji resides in the lower left section

The coolest part for HBase developers or prospective HBase developers, however, might be that KijiSchema isn’t just code but is already pre-packaged any ready to deploy. WibiData has created what it calls the Kiji BentoBox — “a fully-functional HBase mini-cluster with KijiSchema on your machine with minimal configuration in under 15 minutes” — that’s available for download on Github.

KijiSchema is also part of a broader Kiji framework for HBase that WibiData plans to open source over the next year or so. People perceive HBase as being complicated to set up and having a steep learning curve, Kimball said, and his teams wants to make it more accessible and lower the barrier for getting started. The ultimate goal is to make the types of HBase applications that folks at Facebook, eBay and other large web shops are building something that any developer can do.

WibiData’s Omer Trajman, formerly VP of technology solutions at Cloudera, describes the ultimate Kiji framework as being akin what the Spring framework if for Java. Despite its complexity, “there are also tens of thousands of developers who have been able to figure [HBase] out,” he said, but learning it might take weeks of intensive training on learning the low-level guts of the Hadoop Distributed File System and other stuff. Why learn to build an enterprise Java application from scratch, Trajman asked, when you can just use Spring?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Cloud and data first-quarter 2013: analysis and outlook