Two years ago, VMware purchased WaveMaker and its technology for simplifying the construction of enterprise Java applications and made it part of its SpringSource business unit. As GigaOM’s Derrick Harris wrote at the time, Rod Johnson, who was then head of that business, said Wavemaker made sense because:

” … applications developed with it actually are Spring applications. That means VMware can tightly align WaveMaker and Spring developments to make WaveMaker an even more fulfilling experience, and after simple applications are built using WaveMaker, an organization’s Spring developers can go in and code away to make it work even better. When it comes to developing Java applications, VMware now has something for almost everybody, and it all works together at some level.”

The mystery here is that SpringSource and associated technologies were spun off to the VMware-EMC-founded Pivotal startup. Why WaveMaker was not included is unclear to me. I’ve pinged Pivotal for comment and will update with a response. And…. here’s that response from a Pivotal spokeswoman:

“Pivotal is focused on bringing a new platform to market based on Spring, Cloud Foundry and Hadoop. We have significant investments in these technologies, and believe that WaveMaker customers are best served by a dedicated effort from Pramati.”

Pramati co-founder and president Vijay Pullur said former Wavemaker exec Michael Harper is with his company, which is expanding into cloud infrastructure. WaveMaker technologies could play a roll in an upcoming cloud venture, he said. We’ll be talking about projects like this and what’s new in cloud computing at Structure 2013 in San Francisco in June.

According to the press release announcing the news:

 ”WaveMaker applications are cloud-ready, highly scalable, multi-device, and backed by a strong developer community that has doubled to 35,000 active monthly users over the last two years. With its long heritage of mission-critical Java application development, Pramati expects to accelerate this growth going forward.”

In other words: stay tuned.

This story was updated at 7:04 p.m. PST with Pivotal comment.

This story

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft
• Infrastructure Q2: Big data and PaaS gain more momentum

In a world of mobile apps and cloud resources, it’s easier than ever to start a business around a simple application. Even in large companies, developers fighting for resources might need to prove an application’s popularity or find a way to boost its monetization. Sometimes, that might even mean injecting some data-processing right into an application.

But whatever the case, if your job revolves around writing code rather than data flows, you might need a little help. Here are 12 tools (listed alphabetically) that aim to help. As usual with this type of list, it’s very possible I left out some good options, so please note any omissions in the comments.

1. BitDeli

BitDeli, a startup that launched in November, lets programmers measure pretty much whatever application metrics they want using Python scripts. Co-founder and CEO Ville Tuulos told me at the time that scripts can be as simple or complex as necessary — even going so far as to incorporate machine learning. Compared with the heavyweight Hadoop, BitDeli thinks of itself as the lightweight Ruby on Rails for analytics.

2. Continuuity

The brainchild of former Yahoo Chief Cloud Architect Todd Papaioannou and Facebook HBase engineer Jonathan Gray, Continuuity wants to help all companies operate like its founders’ former employers. The team created a big data fabric that abstracts the complexities of connecting to Hadoop and HBase clusters and includes a full suite of developer tools. The goal is to make it easy to write big data applications serving either internal or external audiences.

3. Flurry

Flurry is like a one-stop mobile-app shop, and it’s generating nearly $100 million a year in revenue because it’s good at what it does. Not only does the company help developers build mobile apps, but helps them analyze all the data those apps are generating in order to make them even better. The data also underpins the company’s ad network that helps developers monetize their apps by putting the right advertisers in front of the right users.

4. Google Prediction API

Of all the tools in Google’s developer toolbox, the Google Prediction API might be the coolest. If you have good data to train a model, the Prediction API can bring machine learning to work on it in order to discern any number of pattern types and feed the answers into your application. Among the examples Google gives are spam detection, recommendation engines and sentiment analysis — and it gives step-by-step instructions for building those models.

“Mucho bueno” is probably Spanish.

5. Infochimps

Although Infochimps is trying hard to make itself an enterprise IT company (hey, that’s where the money is), the company’s eponymous platform also provides a real value for developers. Sitting atop its technologies for configuring and managing big data environments is Wukong, a framework for creating Hadoop jobs or streaming data flows using Ruby scripts. Infochimps also maintains a data marketplace full of API-accessible or downloadable datasets.

6. Keen IO

Keen IO won our Structure 2012 Launchpad competition with a message of delivering powerful analytics to mobile developers. With just a single line of code inserted that dictates what to track, the company claims developers can track pretty much whatever they want within their applications. At that point, it’s just a matter of creating a dashboard or query process in order to turn all that data into usable information.

7. Kontagent

Kontagent‘s bread-and-butter business is its analytics platform for mobile, social and web applications, but it’s all built atop a Hadoop infrastructure designed to handle really big data. Earlier this year, the company turned that infrastructure loose with a new product that lets users mine their application data using the SQL-like Hive query language for Hadoop. Instead of tracking predetermined variables, they can dig in however they choose.

8. Mortar Data

Mortar Data is Hadoop for developers, plain and simple. The company has offered its cloud service — which replaces MapReduce with a combination Pig and Python — for almost a year. In November, it released the open source Mortar framework in order to build a community around sharing datasets and making it easier to write Hadoop pipelines. Mortar Data runs atop Amazon Web Services and currently supports Amazon S3 and MongoDB (hosted on Amazon EC2) as data sources.

9. Placed Analytics

Placed does away with scripts, APIs and any other developer legwork and just delivers the results. In the case of Placed, those results are detailed information about where and when consumers are actually using mobile apps and web sites — right down to the name of the business. This type of info can be useful for attracting advertisers as well as informing app design (e.g., implementing voice controls if people are using an app while driving).

10. Precog

Precog might look like any other proprietary business intelligence service, but underneath its covers there’s a twist. The company offers a service called Labcoat, which is an interactive development environment for writing analytics jobs based on the open source Quirrel query language. The IDE includes a tutorial for learning the language, as well as some complex functions, and Precog COO Jeff Carr told me even non-technical people can learn it in hours.

11. Spring for Apache Hadoop

Hadoop is written in Java, but that doesn’t mean it’s easy for Java developers to learn or use. That’s why, in early 2012, SpringSource announced the Spring for Apache Hadoop project, which brings the ease of building Java applications with the Spring framework to Hadoop jobs. That means integration with other Spring apps, scripting using JVM-based languages and a generally easier way to develop applications that utilize Hadoop or related technologies such as Hive or HBase.

12. StatsMix

In the same vein as BitDeli and Keen IO, StatsMix wants to let developers start collecting and analyzing application data using the languages they already know. The service automatically tracks certain metrics, but developers can add their own using the StatsMix API and predefined code libraries. The results are delivered via a collection of dashboards that users can customize, share and use to mashup multiple data sources into a single view.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Odersky is something of a superstar in the Java programming world. He wrote Javac, the most widely used Java compiler, and now he’s the force behind the fast-growing Scala programming language. That language makes it easier for developers to code for “parallelism,”  which is what allows tens of thousands of people to use an application at the same time without crashing it.

In the pre-cloud, client-server era, you might have had a couple hundred  – or maybe thousand –  users hammering on a server-based application. But they accessed it from company-issued PCs, so programmers could assume a finite number of users and make sure they had enough server power and bandwidth to support that number.

That all goes away in today’s world, where millions of people use popular applications at the same time. How many smartphone users hit Twitter during a major sporting event? Or on election day? Your guess is as good as mine. It is that scale that parallelism enables and that Scala helps make easier to program.

Why Scala?

Up until a few years ago, to make applications perform better you installed a faster processor. But we’ve pretty much maxed out the speed limit for individual chips. So best way to get better performance now is to use chips with multiple cores, all operating at a high speed, and to spread the workload among them. Here’s an admittedly simplistic analogy: Instead of packing multiple tons of cargo onto a single huge freighter, you divvy the load up among an array of smaller boats that can move faster.

The key is to have a captain who knows how to take advantage of that expanded fleet — or, to bring the analogy back to the tech world, an application that knows how to take advantage of those multiple cores. Instead of writing a program that runs on a single core, you have to write a program that’s smarter about deploying the workloads among many cores.

“Single-core performance is running out of steam, and you need to parallelize everything,” Odersky told me in a recent interview. You do that through what’s known as functional programming. Ray Ozzie, the former chief software architect for Microsoft and no slouch when it comes to coding, likens functional programming to spreadsheets where each cell in the spreadsheet containing that formula acts as an independent processor working concurrently to keep the spreadsheet updated. It’s a parallel computing system enabled by functional programming, Ozzie says via email.

Scala works with Java and compiles in the JVM, which is significant because many, many of the world’s enterprise applications are written in Java. It’s not a stretch to say there are millions of Java programmers (Oracle, which now owns Java, claims 9 million.)  With Scala these programmers can keep using their Java libraries, frameworks and the JVM while also taking advantage of functional programming, which tends to be less verbose than Java code.

That brevity leads to more compact, elegant software compared to older-style imperative programming. With imperative programming, variables can evolve over time, while in functional programming variables keep the same value. The notion of shifting variables poses a problem in a parallel process where one part of the program executes based on an older value that has since changed.

Scala gains steam

In September, Redmonk analyst Stephen O’Grady used data from Github and Stackoverflow to show Scala on its way to becoming a top-tier language, along with Java, Javascript, PHP, and Python. Other functional languages such as Erlang and Haskel have their admirers but their user base isn’t growing as fast, according to this data.

Odersky, a professor at the Ecole Polytechnique Federale de Lausanne (EPFL) in Switzerland, is also co-founder and CTO of Typesafe, a San Francisco startup that promotes the use of Scala and related Akka middleware, especially in the enterprise. Typesafe customers include LinkedIn and the Dutch Border Patrol, which uses a Scala-based application to photograph every car coming into the country and quickly know — based on the license plate — whether to stop that car or not. Twitter and FourSquare are also Scala users. Odersky also teaches a Coursera class on Scala that drew an astounding 45,000 registrants.

“I studied with lots of amazing people at MIT, but very few successfully cross that academic-to-business divide, and Martin has,” says Bill Kaiser, a partner with Greylock Partners, a Typesafe investor. Odersky’s ongoing interaction with students allows him to stay involved in what’s new in programming, adds Mark Brewer, CEO of Typesafe, who jokes that Odersky spends about 50 hours a week on Typesafe business and another 50 teaching.

The programming pantheon

Those two and others make the case that Odersky belongs in the same pantheon of programming gods as James Gosling, the father of Java itself; Niklaus Wirth, who wrote Pascal (and with whom Odersky studied); Anders Hejlsberg, of Turbo Pascal fame; and Bjarne Stoustrup, who wrote C++ and other languages.

Rod Johnson, the co-founder of Springsource, now part of VMware, and now a director of Typesafe, says Martin “absolutely” belongs in this august company. “Considering prior art in each case, I would rate Scala as a more impressive — and original — achievement than Java or C# and on a par with C++,” Johnson says via email. “The way Scala successfully mixes functions and objects; the way in which it resolves the multiple inheritance problem; its effective type inference; and its interoperability with Java are all particularly impressive.”

The next frontier

Odersky said the explosion of mobile devices continues to challenge programmers. While new languages and tools like Scala helped, more needs to be done to deliver software that keeps up with the new hardware. For one thing, it needs to be much easier for non-programming geniuses to both write and troubleshoot such software.

“If you want to write a multi-threaded application now it’s still nightmarishly difficult. There are lots of mistakes that are hard to detect. We have to make programming these kinds of applications feasible for everyone, not just experts and that is very hard,” he says.

For more on Odersky and Scala, check out this video of a talk he gave at Intel.

Intel hosts Dr. Martin Odersky presenting Scala 2.10 from Typesafe on Vimeo.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft
• Survey: How apps can solve photo management

Typesafe, based in Menlo Park, CA,  is positioning both the Scala language and related Akka software stack as top-tier tools for both web and enterprise development because they attack two key issues: The language makes it easier to write code to run on multiple cores and Akka eases the creation of applications that run across multiple servers. A third piece of the Typesafe puzzle is Play, a Ruby-on-Rails-style framework.

In a statement, Johnson said:

“Today’s computing environments are moving towards multicore hardware and cloud computing workloads. Typesafe is strategically positioned to provide innovate solutions with its modern Scala and Akka-based software stack and developer tools for the next wave of applications.”

SpringSource co-founder Rod Johnson

Update: Typesafe co-founder and chief architect Martin Odersky wrote the current version of the  original Java compiler for Sun Microsystems, and is something of a superstar for programmers. His Coursera class on the language drew a whopping 45,000 registrants and he will be talking about the upcoming Scala release at JavaOne this week in San francisco. For more on Scala Release 2.10, see this InfoWorld report. Typesafe also  launched a mobile app Friday for Android and iPhones for Scala developers going into the show

Last June, Typesafe brought in new CEO, Mark Brewer, former VP of business operations for VMware’s Cloud Application Platform,  to drive its enterprise push.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft
• A 2011 Infrastructure Forecast

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Amazon’s DynamoDB: rattling the cloud market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond

I was surprised, if not shocked, when I first heard the rumor of Gelsinger replacing Maritz as VMware CEO. Without details, the natural connotation of such a move is, “We’re firing you because we don’t think you’re doing a good job.”

Which was strange in this case, because most people bought into Maritz’s vision of an application-centric future whole-hog. The premise is simple and, more importantly, true: Servers, virtual machines, databases, operating systems — everything below the application — will become commodotized, leaving all the room for differentiation at the application layer.

It seems EMC bought into it, too, just maybe not as the ideal business for an infrastructure company such as VMware. Read between the lines of EMC CEO Joe Tucci’s new job descriptions for Maritz and Gelsinger:

Pat will now lead Cloud Infrastructure at VMware, and Paul will look across our technology strategy with a particular focus on Big Data and the next generation of cloud-oriented applications that will be built on top of these foundations.

The cloud computing and big data spin-out we reported Monday (or something very much like it) seems destined to happen, and Maritz will stay on board to ensure it stays true to his vision. Tucci has even given him the M&A and strategic investment reins. Well played, EMC. I think.

Give innovation room to roam

The cloud spin-out could do a lot of good by helping get EMC out of the way both technologically and culturally. As my colleague Stacey Higginbotham noted in covering the spin-out rumors on Monday, EMC’s insistence that VMware build Cloud Foundry on its pricey, absolutely-not-commodity storage gear rankled a lot of the team and might have helped catalyze the idea for a separate entity. That’s just not how successful cloud services are built.

There’s also a general feeling among the platform-as-a-service world that while Cloud Foundry is a great open-source project, VMware’s hosted version is just a ploy to sell more software licenses.

And as I explained in a post on Monday, VMware is losing (and seemingly not replacing) a lot of key talent that are leaving to free their innovative sides. Large companies under the thumb of even-larger companies have rarely (never?) been a breeding ground for innovation.

The proposed cloud spin-out could solve all these problems. Cloud Foundry, Greenplum and, it appears, VMware’s current suite of cloud-based applications, will be able operate as they see fit (i.e., initially building out large ecosystems of application developers) without having to worry about padding EMC’s or VMware’s bottom line and keeping shareholders happy.

Free from the large-enterprise constraints and operating in some white-hot areas, a spin-out might actually lure top talent and create some compelling products.

The infrastructure ball is in VMware’s hands

Of course, VMware and EMC stand to gain from this, too. If Maritz really did take his eye off the hypervisor ball to focus on applications, new CEO Gelsinger can put all his focus back on the infrastructure. VMware will still make a lot of money from its virtualization-management software — all that gear and all those virtual machines atop which applications run aren’t going to manage themselves — but it’s going to take some serious attention to find the right strategy for the future.

I would argue the answer to receding market share as more companies adopt the Microsoft Hyper-V, Citrix XenServer and KVM hypervisors for new workloads is to make vSphere entirely free and ship it on as many physical servers as it can. VMware CTO Steve Herrod’s software-defined data center vision is legit, and it’s where VMware has to make its infrastructure money as hypervisors become commodities. But when customer suspect they’re being bent over the barrel by a vendor trying to lock them in at every level and extract as much money as possible, they get understandably leery about buying anything at all.

Or maybe VMware envies Oracle and wants to go even further in that direction (although its DynamicOps acquisition suggests otherwise). Perhaps, as some Wells Fargo analysts suggested in a note this morning, the answer is for EMC to buy back VMware and bring it entirely back into the mothership. Whatever the case, the infrastructure business is in Gelsinger’s court now. He’s free to do what he thinks is best without having to worry about things like applications and cloud platforms.

Or is he …?

Products without homes

Some commentators have already questioned the wisdom of putting both an infrastructure-as-a-service cloud and a PaaS offering within the control of the proposed spin-out, and they might be right. According to my sources, Project Rubicon (the code name for VMware’s IaaS cloud) is already up and running and more closely resembles a hosted private cloud option than it does a commodity cloud such as Amazon Web Services. In fact, I’m told, it’s being headed up by former Terremark executives Jason Lochhead and Randy Rowland.

That sounds a lot more like a Gelsinger business than it does a Maritz business. Even relatively recently, it sounds like VMware wasn’t sure how to position the service (don’t forget, it already has hundreds of vCloud Data Center partners) or where it should live.

SpringSource, too, seems like something of a motherless child. It’s not inherently cloudy and it’s a Java application platform that kind of lets VMware compete with Oracle and IBM in what’s still a big market. My assumption, though, given that it also comprises the data components of VMware’s current application-centric focus, is that if a spin-out happens, SpringSource goes with it.

At some level, everything EMC and VMware have announced in the past day and are rumored to be doing make sense. There’s the potential for a big business if an EMC-VMware spin-out can become the leader in helping developers build next-generation cloud and big data applications. There’s also  plenty of business yet to do managing companies’ virtualized resources. But there’s a lot of gray area between those two businesses, and navigating that will be the hard part.

Feature image courtesy of Shutterstock user nokhoog_buchachon.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The fourth quarter of 2012 in cloud
• Infrastructure Q1: Cloud and big data woo enterprises

Paul Maritz CEO of VMware at Structure 2011.

Over the past four years, VMware under CEO Paul Maritz, has been nothing if not ambitious.

It entered the battle for software developers with its SpringSource acquisition in 2009. It jumped into email and applications by buying Zimbra the following year. Those moves, by a CEO who had spent years at Microsoft, led many to conclude that he was recreating Microsoft’s development tools-and-applications model in an effort to go head to head with the mothership. Later, VMware also entered the platform as a service fray with CloudFoundry while Microsoft built up its Azure PaaS. GigaOM Monday broke the news that VMware may spin out CloudFoundry as part of a cloud-focused business.

What VMware hasn’t done, according to critics, is protect its bread-and-butter server virtualization business — and that may prove costly.

Protecting (or not) the core

VMware left its core business exposed, they say, first by announcing heavy-handed vSphere price hikes last year that, in the words of one VMware watcher, “kicked the door open for Microsoft Hyper-V.” VMware has yet to recover from that, in his view.

“Silicon Valley is baffled at how easy VMware has made it for Microsoft to come in and take all the easy stuff,” this source said. “They’re trying to optimize for revenue instead of market share and — good, God — Maritz if anyone should know that they need to occupy the high-share, high-volume, low-price position, which is what Microsoft did to destroy the legacy Unix OS business. VMware is behaving more like a legacy player than anything.”

Analyst Bernd Harzog of The Virtualization Practice sums it up: “Someone once said that Microsoft gets things right on the third try. Well guess what? Here comes Hyper-V release 3.”

The problem for VMware is that while Hyper-V started out slow, it’s catching up, said Greg Shields, senior partner at Concentrated Technology, a firm specializing in IT industry analysis. “IT pros didn’t get that warm fuzzy of confidence with its early versions,” he said.  ”Many features weren’t designed to ‘just work’. Contrast that with VMware vSphere, where a lot of the tools worked intuitively. Microsoft understands that now, and key aspects like clustering are far more smartly designed [in Windows Server 12 and Hyper-V 3].”

And because HyperV comes bundled with Windows Server, it gets traction in thousands of Microsoft shops. It’s understandable that VMware would want to focus on higher-level (and paid) virtualization management tools and applications as hypervisors got commoditized. But, it  also needed to shore up its base to ward off incursions not only by Hyper-V but two capable open-source hypervisors as well: KVM (backed by Red Hat and others) as well as Xen (backed by Citrix and others.)

Tier1 Research analyst Carl Brooks said VMware took it’s eye off that core business and in doing so provided an opening for others to move in on a growth market as the world moves to cloud computing. “I’d say that [VMware's] software [buying] spree was a bizarre side quest for them. They need to continue to move into true multiple resource and multiple environment management and do it quick or there’s a real risk they could be the Novell of cloud 5 to 10 years from now.”

Managing beyond VMware

VMware’s acquisition two weeks ago of DynamicOps illustrates the company’s growing interest in managing cloud infrastructure outside the vCloud realm. DynamicOps promises to enable vCloud Director users to consume resources in the Xen and Hyper-V world as well as Amazon EC2, as Derrick Harris reported at the time. This could be an acknowledgement by VMware that it needs to deal with outside virtualization and management tools.

While observers disagree about how much damage has been done — Interarbor Solutions’ Analyst Dana Gardner said he sees no evidence of VMware ESX and ESXi losing ground to Hyper-V even as new workloads get virtualized. But, for better or worse, the perception remains that VMware has moved on from that key foundational technology. And by taking  its core hypervisor business for granted, the company left a gap for competitors to exploit and they are bent on doing so.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• VMware’s Cloudy Ambitions: Can It Repeat Hypervisor Success?
• The promise of software-defined networking

Typesafe CEO Mark Brewer

Details are slim since neither Juniper or Typesafe will say what — if any — technology Scala and Akka will replace — there are current Juniper job postings seeking programmers with Java, C and C++ experience. Nor did the companies detail what products Scala and Akka will be used for. Still, the endorsement by a big network hardware company is worth noting.

In a statement, Will Eatherton, VP of engineering for Juniper’s core routing business, said that the Typesafe Stack — including Scala and Akka bring a “fresh approach to software development.” Using those tools, he added, Juniper developers will be able to “quickly and reliably create distributed software based on Akka middleware that can scale to take advantage of modern multi-core processors.”

That statement gets to what makes Scala special. The language makes it easier to write code for multiple processor cores and Akka eases creation of distributed applications that run across many servers.

Typesafe recruits new CEO from VMware

To boost Scala’s profile, Typesafe earlier this month brought aboard a new CEO in Mark Brewer, former VP of business operations for VMware’s Cloud Application Platform. Brewer also joined the board, joining Martin Odersky, Typesafe Chairman and Chief Architect , Bill Kaiser, and François Stieger.

Brewer said while at Springsource and then at VMware, (he joined VMware when it acquired Springsource three years ago) he kept an eye on Scala’s and Akka’s progress and was intrigued by what he saw.

“Akka is really the best lightweight, distributed platform for running Java or Scala apps — it runs across cloud in a very light fashion — we couldn’t do that at VMware. And we started to see adoption in the enterprise — not just in web property companies,” he said. ”LinkedIn and Twitter use Scala for its performance but now we’re seeing Scala and Akka in use at more traditional enterprises,” Brewer told me in a recent interview.

Booming demand for Scala and Akka

He points to significant growth in the past 12 months with downloads of Scala more than doubling from 28,000 to 60,000  and Akka downloads quadrupling from 4000 to 20,000 in that period.

And, while he does not see the Scala/Akka tandem competing with VMware’s software development stable now given that many Spring projects can work with Scala projects, that will change:

“We’re not competitive today but most definitely in the future we’ll compete with VMware,” he said.

According to Github, the bible of software developers, Scala is now the 17th most-watched programming language, after such crowd pleasers as JavaScript, Ruby, Python, Java, C++ and others, but is coming up fast, Brewer said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• The Case for Increased M&A in 2011: Actions and Outlooks
• The Structure 50: The Top 50 Cloud Innovators

Stephen Herrod - CTO, VMware at Mobilize 2011

For a platform as a service, it’s no longer enough to support all the major languages and development frameworks, it must now also run across all the major cloud platforms.

That’s the rationale behind CloudFoundry’s new open-source BOSH management process, which should make it easier to deploy VMware’s PaaS across multiple clouds and on the biggest clouds.

BOSH ”is a Google-style solution for a Google-style problem,” VMware CTO Stephen Herrod told attendees of an event commemorating CloudFoundry’s first birthday Wednesday.

According to VMware’s web site, BOSH is “an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services.” As such it is for “serious devops guys who can deal with a command line interface, understand what YAML is , can read an IP address,” Mark Lucovksy, VMware VP of engineering, told attendees. Details about BOSH are posted to Github.

“We preach continuous improvement, iterative development. There are rough edges, but they’re the type of rough edges these guys demand,” he added.

BOSH as service providers’ best friend

The process should come in handy for service providers who want to host CloudFoundry, VMware co-president Tod Nielsen said in an interview late Wednesday.  ”It lets people easily manage life-cycle roll-outs, updates, it’s all about the engine room operations. The developer doesn’t care,  but if you run a cloud service, your life just gets easier because you’re not going to have to write your own crazy Puppet and Chef scripts,” Nielsen said.

The CloudFoundry event seemed about  proving that VMware, which built its business on proprietary, commercial technology, is a good open source citizen now. Nielsen said the company’s purchase of SpringSource in 2009, its subsequent handling of that franchise, and the way it’s run CloudFoundry should allay any concerns. ”People thought we would tightly couple CloudFoundry to vSphere as a service and what happened is you can run CloudFoundry on vSPhere but you can also run it on Amazon and other infrastructure,” he said.

Companies go multi-cloud

It makes sense to be cloud agnostic. Since some highly publicized outages, many companies hesitate to  trust their entire cloud computing load on a single cloud provider.

VMware really wants CloudFoundry to be, as Herrod put it, the “Linux of cloud,” abstracting out differences between underlying IaaS structures so developers don’t have to worry about it. Just as Linux runs on all the major server platforms, CloudFoundry and its primary competitors must run on all the major clouds.

Going back a year, most PaaS vendors raced to support all the major programming languages and development frameworks. Now it looks like supporting all the major clouds is the new table stakes.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A 2011 Infrastructure Forecast
• VMware’s Cloudy Ambitions: Can It Repeat Hypervisor Success?

Hadoop, of course, is the Apache Software Foundation project for storing and processing large volumes of unstructured data, often referred to as “big data.” The project gained popularity within large web companies such as Yahoo and Facebook that had to find a way to deal with and analyze mountains of user data in the form of logfiles, photos, clickthroughs and other new formats. Hadoop was inspired by Google’s work on the MapReduce parallel-processing framework and its distributed Google File System

There is now a whole ecosystem of companies selling Hadoop-based products, from commercial distributions to application-specific analytics software, and many mainstream organizations are using it to analyze their own data. However, the Hadoop MapReduce framework is notoriously difficult to program, which has helped keep mainstream developers away from the valuable data stored within Hadoop.

According to VMware’s press release:

Key aspects of Spring Hadoop include:

• Support for configuration, creation and execution of MapReduce, Streaming, Hive, Pig and Cascading jobs via the Spring container
• Comprehensive HDFS data access support through JVM scripting languages (Groovy, JRuby, Jython, Rhino, etc.)
• Declarative configuration support for HBase
• Dedicated Spring Batch support for developing powerful workflow solutions incorporating HDFS operations and all types of Hadoop jobs
• Support for use with Spring Integration that provides easy access to a wide range of existing systems using an extensible event-driven pipes and filters architecture
• Powerful Hadoop configuration options and a templating mechanism for client connections to Hadoop
• Declarative and programmatic support for Hadoop Tools, including FsShell and DistCp

Spring Hadoop is also open source, available under the Apache 2.0 license. More information on programming Hadoop jobs using Spring is available on the SpringSource blog.

This isn’t the first time VMware has given Spring developers access to new types of data stores. In 2010, VMware bought  GemStone for its low-latency distributed data grid technology. That’s now part of the Spring Data project, which also features tie-ins to NoSQL databases MongoDB, Riak, Redis and Neo4j

News of Spring Hadoop comes a day after Microsoft exposed an expanded Hadoop strategy that includes a new JavaScript framework for writing Hadoop jobs and an integration with its Excel spreadsheet application. Microsoft’s efforts will open up Hadoop to tens of millions of developers and business users familiar with Microsoft’s products.

Making Hadoop accessible to more developers and, thus, more application types will go along way toward making it the de facto big data platform for future applications, a development we’ll discuss in more detail at our Structure:Data conference March 21-22 in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• Infrastructure Q1: Cloud and big data woo enterprises
• 9 Companies that Pushed the Infrastructure Discussion in 2010