(L to R): Ashok Srivastava, Venture Advisor for Trident Capital and Chief Data Scientist, Verizon; Silvius Rus, Director, Big Data Platforms, Quantcast; Todd Papaioannou, Founder and CEO, Continuuity; Bhaskar Ghosh, Senior Director of Engineering, Data Infrastructure, LinkedIn; Michael Driscoll. CEO, Metamarkets Structure Data 2013 Albert Chau itsmebert.com

Last week it was Sawzall‘s time to shine. The language, named after the popular saw that cuts through anything (and I mean anything), comes out of Google.

Silvius Rus, director of big data platforms for Quantcast, gave Sawzall a shout-out during a Structure Data Guru panel last week. ”It’s a lightweight language developed by Google that ridges procedural and interpretive languages,” Rus said.

Michael Driscoll, CEO of Metamarkets and moderator of the panel, later explained why that’s important. With a declarative language, the programmer tells the computer what to do in almost English-language-like sentences. To tell the computer to draw a circle, a declarative or imperative programmer might say “draw.circle with a size attached,” Driscoll said.

Procedural languages, on the other hand, are much more detailed step-by-step instructions — they sound more like math. A procedural approach would “define the actual pointer and tell it to move one degree to the left and one degree up and the square root of 2 up to the diagonal and repeat X times,” Driscoll said.

Sawzall is a nice blend between a declarative language that might be too high level to do all of what the programmer really wants and procedural, “which is way too in the weeds” to be fully productive, Driscoll said. More broadly, Sawzall is a powerful and compact language for log data aggregation and transformation. And, he added, it plays well with Hadoop MapReduce.

New toolsets for webscale computing

Another tool ranking high on the hit list was YARN (or Yet Another Resource Manager) aka MapReduce 2.0, cited by Todd Papaioannou, founder and CEO of Continuuity is a fan.

Yarn was built to “just think about mass-produced jobs.” Continuity is building a real-time streaming engine called Big Flow and using Yarn for all the resource deployment and management.

He also gave kudos to Weave, a higher-level framework. Weave “allows you to build a much wider class of applications on top of Yarn. So,t Yarn is  … something that we will be going forward with for at least the next half a decade [and] Weave allows you to actually build more wide scale applications on top of that.”

Bhaskar Ghosh,  senior director of engineering at LinkedIn, touted Helix, a generic distribution cluster manager developed at LinkedIn and which is now an Apache incubator project.  Helix simplifies distributed system development by separating cluster management from the primary component tasks of a distributed system, according to LinkedIn.

Kafka, Storm slake the thirst for real-time frameworks

Driscoll also sees traction for Kafka, a real-time framework for ingesting and managing data streams and Storm, out of Twitter, for processing those streams. “Think of Kafka and Storm as the HDFS and MapReduce analogs but for real time — Kafka for storage and Storm for compute,” Driscoll said.

On its blog, LinkedIn describes Kafka as a distributed publish-subscribe messaging system — also now an Apache project. Kafka is used by Twitter and Square for log aggregation, queeuing, and real-time monitoring and event processing.

This list is by no means complete. When I spoke with Github co-founder Tom Preston-Werner a few weeks ago, he said Clojure, heretofore a rather obscure dynamic programming language, is gaining momentum. “It’s getting a lot of buzz round on the enterprise side,” Preston-Werner said.

The continued popularity of the Java Virtual Machine has breathed new life into languages like Clojure and Scala, he added. Indeed, the JVM remains nearly ubiquitous and that is a huge advantage for languages that support it. If you’re a developer, you want the widest possible audience.

“The JVM is still the modern foundation that lets you run everywhere and Clojure has benefited from that,” Driscoll agreed. “It’s certainly gained steam among an elite set of programmers in Silicon Valley.”

Sawzall photo courtesy of Flickr user Charles & Hudson

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang
• Social 2013: The enterprise strikes back

Of course, as a commenter on a Hacker News item, pointed out, numbers can be sliced and diced many ways. If you look at Indeed.com’s chart showing “absolute” number of job postings, it’s clear that demand for Java skills reigns supreme still.

“The fastest growing languages also happen to be the languages with the least jobs,” wrote commenter Zandana. “I imagine Scala’s growth will tail off a long time before it can compete in [the] sheer number of jobs with JavaScript or Java.”

There’s little doubt that the use of Scala, the brainchild of Martin Odersky, is on the rise. Twitter and LinkedIn are big Scala users. One advantage is it enables Java programmers — and there are millions of them — to keep using familiar libraries and develop web-scale applications.

But then again, these charts only show what parameters are put in. If you add Objective-C to the mix, as another Hacker News commenter pointed out, you get still another view of what’s going on.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Overview, Q2 2010
• New challenges for the IT organization

Odersky is something of a superstar in the Java programming world. He wrote Javac, the most widely used Java compiler, and now he’s the force behind the fast-growing Scala programming language. That language makes it easier for developers to code for “parallelism,”  which is what allows tens of thousands of people to use an application at the same time without crashing it.

In the pre-cloud, client-server era, you might have had a couple hundred  – or maybe thousand –  users hammering on a server-based application. But they accessed it from company-issued PCs, so programmers could assume a finite number of users and make sure they had enough server power and bandwidth to support that number.

That all goes away in today’s world, where millions of people use popular applications at the same time. How many smartphone users hit Twitter during a major sporting event? Or on election day? Your guess is as good as mine. It is that scale that parallelism enables and that Scala helps make easier to program.

Why Scala?

Up until a few years ago, to make applications perform better you installed a faster processor. But we’ve pretty much maxed out the speed limit for individual chips. So best way to get better performance now is to use chips with multiple cores, all operating at a high speed, and to spread the workload among them. Here’s an admittedly simplistic analogy: Instead of packing multiple tons of cargo onto a single huge freighter, you divvy the load up among an array of smaller boats that can move faster.

The key is to have a captain who knows how to take advantage of that expanded fleet — or, to bring the analogy back to the tech world, an application that knows how to take advantage of those multiple cores. Instead of writing a program that runs on a single core, you have to write a program that’s smarter about deploying the workloads among many cores.

“Single-core performance is running out of steam, and you need to parallelize everything,” Odersky told me in a recent interview. You do that through what’s known as functional programming. Ray Ozzie, the former chief software architect for Microsoft and no slouch when it comes to coding, likens functional programming to spreadsheets where each cell in the spreadsheet containing that formula acts as an independent processor working concurrently to keep the spreadsheet updated. It’s a parallel computing system enabled by functional programming, Ozzie says via email.

Scala works with Java and compiles in the JVM, which is significant because many, many of the world’s enterprise applications are written in Java. It’s not a stretch to say there are millions of Java programmers (Oracle, which now owns Java, claims 9 million.)  With Scala these programmers can keep using their Java libraries, frameworks and the JVM while also taking advantage of functional programming, which tends to be less verbose than Java code.

That brevity leads to more compact, elegant software compared to older-style imperative programming. With imperative programming, variables can evolve over time, while in functional programming variables keep the same value. The notion of shifting variables poses a problem in a parallel process where one part of the program executes based on an older value that has since changed.

Scala gains steam

In September, Redmonk analyst Stephen O’Grady used data from Github and Stackoverflow to show Scala on its way to becoming a top-tier language, along with Java, Javascript, PHP, and Python. Other functional languages such as Erlang and Haskel have their admirers but their user base isn’t growing as fast, according to this data.

Odersky, a professor at the Ecole Polytechnique Federale de Lausanne (EPFL) in Switzerland, is also co-founder and CTO of Typesafe, a San Francisco startup that promotes the use of Scala and related Akka middleware, especially in the enterprise. Typesafe customers include LinkedIn and the Dutch Border Patrol, which uses a Scala-based application to photograph every car coming into the country and quickly know — based on the license plate — whether to stop that car or not. Twitter and FourSquare are also Scala users. Odersky also teaches a Coursera class on Scala that drew an astounding 45,000 registrants.

“I studied with lots of amazing people at MIT, but very few successfully cross that academic-to-business divide, and Martin has,” says Bill Kaiser, a partner with Greylock Partners, a Typesafe investor. Odersky’s ongoing interaction with students allows him to stay involved in what’s new in programming, adds Mark Brewer, CEO of Typesafe, who jokes that Odersky spends about 50 hours a week on Typesafe business and another 50 teaching.

The programming pantheon

Those two and others make the case that Odersky belongs in the same pantheon of programming gods as James Gosling, the father of Java itself; Niklaus Wirth, who wrote Pascal (and with whom Odersky studied); Anders Hejlsberg, of Turbo Pascal fame; and Bjarne Stoustrup, who wrote C++ and other languages.

Rod Johnson, the co-founder of Springsource, now part of VMware, and now a director of Typesafe, says Martin “absolutely” belongs in this august company. “Considering prior art in each case, I would rate Scala as a more impressive — and original — achievement than Java or C# and on a par with C++,” Johnson says via email. “The way Scala successfully mixes functions and objects; the way in which it resolves the multiple inheritance problem; its effective type inference; and its interoperability with Java are all particularly impressive.”

The next frontier

Odersky said the explosion of mobile devices continues to challenge programmers. While new languages and tools like Scala helped, more needs to be done to deliver software that keeps up with the new hardware. For one thing, it needs to be much easier for non-programming geniuses to both write and troubleshoot such software.

“If you want to write a multi-threaded application now it’s still nightmarishly difficult. There are lots of mistakes that are hard to detect. We have to make programming these kinds of applications feasible for everyone, not just experts and that is very hard,” he says.

For more on Odersky and Scala, check out this video of a talk he gave at Intel.

Intel hosts Dr. Martin Odersky presenting Scala 2.10 from Typesafe on Vimeo.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft
• Emerging trends in the non-relational database market

Typesafe, based in Menlo Park, CA,  is positioning both the Scala language and related Akka software stack as top-tier tools for both web and enterprise development because they attack two key issues: The language makes it easier to write code to run on multiple cores and Akka eases the creation of applications that run across multiple servers. A third piece of the Typesafe puzzle is Play, a Ruby-on-Rails-style framework.

In a statement, Johnson said:

“Today’s computing environments are moving towards multicore hardware and cloud computing workloads. Typesafe is strategically positioned to provide innovate solutions with its modern Scala and Akka-based software stack and developer tools for the next wave of applications.”

SpringSource co-founder Rod Johnson

Update: Typesafe co-founder and chief architect Martin Odersky wrote the current version of the  original Java compiler for Sun Microsystems, and is something of a superstar for programmers. His Coursera class on the language drew a whopping 45,000 registrants and he will be talking about the upcoming Scala release at JavaOne this week in San francisco. For more on Scala Release 2.10, see this InfoWorld report. Typesafe also  launched a mobile app Friday for Android and iPhones for Scala developers going into the show

Last June, Typesafe brought in new CEO, Mark Brewer, former VP of business operations for VMware’s Cloud Application Platform,  to drive its enterprise push.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft
• A 2011 Infrastructure Forecast

Collison sparked a discussion about Go’s prospects when he tweeted that he felt Go will become the dominant language for systems work in infrastructure as a service, orchestration and platform as as service.

Prediction: Go will become the dominant language for systems work in IaaS, Orchestration, and PaaS in 24 months. #golang—
Derek Collison (@derekcollison) September 11, 2012

Reached by email, Collison said Go or “Golang” appeals to people working above the kernel or the driver layer which is typically written in C or C++. Go’s support of static typing is key and makes it suitable for  building high-performance systems.

“Static typing essentially means explicitly declaring everything prior to a compile,” said RedMonk analyst Stephen O’Grady via email. “Dynamically-typed systems are much more loose, and therefore generally faster to code in. The advantages of static typing tend to be in high performance systems, because there is no decision to be made about the type at run time, or systems of substantial complexity, because looser [dynamic] typing can lead to difficulty in debugging errors.”

I have now completed two projects in Go. I predict that it's going to be the dominant language for server work. #golang—
Tobias Lütke (@tobi) September 12, 2012

Spurred by the discussion, O’Grady did his own analysis using Github and StackOverflow data. The resulting RedMonk language rankings found JavaScript, Java, PHP, Python and Ruby still ensconced as the top five languages overall although Go has made some progress, moving from number 32 in 2011 to number 30 this week. 

O’Grady wrote that this progress may sound modest

” … but means that in that time [Go] has improved more in popularity than Scala or Haskell and as much as Java, at least from a rankings standpoint (obviously growth becomes more difficult the more popular the language becomes). Second, there’s its age. At a bit less than three years of age, Go’s position as a solidly second tier language is enviable, given the fact that there are much older languages like Smalltalk that have yet to break that barrier.”

Apcera is using Go, Collison said, as are Heroku, CloudFoundry, Google where the language got its start,  and other companies. In his view, C will always have its place but Ruby and Python code bases will go to Go.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Platform as a Service in 2012
• PaaS market accelerators, 2012–2013

Typesafe CEO Mark Brewer

“A year ago most of the apps [using Scala] were scale-out big web applications like Twitter, Foursquare and LinkedIn but in that time we started seeing more traditional business applications where developers chose Scala over Java,” Brewer said. He said many developers find Scala more lightweight and streamlined than Java itself.

“Scala is extremely intuitive and … it is also extremely easy to access libraries from Java,” said Jason Pressman, a Shasta managing director who is now joining the Typesafe board.  Shasta has a history of backing open-source-oriented companies including LucidWorks, once known as Lucid Imagination, and Makara, which was acquired by Red Hat and became the basis of its OpenShift platform as a service.

Typesafe will also continue to build out the Scala-Akka stack adding more components like the recently announced Slick database connector, which makes it easier for developers to use Scala with relational and non-relational databases.

The new funding comes a year and a half after Typesafe netted a $3.5 million Series A round and includes contributions from existing backers Greylock Partners and Francois Stieger.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• WAN design for the cloud age
• A near-term outlook for big data

Typesafe CEO Mark Brewer

Details are slim since neither Juniper or Typesafe will say what — if any — technology Scala and Akka will replace — there are current Juniper job postings seeking programmers with Java, C and C++ experience. Nor did the companies detail what products Scala and Akka will be used for. Still, the endorsement by a big network hardware company is worth noting.

In a statement, Will Eatherton, VP of engineering for Juniper’s core routing business, said that the Typesafe Stack — including Scala and Akka bring a “fresh approach to software development.” Using those tools, he added, Juniper developers will be able to “quickly and reliably create distributed software based on Akka middleware that can scale to take advantage of modern multi-core processors.”

That statement gets to what makes Scala special. The language makes it easier to write code for multiple processor cores and Akka eases creation of distributed applications that run across many servers.

Typesafe recruits new CEO from VMware

To boost Scala’s profile, Typesafe earlier this month brought aboard a new CEO in Mark Brewer, former VP of business operations for VMware’s Cloud Application Platform. Brewer also joined the board, joining Martin Odersky, Typesafe Chairman and Chief Architect , Bill Kaiser, and François Stieger.

Brewer said while at Springsource and then at VMware, (he joined VMware when it acquired Springsource three years ago) he kept an eye on Scala’s and Akka’s progress and was intrigued by what he saw.

“Akka is really the best lightweight, distributed platform for running Java or Scala apps — it runs across cloud in a very light fashion — we couldn’t do that at VMware. And we started to see adoption in the enterprise — not just in web property companies,” he said. ”LinkedIn and Twitter use Scala for its performance but now we’re seeing Scala and Akka in use at more traditional enterprises,” Brewer told me in a recent interview.

Booming demand for Scala and Akka

He points to significant growth in the past 12 months with downloads of Scala more than doubling from 28,000 to 60,000  and Akka downloads quadrupling from 4000 to 20,000 in that period.

And, while he does not see the Scala/Akka tandem competing with VMware’s software development stable now given that many Spring projects can work with Scala projects, that will change:

“We’re not competitive today but most definitely in the future we’ll compete with VMware,” he said.

According to Github, the bible of software developers, Scala is now the 17th most-watched programming language, after such crowd pleasers as JavaScript, Ruby, Python, Java, C++ and others, but is coming up fast, Brewer said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• The Case for Increased M&A in 2011: Actions and Outlooks
• The Structure 50: The Top 50 Cloud Innovators

Many companies would like to give their developers a PaaS option but are hesitant to go all-in with the public cloud. Just a few PaaSes, like VMware’s Cloud Foundry, offer that sort of hybrid cloud deployment choice.

CloudBees’ new AnyCloud service supports JVM-based languages and frameworks like Jruby, Groovy, Scala et al. while other PaaSes stress broader multi-language support.

Steve Harris, SVP of new products for the Woburn, Mass.-based company said flexible deployment is a major benefit for many companies that need to keep their options open. “Many enterprises have existing investments in infrastructure and local resources they need to take advantage of. They can also opportunistically put stuff on the public cloud,” he said.

CloudBees provides the company with what Harris called a fully serviced platform. “We deliver the platform as a service whether it runs on public cloud, datacenter or hosted data center — you do not install anything. We architected it in a way where you identify the resources in your data center or hosted provider but you use us to deploy and manage it,” he said.

CloudBees competes not only with other PaaSes, like Red Hat’s Java-focused OpenShift and the aforementioned Cloud Foundry, but also with traditional Java-based middleware from Oracle and other companies, Harris said.

The CloudBees AnyCloud PaaS is available in North America and Europe. The two-year-old company, backed by VCs Matrix Partners and Lightspeed Venture Partners, was founded by Sacha Labourey, former CTO of JBoss, which is now owned by Red Hat.

While most of the PaaS players continue their arms race by adding more language support, CloudBees is pushing cloud deployment options, which could make it a power in the still-vibrant Java development world.

Photo courtesy of Flickr user Joost J. Bakker IJmuiden.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Infrastructure Q3: OpenStack and flash step into the spotlight

Martin Odersky

To hear Typesafe tell it, the Scala programming language is about to join the ranks of top-tier development tools such as Java, C++, Ruby, and PHP. A new Scala plugin for the Eclipse integrated development environment (IDE) should help pave the way.

The Scala software stack attacks two sticky problems for software developers. The Scala language aims to make it easier to write code for multiple cores, and the Akka middleware attempts to ease the creation of applications that run across multiple servers.

A third piece, a Ruby-on-Rails-style framework called Play, makes development more productive and fun than is usually the case in Java, said Donald Fischer, CEO of Typesafe, the startup charged with promoting and supporting the stack.

The ability to better automate programming for multi-core chips and distributed servers is a key criterion for developing cloud applications that scale up big time.

Recapping Scala history

Scala was initially developed by Martin Odersky, who wrote the original Java compiler for Sun Microsystems. ”He did [the compiler] in 2000 as part of making Java better. Then he handed it off to Sun and turned to making a better Java. That’s the genesis of Scala,” Fischer said.

Scala is used by some pretty big names in web infrastructure: Twitter, LinkedIn, and Yammer for key parts of their infrastructure. There was a ruckus last month when an email between a Yammer employee and Typesafe leaked that seemed to indicate that the company was pulling back from Scala use. Yammer clarified the matter in a subsequent blog post.

Typesafe, Menlo Park, CA., was founded earlier this year with $3 million funding from Greylock Partners.

Getting beyond those awkward years

One thing still lacking for Scala, an eight-year-old programming language that runs on the Java Virtual Machine (JVM), is that some of its ancillary tools remain a “little raw,”  Fischer acknowledged.

“The support for Eclipse wasn’t all cooked. With 2.0, working with the Scala community, we’ve smoothed out the rough edges,”  Fischer said. Eclipse is the IDE of choice for the vast majority of Java developers, so smooth integration with it is key for any tool targeting that constituency.

The Scala IDE for Eclipse 2.0 adds automatic type-checking to ping programmers of errors before they build the code. There is also better dependency tracking between source files.

There is anecdotal evidence to back up Fischer’s contention that Scala is outgrowing its awkward years: the community and number of projects is growing, as evidenced by stats on Github and other sources.

“We think there’s one Scala job for every 100 Java jobs advertised,” said Fischer. “That sounds small, but 1 percent of 10 million is pretty good.”  (The 10 million number is Oracle’s estimate of the Java programmer population.) This is impressive because Scala has only been around since 2003. Fischer said the number of Scala jobs has tripled in the past year.

A quick check of Indeed.com finds that there are 404 listings when “Scala developer” is the search term compared to 44,769 for “Java developer” and 5,841 for Ruby developer.

As the demand for scalable and capable cloud applications continues to grow, it’s likely that the demand for capable — and polished — development tools will mushroom as well.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Open-Source Startups Follow Red Hat’s Path To Profit
• Cloud and data third-quarter 2012
• Continuous delivery and the world of devops