• EMC Greenplum lays down the SQL-on-Hadoop gauntlet: The company’s new Pivotal HD Hadoop distribution fuses its analytic database technology with Hadoop to create a single data store for everything. Greenplum co-founder Scott Yara claims the data warehouse — where Greenplum got its start — is the new mainframe.
• Intel does Hadoop: Intel’s Hadoop distribution is interesting for so many reasons, but the biggest might be the sense that it’s an attempt to keep x86 relevant as ARM pushers pursue big data workloads. Among Intel’s hardware partners are Cray, SuperMicro and Cisco.

• How Friendster died and Facebook might die: Researchers studied the collapse of Friendster and decided that a dimished cost-benefit analysis and users’ average number of friends contributed to its demise. The fewer friends, the more influential one friend’s decision to quit. And people quit when services begin to suck.
• Using memristors to recreate the brain: This is a heady research project based on the theory that memristors are similar enough to synapses in the human brain that they could help create an artificial brain. Memristors are a nanotechnology that allow electrical currents to pass between circuits based on the past currents they have  transmitted.
• MapR and Google in a high-performance lovefest: MapR is all about faster Hadoop, and Google is all about touting how great its Compute Engine cloud is for high-performance job. A MinuteSort benchmark test of MapR on Compute Engine bested the previous record (and crushed the previous Hadoop record for MinuteSort) — and on standard cloud servers, no less.
• LinkedIn open sources Databus: Databus is LinkedIn’s tool for updating changes in data between its various storage systems and applications at high speed. It could be pretty valuable, and I assume it’s something LinkedIn’s Bhaskar Ghosh will discuss during our guru panel at Structure: Data next month.

• Continuuity free beta now open to the public: Continuuity is the startup from former Yahoo VP Todd Papaioannou and Facebook engineer Jonathan Gray that’s building a platform as a service for developing big data applications. On Wednesday, it opened a beta version to developers who want to test the experience of building Hadoop applications on the cloud-based platform.
• Placed Analytics shows who shops in stores but buys online: This is the latest piece of research from Placed, a startup tracking mobile phone data to determine what businesses people like to visit, or at least hang out near. This report highlights which businesses are most at risk from consumers viewing products in their stores and then buying them on Amazon.
• IBM, South Korea and weather predictions:Weather forecasting has always been a good area for big data and high-performance computing, so this use case is pretty much straight data porn. From the press release: “IBM has provided KMA and NMSC with the latest IBM storage technologies capable of recording 20 gigabytes (equivalent to 400,000 web pages) of data per second … [w]ith a total storage capacity of 9.3 petabytes.”
• Virtustream using Druid for cloud analytics service: Virtustream is dead serious about staking its claim as theenterprise cloud provider, and this partnership with Metamarkets (see disclosure) is a good way to expand its reach into big data applications. Essentially, Metamarkets will provide consulting services for companies wanting to build apps atop Hadoop and Druid, the in-memory analytic database that Metamarkets created.

In addition to LinkedIn’s Ghosh, the founders of Placed, Continuuity and Metamarkets will all be on stage at Structure: Data talking about everything from building Hadoop applications, to managing massive data infrastructure to the new era of web privacy, so please come come and watch.

Disclosure: Metamarkets is a portfolio company of True Ventures, which is also an investor in GigaOM. Om Malik is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Cloud and data first-quarter 2013: analysis and outlook
• How search can unlock the power of big data

In a world of mobile apps and cloud resources, it’s easier than ever to start a business around a simple application. Even in large companies, developers fighting for resources might need to prove an application’s popularity or find a way to boost its monetization. Sometimes, that might even mean injecting some data-processing right into an application.

But whatever the case, if your job revolves around writing code rather than data flows, you might need a little help. Here are 12 tools (listed alphabetically) that aim to help. As usual with this type of list, it’s very possible I left out some good options, so please note any omissions in the comments.

1. BitDeli

BitDeli, a startup that launched in November, lets programmers measure pretty much whatever application metrics they want using Python scripts. Co-founder and CEO Ville Tuulos told me at the time that scripts can be as simple or complex as necessary — even going so far as to incorporate machine learning. Compared with the heavyweight Hadoop, BitDeli thinks of itself as the lightweight Ruby on Rails for analytics.

2. Continuuity

The brainchild of former Yahoo Chief Cloud Architect Todd Papaioannou and Facebook HBase engineer Jonathan Gray, Continuuity wants to help all companies operate like its founders’ former employers. The team created a big data fabric that abstracts the complexities of connecting to Hadoop and HBase clusters and includes a full suite of developer tools. The goal is to make it easy to write big data applications serving either internal or external audiences.

3. Flurry

Flurry is like a one-stop mobile-app shop, and it’s generating nearly $100 million a year in revenue because it’s good at what it does. Not only does the company help developers build mobile apps, but helps them analyze all the data those apps are generating in order to make them even better. The data also underpins the company’s ad network that helps developers monetize their apps by putting the right advertisers in front of the right users.

4. Google Prediction API

Of all the tools in Google’s developer toolbox, the Google Prediction API might be the coolest. If you have good data to train a model, the Prediction API can bring machine learning to work on it in order to discern any number of pattern types and feed the answers into your application. Among the examples Google gives are spam detection, recommendation engines and sentiment analysis — and it gives step-by-step instructions for building those models.

“Mucho bueno” is probably Spanish.

5. Infochimps

Although Infochimps is trying hard to make itself an enterprise IT company (hey, that’s where the money is), the company’s eponymous platform also provides a real value for developers. Sitting atop its technologies for configuring and managing big data environments is Wukong, a framework for creating Hadoop jobs or streaming data flows using Ruby scripts. Infochimps also maintains a data marketplace full of API-accessible or downloadable datasets.

6. Keen IO

Keen IO won our Structure 2012 Launchpad competition with a message of delivering powerful analytics to mobile developers. With just a single line of code inserted that dictates what to track, the company claims developers can track pretty much whatever they want within their applications. At that point, it’s just a matter of creating a dashboard or query process in order to turn all that data into usable information.

7. Kontagent

Kontagent‘s bread-and-butter business is its analytics platform for mobile, social and web applications, but it’s all built atop a Hadoop infrastructure designed to handle really big data. Earlier this year, the company turned that infrastructure loose with a new product that lets users mine their application data using the SQL-like Hive query language for Hadoop. Instead of tracking predetermined variables, they can dig in however they choose.

8. Mortar Data

Mortar Data is Hadoop for developers, plain and simple. The company has offered its cloud service — which replaces MapReduce with a combination Pig and Python — for almost a year. In November, it released the open source Mortar framework in order to build a community around sharing datasets and making it easier to write Hadoop pipelines. Mortar Data runs atop Amazon Web Services and currently supports Amazon S3 and MongoDB (hosted on Amazon EC2) as data sources.

9. Placed Analytics

Placed does away with scripts, APIs and any other developer legwork and just delivers the results. In the case of Placed, those results are detailed information about where and when consumers are actually using mobile apps and web sites — right down to the name of the business. This type of info can be useful for attracting advertisers as well as informing app design (e.g., implementing voice controls if people are using an app while driving).

10. Precog

Precog might look like any other proprietary business intelligence service, but underneath its covers there’s a twist. The company offers a service called Labcoat, which is an interactive development environment for writing analytics jobs based on the open source Quirrel query language. The IDE includes a tutorial for learning the language, as well as some complex functions, and Precog COO Jeff Carr told me even non-technical people can learn it in hours.

11. Spring for Apache Hadoop

Hadoop is written in Java, but that doesn’t mean it’s easy for Java developers to learn or use. That’s why, in early 2012, SpringSource announced the Spring for Apache Hadoop project, which brings the ease of building Java applications with the Spring framework to Hadoop jobs. That means integration with other Spring apps, scripting using JVM-based languages and a generally easier way to develop applications that utilize Hadoop or related technologies such as Hive or HBase.

12. StatsMix

In the same vein as BitDeli and Keen IO, StatsMix wants to let developers start collecting and analyzing application data using the languages they already know. The service automatically tracks certain metrics, but developers can add their own using the StatsMix API and predefined code libraries. The results are delivered via a collection of dashboards that users can customize, share and use to mashup multiple data sources into a single view.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The cash infusion will help Palo Alto, Calif.-based Continuuity further develop technology that abstracts the complexity of building and deploying Hadoop applications and also opens Hadoop up to entirely new types of workloads. Although Continuuity’s AppFabric is built atop Hadoop and a collection of related technologies, it lets developers build what might be called big data applications, generally, rather than just Hadoop (read “inherently limited MapReduce“) applications. Taking advantage of YARN and HBase, for example, the company was able to build out methods for processing stream data in real time and elastically scaling a Hadoop cluster even while jobs are running.

The beauty of it all, though, are the high-level APIs, development environment and abstraction from the infrastructural complexity under the covers. Even though Hadoop, HBase and other foundational elements might be running on their own individual clusters and be linked together with a set of low-level APIs, Papaioannou says developers shouldn’t have to worry at all about nodes, clusters, data blocks or anything else that keeps engineers up at night.

This is the same idea driving the growing ecosystem of web-app-focused Platform-as-a-Service offerings such as Heroku, Cloud Foundry and Windows Azure, as well as a growing collection of data-focused services. Continuuity actually plans to join many of them in the cloud next year by launching a publicly hosted version of AppFabric to complement the on-premises version that’s currently in private beta.

That and other engineering projects are where this $10 million investment comes in. The seven-man Continuuity team — which helped build some of the largest Hadoop and HBase deployments in the world at Yahoo and Facebook — knows its way around big data, and there’s a lot of interest in the product, Papaioannou said. “But,” he added, “we know there’s a ton of work we need to do.”

Fellow co-founder Gray, whose last gig involved building Facebook Messages, said there’s nothing less than the fate of the Hadoop market riding on whether Continuuity and its brethren can get the technology out of the weeds and into the hands of developers. “If there are no applications … then the market is going to go away,” he said. “Fundamentally, people who pay for infrastructure really want to pay for apps.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010
• Sector RoadMap: SQL-on-Hadoop platforms in 2013

The company calls its product the AppFabric, and it’s essentially a software development kit and a set of high-level APIs sitting above a fabric layer that connects to all of a company’s various Hadoop clusters. Co-founder and CEO Todd Papaioannou, former entrepreneur in residence at Battery Ventures and VP and chief cloud architect at Yahoo, says Continuuity wants to make it easy for developers at fast-follower companies to build big data applications without having to learn Hadoop’s low-level APIs and the general complexity of the distributed framework. Very few people, he said, really want to be part of the ”home-brew computing club experience.”

In some ways, Continuuity is trying to do for big data what Heroku and other PaaS providers have done for web applications. There’s a drag-and-drop user interface, an integrated developer environment to make development easy on a laptop before pushing an application into production, and resources scale elastically.  Papaioannou said developers get a UI for DevOps, as well, and can connect Continuuity apps with server-side web apps wherever they’re hosted, even on other PaaS offerings.

Presently, Continuuity is available in private beta for its single-node developer version and private cloud version. It expects a public cloud version will be available at some point in 2013. The platform currently supports Java applications only — a natural first step given that Hadoop is written in Java — but Papaioannou said Continuuity plans to support other languages going forward.

Papaioannou’s fellow co-founder, Jonathan Gray, who was integral in building Facebook’s Messaging and Puma analytics applications on top of the Hadoop-based HBase database, says even early big data adopters might be interested in what Continuuity is selling. For example, he explained, a company like Facebook will have separate clusters for data warehousing, for HBase and for caching. A product like AppFabric would abstract the differences between those various systems and allow developers to easily write applications connecting to any or all of them.

Even if they’ve made it easy for non-engineer developers to write batch-processing jobs (like Facebook has with Hive for Hadoop), both Gray and Papaioannou say large companies can have hard time giving developers even more functionality. AppFabric might let developers create streaming or transactional apps, too, without requiring a whole new set of internally developed tools.

Really, Papaioannou says, the idea behind a PaaS for big data is no different than that behind platforms as a service, generally. The key question is: “Can a programmer sit down in an afternoon and develop something he can take to his boss?” Any company that can really make this happen for Hadoop — be it Continuuity, Infochimps, Mortar Data or whomever — should be in a good position to grow. Hadoop skills are in high demand but short supply, and there are plenty of developers waiting to get their hands on all that data-processing power.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The importance of putting the U and I in visualization
• A near-term outlook for big data

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Papaionnou calls this unstructured data “digital exhaust,” everything consumers do on a daily basis — clicks, tweets, searches, Facebook posts. Companies can use it to offer more customized experiences for consumers online – content and deal targeting, advertising and sentiment analysis — but they have to process it first.

“Who cares if you’re able to get data signals if you can’t act on it?” Papaiaonnou said, noting Hadoop isn’t real time. “To get to real time you have to take the human out of the equation and [start] allowing machines to make these decisions.” He pointed to initiatives like Yahoo’s S4 and Twitter’s Storm as ones to watch.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

Big data is hot, but infrastructure-level platforms such as Hadoop, which focus on storage and processing, still need help to take them into the mainstream. They need a killer app or two that will let companies analyze, visualize and act on all that data without hiring a team of Stanford Ph.Ds, or that will let developers write big-data apps without having to reinvent the wheel.

Here are five startups (in alphabetical order) either in stealth mode or just out of it that could help take Hadoop and its ilk to the promised land.

1. BloomReach

The stealth-mode BloomReach is taking a very targeted, very hands-free approach to big data for its customers. It’s offering a SaaS-based product that job listings say is for “helping leading online businesses uncover the highest quality, most relevant content sought by their consumers, when and where they want it.” Founded by a team with roots at Google, Cisco, Facebook and Yahoo, among other companies, BloomReach has, according to one estimate, about 160 customers — all of them among the top 10,000 websites, and most of them in the retail space. Among its core technologies and methods are Hadoop, Lucene, Monte Carlo simulations and large-scale image processing.

2. Continuuity

Continuuity, the just-launched stealth-mode startup by former Yahoo VP and chief cloud architect Todd Papaioannou, wants to make it easier to build applications that can leverage both cloud computing and big data technologies. As Papaioannou told me recently, most developers shouldn’t have to go through what Yahoo, Facebook and others did in order to write large-scale, data-driven applications. He also said “the data fabric is the next middleware” and noted that the company name is a play on “continuum.” You figure out what it’s up to.

3. Odiago

Odiago is the brainchild of Hadoop and analytics experts Christophe Bisciglia and Aaron Kimball, and aims to improve the state of web analytics. Its first product, Wibidata, which is in private beta, lets websites better analyze their user data to build more-targeted features. It’s built atop Hadoop and HBase, but also plugs into companies’ existing data-management and BI tools. Current customers include Wikipedia, RichRelevance, FoneDoktor and Atlassian (with whom it shares office space).

4. Platfora

Platfora, which launched in September with $5.7 million in funding, wants to make big data analytics accessible to the masses. Founder and CEO Ben Werther, formerly of Greenplum and NoSQL startup DataStax, told me when Platfora launched that its intuitive, visually stunning interface will make Hadoop-based analytics so easy even a history major could use it. Platfora’s product isn’t available yet, but the company is currently hiring, with an emphasis on frontend and user-experience skills.

5. SkyTree

Skytree is probably the stealthiest of the group, but it’s also is one of the more ambitious — because it’s trying to bring high-performance machine learning to mainstream companies. Machine learning is an impressive technique in which the system itself gets smarter as it digests more data, but it usually doesn’t find its way out of research environments or cutting-edge analytics teams. Skytree is putting together an impressive team, including co-founder Alexander Gray, who also teaches machine learning at Georgia Tech and spent six years at NASA’s Jet Propulsion Laboratory. The company will officially launch later this quarter.

We’ll be addressing many of the issues these companies are trying to resolve at our Structure: Data event that takes place March 21-22 in New York City. Founders from Continuuity, Odiago and Skytree will be speaking at the event, as will dozens of other data visionaries from companies such as IBM, Google, @WalmartLabs and Hortonworks.

Feature image courtesy of Flickr user jurvetson.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• How data warehousing is now a cost-effective solution for businesses

Todd Papaioannou at Structure 2010 cocktail event at GigaOM offices. Photo by Om Malik

Former Yahoo VP and chief cloud architect Todd Papaioannou is working on a stealth-mode startup called Continuuity, which aims to make it easier for developers to create data-focused cloud applications. The company is based in Palo Alto, Calif. and has raised a $2.5 million seed round from Battery Ventures, where Papaioannou had been serving as an entrepreneur in residence, and Andreessen-Horowitz.

Continuuity is keeping details on exactly what it’s doing vague for the time being, but Papaioannou, who has been working in the big data space for about a decade at Teradata, Greenplum and Yahoo, said he thinks “cloud and big data are the two megatrends of this decade.” They will shape the way companies build infrastructure and handle data, he said, and they’re leading to an era where “the data fabric is the next middleware.”

Continuuity wants to help new developers build apps that can leverage those trends, but with less difficultly than Yahoo, Facebook and other fast followers experienced in building out their massive applications. Papaioannou said he’s put together a great team, too, which brings together folks from Yahoo, Facebook, IBM, Teradata and Greenplum, among other places.

The company will formally unveil itself in March.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future