Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Cloud computing infrastructure: 2012 and beyond
• The future of mobile: a segment analysis by GigaOM Pro

SpaceCurve has been pretty silent since its inception, presumably because it’s busy building out a product that has to convince businesses to build applications or run analytics in ways they might not have even seriously considered before. By digesting myriad streams of data from mobile devices, sensors, social media and anything else spitting out data about time or places, the company claims on its website it can ”tie people or entities to a precise point in time and space, immediately discover the social and semantic relationships between them, and deliver this  real-time intelligence instantly to identify new opportunities, and support better decisions and more profitable actions.”

Digging through the marketing lingo, one can spot the opportunities SpaceCurve could help solve if it works as advertised. Rather than just knowing where someone or something is at any given time, you could know how they’re related to the other things around them and perhaps figure out who’s who among a sea of anonymous devices. Marketers trying to master geo-targeting might like these capabilities, as might intelligence agencies trying to, I don’t know, comb through metadata from millions of call records.

SpaceCurve closed a $3.5. million Series A round in August 2012 and has raised $15.2 $17.3 million overall since it was founded in 2009.

Feature image courtesy of Shutterstock user twobee.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in Giga Omni Media, the company that publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Since then, Accel has put a name on the types of companies it’s seeking to fund with the new allocation — namely, those selling what it calls “data-driven software.” That’s a fancy way of saying that it’s not looking to fund infrastructure-level software such as Hadoop or NoSQL databases, but rather software that leverages these technologies and others in order to make analytics simpler. It wants to fund startups targeting business users rather than data scientists.

Accel Partner Ping Li (second from right) at Structure 2011. (c) Pinar Ozger

This type of company isn’t too difficult to come by anymore. Just about everywhere you look, someone is trying to put a big data spin on an old problem or invent some new methods for doing business intelligence. Accel has recently funded a number of them including RelateIQ, Opower, Sumo Logic  and Causata. Among the non-Accel-funded startups GigaOM has covered in just the past few months are Ayasdi, Wise.io, Spinnakr, Statwing and BloomReach.

All this interest in data-driven software is no doubt inspired by the proven utility and wildly successful initial public offerings by enterprise data software companies such as Splunk and Tableau. Entrepreneurs can see the value in rethinking legacy business software or processes for the era of big data and cloud computing, and investors have dollar signs in their eyes as they try to get a piece of the most-promising companies.

As with all trends, much of this startup and investing activity will prove to be overkill, but there’s no denying the promise that the right products have for everyone involved. Businesses really are hurting for better ways to make sense of all the data they’re generating and being exposed to, and they’ll pay handsomely to software vendors that can solve the problem.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Big data budgets on the rise
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

It might sound futuristic, but the techniques behind Corvix have actually been around for a while. The platform uses agent-based modeling to build, essentially, a reasonable facsimile of some sort of complex system and then simulate its evolution over time. The “agents” represent the atomic units of those systems, such as individual people in the case of human populations or perhaps cells in the case of a biological simulation. They act according to a set of rules in any given situation, which is how the models are able to keep the simulations progressing.

However, thanks to the advent of big data, GE Healthcare Chief Economist Mitch Higashi thinks the time is right for a platform like Corvix to provide some real value to real-world decisions. There’s enough raw computing power, machine intelligence and data-modeling expertise to start doing fast, accurate simulations over very large and complicated datasets. Also, advances in user-interface design have made these types of models more consumable: GE’s Corvix uses a game-like UI “that any 10-year-old can figure out how to use in 10 minutes,” Higashi said.

In the field in India

The first live run for Corvix happened in the state of Andhra Pradesh in India, where the system simulated a population of 80 million people in order to figure out where to build hospitals and medical training centers over the coming years. The GE team used two census datasets and one health care survey in order to build what Higashi calls “a reasonable representation of 80 million people,” as well as a map of India’s existing hospital and energy grid. Health care analysts studying the problem of where to build can drag a new hospital over an area on the map and see how the situation plays out, Higashi explained.

The original plan, said Chaitanya Sarawate, GE’s head of health economics and reimbursement for India, was for the Public Health Foundation of India to invest $2 billion building training institutions in different cities over the next five years. Corvix suggested some possible changes in location of those institutions, including placing two institutions in the country’s most-populous state, Uttar Pradesh, instead of just one as was originally planned. The advice is part of a report from the foundation to India’s Minstry of Health, which will make the ultimate decision.

Developing countries such as India are great places to use this type of technology, Higashi explained, because they are doing greenfield investing in areas such as health infrastructure and a lot of good can happen if they get it right off the bat. The problem, Sarawate noted, is that they often lack detailed data that can help governments make objective comparisons — that’s the kind of stuff a company like GE, in this case, can track down and try to feed into a model that takes into account its relative importance.

In fact, GE is already working on projects with other governments in India, as well as with private organizations and governments in other developing countries.

More data, better simulations

GE has bigger plans for Corvix, though, including deployments in countries like the United States, and possibly into different areas within health care and some outside of health care. Ruslan Horblyuk, director of health economics for GE Healthcare, said deploying Corvix in developed economies will probably be a bit easier because there’s often more data available to choose from when building the models.

“You do need a lot of data, there’s no doubt about it,” he explained, but you also need the right data to avoid making the models too simplistic on the one hand or too complex on the other. So, the more data available on specific diseases, for example, the deeper GE could go on modeling its progression over time — maybe across certain demographic groups or down to the level of a single organ. If exact data (e.g., literacy or disease state) aren’t available for people, there might be proxy data points available that could serve the same purpose.

Independent agents act on their own.

Horblyuk points to the U.S. Affordable Care Act (aka Obamacare) as a good example of where Corvix could provide value. As hospital chains grow larger but also try to improve their efficiency, an agent-based model could help them figure out how to make the best use of their available infrastructure. Maybe that means targeting certain illnesses in certain geographies, or moving from general-purpose facilities to facilities focused on treating specific conditions.

Assuming governments and other stakeholders are willing to trust the data at least in part, Higashi sees an even broader applicability for the types of simulations GE is enabling via Corvix. Like all things big data, the promise with simulations is about speed. Compared with traditional methods of making policy decisions, which might take years of data gathering and analysis, data-based simulations can produce numerous forecasts in mere weeks or months and can be iterated upon as new data becomes available. The forecasts might not be entirely accurate, Higashi acknowledges, but “we can tell you the likely path the country’s on.”

“One day, all policy will be developed in the digital world before being implemented in the real world,” he said. “… There is a not a better way to plan your resources for the future. … (If you have $400 or $500 million to build hospitals), you don’t get to do a redo.”

Feature image courtesy of Shutterstock user discpicture.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• The importance of putting the U and I in visualization

The Yahoo work is actually pretty important. Among the features Hadoop users have been demanding from the platform is a transition from batch-processing-only mode into something that can actually deal with data in real time. The reason for the demand is quite simple: Although being able to analyze or transform data minutes to hours after it’s generated is helpful for certain analytic tasks, it’s not too helpful if you want an application to be able to act on data as it hits the system.

A service like Twitter is a prime example of where Storm can be valuable. Twitter uses Storm to handle tweets so users’ Timelines are up to date and do things like real-time analytics and spotting emerging trends. In fact, it was Twitter that open sourced Storm in 2011 after buying Storm creator Backtype in order to get access to the technology and its developers.

Among web companies, Storm has become quite popular as a stream-processing complement to Hadoop since then. And now Yahoo has made possible a much tighter integration between the two — even to the point that Storm can borrow cycles from batch-processing nodes if it needs some extra juice. That’s a valuable feature — just last week I heard Twitter engineer Krishna Gade bemoan Storm’s auto-scaling limitations during a talk at Facebook’s Analytics @ Web Scale event.

Krishna Gade talking Storm at the Facebook event.

The Storm-on-Hadoop work is among the first of many promised improvements to come thanks to YARN, a major update to the Apache Hadoop 2.0 code that lets Hadoop clusters run multiple processing frameworks simultaneously. Twitter has been using the open source Mesos resource manager to achieve the same general capabilities, but Gade’s colleague Dmitriy Ryaboy said during the same talk that the company plans to begin using YARN for some big data workloads when it upgrades to Hadoop 2.0. He expects — probably correctly — much more community effort will go toward continuously improving its capabilities and building applications for YARN.

Facebook’s Wormhole project isn’t open source (as far as I can tell), but its lessons are still valuable (and LinkedIn has open sourced a similar technologies named Kafka and Databus). It’s what’s called a publish-subscribe system, which is essentially a concise way of saying that it manages communications between applications that publish information (e.g., updates to a database) and subscribe to the information their fellow applications are publishing. At Facebook, for example, Wormhole sends changes to Facebook’s master user database to Graph Search so that search results are as up to date as possible, or to its Hadoop environment so analytics jobs have the newest data.

Of course, like all things Facebook (its new Presto interactive query engine comes to mind), Wormhole is built to scale. Latency is in the low milliseconds and, blog post author Laurent Demailly notes

“Wormhole processes over 1 trillion messages every day (significantly more than 10 million messages every second). Like any system at Facebook’s scale, Wormhole is engineered to deal with failure of individual components, integrate with monitoring systems, perform automatic remediation, enable capacity planning, automate provisioning and adapt to sudden changes in usage pattern.”

Although they were developed within separate companies, there’s actually a tie that binds Yahoo’s Storm-in-Hadoop work and Facebook’s Wormhole. As web companies grow from their initial applications into sprawling business composed of numerous applications and services, so too do their infrastructures. To address the differing needs of their various systems at the data level, the companies have begun breaking them down by their latency requirements (i.e., real-time, near real-time and batch, however they choose to word them) and then building tools such as Storm and Wormhole to manage to flow of data between the systems.

We’ve previously explained in some detail how LinkedIn and Netflix have built their data architectures around these principles, and we’ll hear a lot more about how they and other web companies are tackling this situation at Structure next week. Among the speakers are senior engineers and technology executives from Facebook, Google, LinkedIn, Box, Netflix and Amazon.

Update: This post was updated at 1:46 p.m. to clarify that Twitter is not eliminating Mesos for all its workloads. 

Feature image courtesy of Shutterstock user agsandrew.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Altiscale is in some ways a manifestation of Stata’s seven years of experience helping turn Hadoop from a cute little project into a production system running across 42,000 nodes. It might not be not pretty, but it gets the job done. And, thanks to the handful of former senior Yahoo, Google and LinkedIn engineers that joined Stata (who’s the company’s CEO) at Altiscale, the company knows Hadoop cold.

Team Altiscale (State is middle row, second from left). Source: Altiscale

The deep knowledge of Hadoop shows itself in the product design and business model. The company is “all Hadoop, all the time,” he explained, and everything — including the hardware and the network — is optimized for particular aspects of Hadoop workloads and operations. Essentially, Stata told me, Altiscale wants to be companies’ Hadoop dial tone — when users need to run a job, the service should just be there ready to do it.

So, although Altiscale is a hosted service, it’s not exactly a cloud service as many people would define it. Rather than charge by the hour, for example, Stata’s experience suggests Hadoop services are best charged based on a monthly baseline usage with room even built in for reasonable overages. This is because companies familiar with Hadoop usually understand their baseline requirements, give or take a handful of additional jobs, and would prefer to be able to budget for that each month.

He compares traditional hourly cloud billing to cell-phone billing in the 1990s: “At the end of the month,” he joked, “you were typically surprised on the wrong side.” Altiscale is more like a wireless plan with a maximum amount of minutes per month and some rollover minutes included. In fact, Stata said,  ”We’re pretty forgiving in terms of the limits. … As long as you’re not abusive, you don’t get charged more for it.”

And unlike many other Hadoop services, Altiscale isn’t immediately going after developers who want to try their hand at big data or deal with data through a wizbang interface. Rather, its initial audience is current Hadoop users — companies and data scientists — who know how the technology works but just want a better way to consume it. Right now, users access Altiscale by SSHing into a “desktop” environment (that’s actually hosted on Amazon Web Services) that gives them access to their favorite Hadoop tools such as MapReduce, Hive, Pig and Flume, as well as to data science tools such as R.

“We call that the scaling down problem,” Stata said.

What that means is that it takes a lot of effort to build a true self-service model that greenhorn Hadoop users can dive right into, and Altiscale would be irrelevant if waited to launch until it had figured that out. Part of that is a design problem, and part of that is a matter of Hadoop being designed to run better at scale. Plus, Stata added, the folks who got to first or second gear with Hadoop and then got stuck are way underserved right now.

However, although Altiscale might be about serving experienced Hadoop users with a more-managed experience, it’s not about serving legacy workloads. A lot of companies are using Hadoop today to somehow perform traditional enterprise data warehouse tasks or tie tightly into existing IT environments, he explained, but “we go after what I call ‘new data problems.’” That means online advertising and any workloads — servers log analysis, smart grid data, logistics, etc. — relying heavily on lots of sensor- or machine-generated data that can stream right into Hadoop.

Stata acknowledges it won’t be easy trying to win customer away from established Hadoop vendors such as Cloudera, MapR and Hortonworks (which many of Stata’s former Yahoo comrades founded), but, he told me a few months ago, he thinks its very doable. That’s because no matter how easy they make it to manage Hadoop, there’s a class of customers that’s just better served with a cloud service rather than trying to scale their operations staff and energy bill along with their Hadoop cluster.

“Self-managed Hadoop, essentially, is [those vendors'] ultimate goal,” Stata said. “Our goal is to to just take on the management responsibility, to take on all those management things the Yahoos and Googles do under the covers and just run Hadoop as a managed service. The winds of change are in our favor.”

If you want to hear more about where Hadoop is head, stop by our Structure conference next week, where I’ll be discussing that topic with Google Fellow and MapReduce creator Jeff Dean. Other webscale speakers include Facebook VP of Engineering Jay Parikh, Box VP of Engineering Sam Schillace and Amazon CTO Werner Vogels.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• 2012: The Hadoop infrastructure market booms

Search, advertising, Translate, Play Music, Goggles, Trends and the list goes on — they’re all products that couldn’t exist without lots of data. But data alone doesn’t make products great — they also need to perform fast and reliably, and they eventually need to get more intelligent. Infrastructure and systems engineering make that possible, and that’s where Google really shines.

On Wednesday, the company showed off its chops once again, explaining in a blog post how it’s able to let users better search their photos because it was able to train some novel models on systems built for just that purpose. Here’s how Google describes the chain of events, after it had found the methods it wanted to test (from the winning team at the ImageNet competition):

“We built and trained models similar to those from the winning team using software infrastructure for training large-scale neural networks developed at Google in a group started by Jeff Dean and Andrew Ng. When we evaluated these models, we were impressed; on our test set we saw double the average precision when compared to other approaches we had tried. …

“Why the success now? … What is different is that both computers and algorithms have improved significantly. First, bigger and faster computers have made it feasible to train larger neural networks with much larger data. Ten years ago, running neural networks of this complexity would have been a momentous task even on a single image — now we are able to run them on billions of images. Second, new training techniques have made it possible to train the large deep neural networks necessary for successful image recognition.”

Of course Google had a system in place for training large-scale neural networks. And of course Jeff Dean helped design it.

Google’s system can recognize flowers even when they’re not in the focal point.

For me, Dean is among the highlights of our upcoming Structure conference (June 19 and 20 in San Francisco). I’m going to sit down with him in a fireside chat and talk about all the cool systems Google has built thus far and what’s coming down the pike next. Maybe about what life is like being the Chuck Norris of the internet.

From an engineering standpoint, Dean has been one of the most important people in the short history of the web. He helped create MapReduce — the parallel processing engine underneath Google’s original search engine — and was the lead author on the MapReduce paper that directly inspired the creation of Hadoop. Dean has also played significant roles in creating other important Google systems, such as its BigTable distributed data store (which is the basis of NoSQL databases such as Cassandra, HBase and the National Security Agency’s Accumulo) and a globally distributed transactional database called Spanner.

If you’re into big data or webscale systems, knowing what Dean is working on can be like looking into a crystal ball. When I asked Hadoop creator Doug Cutting what the future holds for Hadoop, he told me to look at Google.

“They send us messages through these technical papers,” Cutting said, “so we can see what’s coming.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Using desktop drives and selling at petabyte-scale might seem insane, but SageCloud is trying to solve a very real problem. Companies of all sorts — Facebook most publicly, perhaps — are trying to figure out how to economically store the huge amounts of data they need to for regulatory purposes or want to for analytic or other purposes. The one caveat, of course, is that they can’t resort to tape because they might still actually want to access and use that data without all the waiting and other work.

Facebook is still looking for the ideal medium to solve its cold storage problem (it’s even considering Blu-ray), but it has built a special section of data designed to house this data that its applications no longer need on a regular basis. Amazon Web Services is offering disk-based archival as a service via its Glacier service. Some companies and institutions that want to host their data on site have turned to Backblaze’s storage pods, a desktop-drive-based storage system designed and then open sources by the cloud backup provider. (The Central Intelligence agency has even shown interest.)

(And if you come to our Structure conference next week in San Francisco, you can hear Facebook VP of Engineering Jay Parikh, Amazon CTO Werner Vogels and Backblaze CEO Gleb Budman talk all about their approaches to solve big, slow data on the cheap.)

So it’s only natural that a company like SageCloud would step in to fill the void for organizations that want to keep data on-premises but don’t want to build something themselves. It’s promising high reliability, low energy consumption and even lower cost. Maybe it’s even more natural that its founder, Jeff Flowers, would spot the opportunity: He was previously co-founder and CTO of cloud backup service Carbonite and likely has seen how high costs can creep even into backup once you reach a certain scale.

It might not be the highest-margin business around — SageCloud obviously eschews the high-performance drives and fancy software that drive enterprise storage costs through the roof — but that’s probably alright if there are customers willing to buy. This business is all about volume, and a petabyte apiece is nothing to scoff at.

SageCloud launched in August 2012 with $3.2 million in venture capital from Matrix Partners. Braemar Energy Ventures led the Series B round, with Matrix also pitching in.

Feature image courtesy of Shutterstock user kubais.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Finding the Value in Social Media Data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout
• Connected world: the consumer technology revolution

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Forecasting the future cloud computing market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight