Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Cloud computing infrastructure: 2012 and beyond
• The future of mobile: a segment analysis by GigaOM Pro

It might sound futuristic, but the techniques behind Corvix have actually been around for a while. The platform uses agent-based modeling to build, essentially, a reasonable facsimile of some sort of complex system and then simulate its evolution over time. The “agents” represent the atomic units of those systems, such as individual people in the case of human populations or perhaps cells in the case of a biological simulation. They act according to a set of rules in any given situation, which is how the models are able to keep the simulations progressing.

However, thanks to the advent of big data, GE Healthcare Chief Economist Mitch Higashi thinks the time is right for a platform like Corvix to provide some real value to real-world decisions. There’s enough raw computing power, machine intelligence and data-modeling expertise to start doing fast, accurate simulations over very large and complicated datasets. Also, advances in user-interface design have made these types of models more consumable: GE’s Corvix uses a game-like UI “that any 10-year-old can figure out how to use in 10 minutes,” Higashi said.

In the field in India

The first live run for Corvix happened in the state of Andhra Pradesh in India, where the system simulated a population of 80 million people in order to figure out where to build hospitals and medical training centers over the coming years. The GE team used two census datasets and one health care survey in order to build what Higashi calls “a reasonable representation of 80 million people,” as well as a map of India’s existing hospital and energy grid. Health care analysts studying the problem of where to build can drag a new hospital over an area on the map and see how the situation plays out, Higashi explained.

The original plan, said Chaitanya Sarawate, GE’s head of health economics and reimbursement for India, was for the Public Health Foundation of India to invest $2 billion building training institutions in different cities over the next five years. Corvix suggested some possible changes in location of those institutions, including placing two institutions in the country’s most-populous state, Uttar Pradesh, instead of just one as was originally planned. The advice is part of a report from the foundation to India’s Minstry of Health, which will make the ultimate decision.

Developing countries such as India are great places to use this type of technology, Higashi explained, because they are doing greenfield investing in areas such as health infrastructure and a lot of good can happen if they get it right off the bat. The problem, Sarawate noted, is that they often lack detailed data that can help governments make objective comparisons — that’s the kind of stuff a company like GE, in this case, can track down and try to feed into a model that takes into account its relative importance.

In fact, GE is already working on projects with other governments in India, as well as with private organizations and governments in other developing countries.

More data, better simulations

GE has bigger plans for Corvix, though, including deployments in countries like the United States, and possibly into different areas within health care and some outside of health care. Ruslan Horblyuk, director of health economics for GE Healthcare, said deploying Corvix in developed economies will probably be a bit easier because there’s often more data available to choose from when building the models.

“You do need a lot of data, there’s no doubt about it,” he explained, but you also need the right data to avoid making the models too simplistic on the one hand or too complex on the other. So, the more data available on specific diseases, for example, the deeper GE could go on modeling its progression over time — maybe across certain demographic groups or down to the level of a single organ. If exact data (e.g., literacy or disease state) aren’t available for people, there might be proxy data points available that could serve the same purpose.

Independent agents act on their own.

Horblyuk points to the U.S. Affordable Care Act (aka Obamacare) as a good example of where Corvix could provide value. As hospital chains grow larger but also try to improve their efficiency, an agent-based model could help them figure out how to make the best use of their available infrastructure. Maybe that means targeting certain illnesses in certain geographies, or moving from general-purpose facilities to facilities focused on treating specific conditions.

Assuming governments and other stakeholders are willing to trust the data at least in part, Higashi sees an even broader applicability for the types of simulations GE is enabling via Corvix. Like all things big data, the promise with simulations is about speed. Compared with traditional methods of making policy decisions, which might take years of data gathering and analysis, data-based simulations can produce numerous forecasts in mere weeks or months and can be iterated upon as new data becomes available. The forecasts might not be entirely accurate, Higashi acknowledges, but “we can tell you the likely path the country’s on.”

“One day, all policy will be developed in the digital world before being implemented in the real world,” he said. “… There is a not a better way to plan your resources for the future. … (If you have $400 or $500 million to build hospitals), you don’t get to do a redo.”

Feature image courtesy of Shutterstock user discpicture.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• The importance of putting the U and I in visualization

After Apple CEO Tim Cook himself gave them an introduction on stage at the WWDC conference in San Francisco on Monday, Anki CEO Boris Sofman (pictured) and a colleague demonstrated how iOS devices can not only control the little cars but also serve as “the brains behind an immersive, real-world experience.”

The cars whizzed around a small track on stage. One car deployed a weapon and shot the other cars off the track, and multiple cars were influenced by the driving behavior of one car. Sensors track where the vehicles are and communicate information back via low-power Bluetooth, and the system determines a long list of options for the next action to take. Watch the video here.

The whole game will become available in the fall, for around $200.

The company wants to go beyond gaming, though. “We are a robotics and artificial-intelligence company, and what you see here is only the beginning,” Sofman said. It’s unclear if the company will go beyond the consumer space to also provide products for businesses, although that certainly seems possible. Sofman’s bio states that he’s worked on “off-road autonomous vehicles and bomb-disposal robots.”

Either way, investors seem to like the company’s prospects. Andreessen Horowitz, Index Ventures and Two Sigma have backed the company with around $50 million, the company announced in a Monday statement.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Forecast: Tablet App Sales To Hit $8B by 2015
• Connected consumer first-quarter 2013: Analysis and outlook
• Flash analysis: smart watches

That first RE.WORK summit will take place on 19 September, which is also the second day of GigaOM’s Structure:Europe conference in London. The RE.WORK program will kick off with a focus on the areas of: the internet of things, 3D printing, nanotech, artificial intelligence, robotics, computing systems and sensors.

“We’re trying to showcase emerging technologies and breakthrough ideas,” summit founder Nikita Johnson told me. “It’s all about reworking big challenges that we’re facing in the future. We want to bring the technology and science aspect, but with mission of positive impact.”

While the first of these summits will focus on technology, others will have different themes: one in December will deal with urbanization, RE.WORK Health will take place next year, and still others will handle energy, education and the environment. The first three meetings will take place in London, with others set for Dublin, Berlin and other European cities. Events will also be followed up with smaller meet-ups, Johnson added.

Apart from Solve For X, other initiatives in this space include TED, to a certain extent (RE.WORK looks to be a bit more collaborative and interactive) and the engineering-led Global Grand Challenges Summit.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

When you’re in Atlanta, as I was during a recent trip, the obvious place to start talking about orange juice and data is with Coca-Cola. Foreman can tell you all about how the beverage giant — whose headquarters tower over the city just a just a mile away from MailChimp’s office — uses advanced algorithms and giant vats of different juices to ensure the proper flavor of its Simply Orange line of orange juice. However, it’s something else Coca-Cola is doing that inspired the way Foreman thinks about data and that’s helping MailChimp re-imagine what it means to engage with fans, readers and customer through their inboxes.

Anyone familiar with how large web companies came to pioneer the practice of what we now call “big data” should appreciate the analogy. Coca-Cola, which also owns Minute Maid, produces a lot of excess pulp when it makes orange juice. For decades, presumably, it had just been throwing that pulp away, but in 2006 it decided to make use of it by launching a new product called Minute Maid Pulpy. Sold primarily in Asian countries, Pulpy has become a billion-dollar business for Coca-Cola.

Once MailChimp is done with its primary business of sending emails, it has a lot of pulp of its own in the form of data. And rather than just ignoring it or writing up some cute blog posts (which he also does), Foreman and his bosses want to turn that data into revenue.

First things first: Making better orange juice

Neil Bainton

Actually, though, MailChimp first brought in Foreman in 2011 to help the company improve its core business of letting users build and send their emails. MailChimp’s culture was built around many things, COO Neil Bainton told me, but data wasn’t one of them. It had “various fits and starts” through the years trying to work data into its business model, and each step just added more complexity.

The challenges were technological as well as cultural, but Foreman had a plan, of which focus was a key aspect. Keeping a tight focus meant Foreman and his lone-developer sidekick could build what they needed to in a short timeframe. It also meant the company didn’t have to worry about some massive overnight transformation into a data-obsessed company like Google.

John Foreman

“[They] don’t need to be afraid the entire culture is gonna fall down if we bring in this weird math guy,” he joked.

Foreman’s first project — deploying artificial intelligence models that would automatically detect spammy email lists from MailChimp’s users – is actually critical to the way MailChimp operates, though. It was up and running in production within a year, after a technologically challenging effort of merging separate database instances for each customer into a single environment that would let MailChimp run complex analyses across its customer base.

It’s such an important project, Foreman explained, because internet service and email providers keep reputation scores on the IP addresses that send email through their systems. Because MailChimp serves as the email engine for its millions of users, sending too many messages that get flagged as spam and lower MailChimp’s reputation will have a negative impact on everyone. The company used to deal with spam manually, and only after recipients began complaining about the messages they received.

“It used to be before we had that AI model in place that everyone had a crappier experience,” Foreman said.

Say goodbye to those ’90s fans, Pearl Jam

Source: MailChimp

Now, however, MailChimp knows some of the telltale signs of spam for which it should be on the lookout. If too high a percentage of email addresses on a given list are also available via publicly available lists or those you can buy on sketchy corners of the internet, it’s probably spam. Too many old and far-more-likely-to-be-dead Earthlink or Compuserve addresses, or letters within one keystroke of each other as if someone just mashed the keyboard? Probably spam.

Thankfully, though, about 98 percent of the spam that MailChimp identifies is what Foreman calls “ignorant” — that is, people or companies that just don’t know the laws or best practices around sending emails. But ignorance doesn’t mean MailChimp relaxes its rules. Recently, it even flagged Pearl Jam for spammy practices because the band was trying to reconnect with old fans whose email addresses read like a who’s who list of 1990s email providers.

Having such a high percentage of ignorant spam actually has a positive effect on the company’s overall goal of monetizing its vast data repositories. Because the AI model automates what used to be a manual process, and because most innocent spammers will fall in line quickly once they’re notified (as opposed to nefarious spammers who constantly try to outsmart the system), MailChimp can pretty much set the model loose, forget about it and get to work on new efforts, Foreman said.

Now, about that pulp

Spam under control, MailChimp can focus its efforts on actually building new products with data, just like Coca-Cola did with that extra pulp. One of its first orders of business is figuring out how to help customers get to know better the people to whom they’re sending their newsletters.

With this in mind, the company built a service called Wavelength that shows customers other newsletters that are similar to theirs. But the system that powers Wavelength also stores pretty much every interaction that every email address in the company’s database has with the newsletters they’re sent. That means what emails they open and when they open them, what links they click and when they click them, and what other newsletters they’re subscribed to. MailChimp also has a feature called Ecommerce360 that lets customers track clicks right through to conversions (marketing speak for someone actually buying something).

The company has been playing around with this data to identify clusters of users based on their behaviors and their interests — some of which Foreman has detailed on the company’s blog — and now it wants to roll it out to customers via a product MailChimp is calling ChimpQuery. Built atop Google’s BigQuery analytics service, ChimpQuery will let customers start doing this type of clustering and segmentation on their own, while saving MailChimp the troubles of hosting that infrastructure itself. (You can play with a monstrous, interactive graph of the entire MailChimp subscriber list here.)

If you sell knitting supplies and you find out there’s a big cluster of people on your mailing list who also are interested in wedding planning and custom jewelry, there might be an opportunity to create your content with these interests in mind or even to partner with companies in those spaces.

A sample cluster of subscribers.

Another topic that has been on Foreman’s mind lately is what he calls “frequency elasticity of engagement.” He’s done research suggesting that blasting the heck out of your email list might actually have detrimental effects in the long term (regardless of how the Obama campaign successfully exploited this strategy) but noted that engagement also has a lot to do with content and a particular company’s given user list. MailChimp’s data could help customers figure out the ideal schedule for emailing their subscribers.

For example, Birchbox has really high engagement because people love the service and have to open their emails to find out what goodies they’re receiving. Emails from a company like Papa John’s, on the other hand, might sit in someone’s inbox essentially as spam until they want to order a pizza and go searching for a coupon. Everyone has to figure out what pace and engagement metrics work for them.

Reining expectations back in

However, now that management is fully sold on the power of data, Foreman sometimes finds himself managing expectations rather than just pitching his ideas. COO Bainton, for example, is adamant that MailChimp start aiding its publishing-industry customers by using techniques such as natural-language processing and semantic analysis to help them personalize emails based on readers stated and unstated interests (that is, what boxes they check when they sign up and what stuff they actually click on).

Foreman, well, he’s pretty sure that’s too big a challenge for MailChimp to tackle considering how many publishing customers it has. MailChimp would have to understand all those customers’ industries to some degree (open source tools tend to highlight technically but not situationally relevant relationships, he said, and don’t always understand things like sarcasm) and probably the different languages they publish in, as well. Rather than understand content, he’d rather focus personalization efforts around how users are connected.

The company also needs to balance its ambitions with what’s legally and socially acceptable. The creep factor might be more important than what’s legal when it comes to email marketing. MailChimp determines the legality of everything it does before rolling it out, Foreman explained, but in era of “post-modern spam” where legitimacy is in the eye of the recipient and where some people use their “spam” button as a proxy for unsubscribing, companies must be careful not to offend.

“The more we can tell you about that list without getting creepy is really useful,” Bainton said. However, he added, ”I think expectation is more important than law.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How big data analytics drives competitive advantage
• Sector RoadMap: Social customer service in 2013
• Why the next front in big data might be psychological

That the bees, which are described as being half the size of a paperclip and weighing less than a tenth of a gram, can fly at all is an engineering marvel in its own right given their minute size. However, the next parts of the project could actually prove to be even bigger challenges.

According to a Harvard University press release:

[T]he next steps will involve integrating the parallel work of many different research teams that are working on the brain, the colony coordination behavior, the power source, and so on, until the robotic insects are fully autonomous and wireless.

Source: Kevin Ma and Pakpong Chirarattananon, Harvard University.

The problems are that building AI-powered brains won’t be easy and that there’s not yet an energy source small enough and dense enough to power a wireless bee.

On the brain side, though, the RoboBees team might have some help. As we explained in October, there’s a team from the Universities of Sussex and Sheffield in the U.K. working on a project called Green Brain that aims to develop a brain that could let robotic bees like those RoboBees is building act autonomously and respond to sensory stimuli.

Again, though, the small scale of the Harvard project could pose some initial challenges depending on how advanced it wants the brain to be. The plan is for the Green Brain project to run on a GPU-powered supercomputer and, presumably, communicate with sensors on the robot. Even if it were possible for a single GPU processor to run the Green Brain at operational speed, that could still prove too big for the tiny RoboBees, which need to do their own processing.

The key to success, however, might lie in RoboBees’ focus on colony behavior, which is somewhat akin to the concepts underlying distributed computing systems. Because the team expects the robotic bees to function like real honeybee colonies, individual bees can get by with less computer power.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to Make Cloud Computing Greener
• GigaOM Research highs and lows from CES 2013
• The case for low-power servers in the data center

With that line, the 2009 paper “The Unreasonable Effectiveness of Data” (co-authored by Google co-workers Alon Halevy and Fernando Pereira), Google Director of Research Peter Norvig all but guaranteed his status as one of most-quoted — or at least most-paraphrased — people in the world of big data. Last week, Norvig — as well as Google VP of Energy Arun Majumdar — was bestowed a slightly more-formal honor, as he was inducted into the American Academy of Arts and Sciences.

Norvig, who previously led Google’s search algorithms team and was head of computational sciences at the NASA Ames Research Center, is best known for his work in the realm of artificial intelligence. In fact, the above quote and the paper in which it appears are essentially a testament to the advances Google has been able to make in AI and machine learning thanks to the massive web page and search dataset that Google has amassed. The more examples it has of words and phrases used together in natural language, the better it can perform semantic analysis to determine what’s related to what.

Norvig and Majumdar are among 198 new inductees into the American Academy of Arts and Sciences’ latest class. According to a blog post from Google, they also join six other Google employees as members: Sergey Brin, Larry Page, Eric Schmidt, Vint Cerf, Alfred Spector, Hal Varian and Ray Kurzweil.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• How to use big data to make better business decisions
• Listening platforms: finding the value in social media data

The Wikidata project was kicked off around a year ago by the German chapter of Wikimedia, which is still steering its gradual development. For Wikipedia, the advantage is simple and powerful — if there’s a central, machine-readable source for facts, such as the population of a city, then any update to that data can be instantly reflected across all the articles in which the facts are included.

To posit a morbid example: a singer may have dozens or even hundreds of language versions of her Wikipedia entry and, if she were to die, the addition of a date of death to the Wikidata database would immediately propagate across all those versions, with no need to manually update each one (yes, I can also see how this might go horribly wrong).

Indeed, Wikidata is now being used as a common data source for all 286 Wikipedia language versions. Here’s the under-development “item” page for Russia, if you want to see what Wikidata looks like in practise.

But the really interesting thing with Wikidata is that it’s not just for Wikipedia – although it’s worth remembering that its API is still under development, the database can be used by anyone as it is published under a Creative Commons 0 public domain dedication. Here’s how Wikidata project director Denny Vrandečić put it in a statement:

“It is the goal of Wikidata to collect the world’s complex knowledge in a structured manner so that anybody can benefit from it, whether that’s readers of Wikipedia who are able to be up to date about certain facts or engineers who can use this data to create new products that improve the way we access knowledge.”

There are already some pretty cool (if bare-bones) examples of what people can do with Wikidata. One is GeniaWiki, which is trying to map the family relationships between famous people (the first and so far only example is that of the Bach family), while a Tree of Life project is trying to put together a viable, Wikidata-based “taxonomy of all life”.

It’s worth noting that the initial funding for Wikidata’s development has come from Google, the Gordon and Betty Moore Foundation, and the Allen Institute for Artificial Intelligence. Ultimately, Wikidata is precisely the sort of venture that is needed to feed the nascent semantic web and AI movement.

It’s far from the only venture in this space – I’d also recommend keeping a close eye on Google’s Knowledge Graph, which powers Google Now, and Wolfram|Alpha, which partly powers Siri – but all these (often intertwined) projects are essentially trying to do the same thing: to turn facts into something that machines can understand.

And that, in conjunction with advances in natural language processing and machine learning, will ultimately help us converse with machines. These are the building blocks of artificial intelligence and the future of search, and Wikidata’s very permissive license should act as an open invitation to anyone dabbling in this space.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Siri: Say hello to the coming “invisible interface”
• A near-term outlook for big data
• The future of mobile: a segment analysis by GigaOM Pro

For patients hoping to uncover the causes behind frustrating or painful symptoms, doctors and specialists have long been the most trusted resources.  Supercomputer Watson, predictive modeling techniques and other kinds of artificial intelligence are increasingly revealing their diagnostic power. But CrowdMed co-founder Jared Heyman believes there’s a third option that’s not getting the attention it deserves.

“I think a crowd can combine the best aspects of an individual expert with the best aspects of an artificial intelligence solution,” he told me. Crowds can blend an individual’s sense of intuition and judgment with a computer’s ability to process large amounts of information, Heyman added.

On CrowdMed, users can choose to “submit a case” or “solve a case.” To submit a case, patients describe their situation, including their symptoms, medical history, family history and other reports. “Medical detectives,” who want to try to solve the case, share their thoughts on potential causes and then place bets on the diagnosis they think is correct.

Before launching CrowdMed, Heyman founded InfoSurv, a market research company that uses predictive market technology. Building on that idea, CrowdMed analyzes the feedback from the crowd to generate diagnostic suggestions.

Patients seeking a diagnosis pay $199, as well as a $20 deposit, which is refunded when patients provide their final correct diagnosis. Detectives are rewarded with cash, prizes and status on an internal leaderboard, so they’re incentivized to participate only when they really believe they can productively contribute, Heyman said. In 20 test cases on Crowdmed, he added, the site’s detectives came up with the correct diagnosis every time.

Heyman said the idea for CrowdMed came from his sister’s recent three-year quest to understand an undiagnosed condition. After visiting more than a dozen doctors and accumulating more than $100,000 in medical bills, she was finally diagnosed with a very rare disease. In a test, CrowdMed came to the same conclusion in a matter of days and at a fraction of the cost.

In addition to opening up to the public, CrowdMed disclosed that had raised $1.1 million from investors including New Enterprsise Associates, Andresseen Horowitz, Greylock Partners, Y Combinator and SV Angel.

While the company plans to expand and build out its product, Heyman said the funding is largely intended to give the company some runway.

“For the company to take off, it will take nothing short of a cultural shift for people to recognize the vast wisdom of crowds. It’s not about individual experts, it’s about intellectually diverse teams collaborating to solve a problem,” said Heyman. “And I don’t know how long it will take for that notion to become accepted wisdom.”

Image by cifotart via Shutterstock. 

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• GigaOM Euro 20: the European startups to watch
• GigaOM Research highs and lows from CES 2013
• Facebook’s IPO filing: ideas and implications

Human intervention is critical at multiple layers, from choosing the algorithms to apply to feature creation to crafting the entire structure within which a machine will learn, said Scott Brave, founder and CTO of Baynote, at GigaOM’s Structure: Data conference Wednesday.

Down the road, he said, there will be more opportunities for machine-man collaboration, as data scientists observe what the machines may be learning and then add new inputs and ideas to the system.

“A lot of times we forget that even though it’s big data, the amount of data that the machine has access to pales in comparison to the amount of data we’re absorbing and have access to,” he said. “We’re building intuitions and holistic pictures in our minds and we see these connections that the machine might not even have the possibility of seeing because it doesn’t have the right data.”

Humans have a powerful role in figuring out the sources of data to give the machine and projecting their intuition, he added.

Still, Timothy Estes, founder and CEO of Digital Reasoning, pointed out that there are three key areas in which machine bests man – and, over time, they could give rise to some interesting social and cultural questions.

Humans will never be able to consume the sheer amount of data machines can process (unless it’s with some “Ray Kurzweil-style” man and machine merging), humans weren’t designed to receive thousands of inputs at once, and we’re ill-equipped to create a unified model of knowledge across that scale of information and make judgements from it, Estes said.

Recognizing that, he said, he predicts a social debate between adopting a “Google”-like model to artificial intelligence, in which the machine simply tells you what to do next, and a software model, that assumes more human agency.

“I believe we’re going to see that [debate] play out in the next decade between the software-centric model – a personal empowerment model – and a collective model,” he said. “And that’s the Skynet problem… you get a computer with intentionality that has access to data and the next thing you know you’re looking for a robot coming back from the future.”

Check out <a href=”http://gigaom.com/2013/03/20/structuredata-2013-live-coverage/“>the rest of our Structure:Data 2013 coverage here</a>, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to use big data to make better business decisions
• Takeaways from the second quarter in cloud and data
• Connected world: the consumer technology revolution