Surrounded by people he didn’t know, Brethenoux says he felt closer even than he sometimes does with members of his own family. At one point, he made eye contact with a young woman and both knew exactly what the other was thinking; her boyfriend wasn’t in on the mind meld. “During those two hours,” Brethenoux told me recently, “I had that feeling of belonging that was a little disturbing.”

And, he added, replicating that feeling is exactly what good advertisers should be looking to big data to accomplish. “How can you take that concept and build trust around it?” he asked. The answer to his rhetorical question is that you have to listen completely to what customers are talking about online and figure out their emotional attachments to certain things.

Manufacturing kinship

Only most marketing folks looking at sales data, for example, can’t tell if there’s Star Trek convention going on in within their customer bases; they just see a gathering of people at a convention center. Brethenoux preaches something he calls  the “kin” theory in order to figure out what’s bringing this cluster of people together and, better yet, to figure out how to be the company bringing them all together.

Done successfully, he said, “the attachment to the brand becomes very Apple-like.” The theory is that consumers will hold a special place in their hearts (or at least their subconscious) for brands they associate with the sense of kinship they experienced, and they’ll be more willing to become repeat customers. Some customers might share a sense of kinship around one topic, while others will rally around something completely different, but it’s that sense of belonging to a group that matters in the end.

Erick Brethenoux

When he was working in the insurance industry, Brethenoux explained, the company discovered a group of young men under 25 years old who owned sports cars and were surprisingly low-risk drivers. This, of course, goes against the conventional wisdom that young men in fast cars are about the least-insurable people on the road. It turns out they were all sports-car aficionados who housed their cars in safe places, didn’t drive them in bad weather and made all their repairs themselves (this was good because it meant fewer expensive trips to the garage).

The company reacted by creating a special policy category tailored to avid car collectors, one that Brethenoux said spread like wildfire and helped the company earn its money back about tenfold. And although, admittedly, the insurance company just cared that these guys took care of their cars, the insured felt like the company really understood their passion.

In the realm of athletic shoes, Brethenoux added, a marketer might look beyond just a shoe’s functionality (i.e., what sport it was designed for) and start looking at what the people who buy it are doing when they’re not wearing shoes. I can’t help but think of number of teenagers sportings Airwalks and Vans in the 1990s, or my yuppie brethren of today sporting barefoot running shoes from REI. The easy conclusion to draw is that we all participate in a certain activity, but the harder part is digging deeper to find out if there are other, more personal interests we might share.

Those ’90s teenagers might be wearing skateboarding shoes, but a love of indie music might be the real tie that binds. My fellow yuppies might all like trail running, but a large number of us might also be into microbrewing and craft beers. It’s capitalizing on this knowledge, Brethenoux said, that really forms a bond between brand and consumer.

Big, social data says a lot

And thanks to all the data people are giving away for free with their web-browsing behavior, as well as on social media, forums, user reviews and other places, brands can drill down pretty deeply, Brethenoux said. The consumer’s voice about who they really are and what they really like is louder than ever.

In the case of Brethenoux’s Star Trek obsession, he said, a marketer might have been able to piece together his affinity for the franchise from other data points. As he explained it, a guy who spends a fortune on Star Trek Lego sets and digital content, who’s a member of the National Space Society but works in software rather than space exploration, and who prefers exploratory video games to first-person shooters, likely feels a strong connection to Star Trek.

Although, he noted, despite all the hype about using analyzing social media data, most companies are still pretty unsophisticated, using it for simplistic and not-too-valuable insights such as overall brand sentiment. “We talk a good game about social data,” he said. “Very few actually leverage it effectively today.”

But hotel and airlines companies, in particular, might want to pay better attention to what’s actually possible. “A little increment in a market that’s so aggressive in terms of competition,” Brethenoux said, “is where a little difference can make the biggest difference.”

Feature image courtesy of Flickr user Magic Madzik.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• Why the next front in big data might be psychological

According to a Los Angeles Times article about Monday’s bomb attack at the Boston Marathon, the FBI has collected 10 terabytes that it’s sifting through in order to seek out clues about what exactly happened and who did it. Maybe I’m just a techno-optimist, but I find this very reassuring.

According the Times, “The data include call logs collected by cellphone towers along the marathon route and surveillance footage collected by city cameras, local businesses, gas stations, media outlets and spectators who volunteered to provide their videos and snap shots.”

Lots of data means lots of potential value

It’s reassuring because I’ve spoken with so many smart people over the years who can do amazing things with data. Ten terabytes isn’t a huge data set by any stretch of the imagination, but it’s plenty to work with if it’s of high quality. It’s very possible there are some needles in that haystack of call logs, and I’m optimistic the analysts within the FBI — possibly with some outside help — will be able to find them.

Techniques around video analysis and facial recognition are better than many people think, too. If there’s a way to stitch together hundreds — maybe thousands — of videos into a single truth of what happened, then I’m confident it will happen. By tracking faces and objects over time and place, we can recreate a crime and track down suspects without relying on after-the-fact accounts by witnesses who weren’t paying any attention until the bomb actually went off.

It’s not that witnesses are lying, it’s just that an attack like this might artificially color certain observations as being more nefarious than they really were. A Middle Easterner standing nearby might seem suspicious in hindsight, for example, but a witness might not have seen that guy cheering on a friend beforehand, stop to get a soda, and then meander over to the area where the bomb went off.

I have no clue what really happened, of course, I just know that cameras — especially hundreds of them at different angle and shooting over different timeframes — don’t suffer from selective or incomplete memories.

Can we crowdsource some surveillance?

I also find all this now-surveillance data reassuring because — if it proves useful — it might actually help to preserve our civil liberties going forward. We don’t necessarily needs drones flying overhead and cameras on every corner if we can crowdsource (at least from densely populated areas or big events) relatively high-resolution videos and photos during the investigation phase. We don’t necessarily need all orders of mobile call and location-tracking if we can collect what we need from the relevant area afterward.

This does little to prevent attacks, of course, and intelligence agencies will no doubt continue to trace phone calls and generally do what they do. That’s fine by me. If airports want to use facial recognition to flag known threats as they walk in the door, I’m not sure I can take issue with that either.

But by and large, it seems there’s precious little that surveillance — especially video — can do to predict crime unless an agency already knows what it’s looking for and has the means to act fast enough to make a difference. (IBM Fellow and general identity analytics guru Jeff Jonas wrote a great blog post in November about what’s actually possible to predict given the data on hand.)

So to the extent anyone thinks additional surveillance is going to help solve crimes that we didn’t see coming, I think I’d rather leave the data in the hands of hundreds or thousands of individuals and businesses rather than a handful of city, state and federal governments that might be tempted to overstep the bounds of what’s acceptable.

Really, though, the notion of how to prevent terrorist attacks and other mass-casualty crimes is a complex issue, and I’m not sure there are many ethically right or wrong answers. But when we get past the tragedy and criminality of what happened in Boston, we have to look at it as part of the bigger picture that’s shaping up around all the data we’re generating, collecting and analyzing. If terabytes of geospatially targeted call records and crowdsourced audio-video surveillance can help solve this type of crime and save all the time, money and privacy concerns of more-intrusive and expansive government efforts, then maybe there’s something worth considering.

Feature image courtesy of Shutterstock user Faraways.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

And although there’s a semantic element to it, this is beyond the realm of semantic search. We’re talking about doing a video chat, sending an email or just surfing the web, and seeing relevant content appear before your eyes. Why? Because the web and, heck, even our laptops are so full of information we don’t always know what to look for or have the extra attention to devote to looking for it.

Most recently, I spoke with Christopher Eakins, CEO of a company called Grapple Data that wants to revolutionize desktop search. Presently, the company’s flagship product, called Aikin, is doing something similar to semantic search on the surface. It’s responding to searches with a list of files, emails, contacts or other content — ranked by relevance — that a standard keyword search wouldn’t detect.

He says the product addresses the problem of information workers “being force-fed more than we can chew,” often across applications that don’t interact with each other at all. One might think of Aikin, he said, as a device that records, indexes and keeps track of everything you do on your machine, so you don’t have to remember specific file names, people or even keywords later on. If you have an idea what you’re looking for, it will find that content and then some.

Going forward, though, Eakins hopes Grapple can do away with desktop search altogether, or at least make it less necessary. That’s where the real innovation comes in. He wants to enable what he calls “information gravitation,” where relevant content would start to surface based on the subject of an email someone is typing, for example. It’s like those targeted ads in Gmail, only in real-time and, presumably valuable to users.

I first came across the concept in April 2012 while covering a company called PureDiscovery. Historically dedicated to semantic search and indexing within large corporate datasets, PureDiscovery CEO Dave Copps explained the company’s plans for going much, much bigger. Essentially — first within corporate networks and then across the entire web — it wants to teach is BrainSpace software to learn how people and pieces of content are related and then surface both automatically based on who you follow, what your interests are or even what text you highlight on a web page.

The plan appears to be coming along. The web part, which is definitely a bigger-picture undertaking, seems to have materialized in the form of a beta-mode application called Grokkit. (I’m still waiting for my invite.)

There’s also the work that Expect Labs is doing around its MindMeld application, which my colleague Om Malik lauded as “herald[ing] the era of anticipatory computing.” MindMeld is a video-chat application that also uses voice recognition and some serious data analysis to figure out what a conversation is about and surface relevant information related to that from the web or users’ social graphs. It also tries to predict where a conversation is going and queue up content that it thinks will be relevant in the future.

The point of all of this stuff — and even some of what we’re seeing in the enterprise IT world with startups like Ayasdi and BeyondCore — is that people don’t always know what they’re looking for or the right queries to enter in order to find it. If more information (or at least more relevant information) really is better, this should be a welcome trend.

Feature image courtesy of Shutterstock user photobank.kiev.ua.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Readers weigh in: future prospects for Twitter
• AWS Storage Gateway jolts cloud-storage ecosystem
• 4 iPad apps to help wrangle data

BeyondCore isn’t your average analytics startup. Although the company is just emerging from stealth mode on Tuesday, it has been around since 2004. It hasn’t yet raised a round of venture capital, although it does have some impressive beta customers — including 11 of the Fortune 100. According to Founder and CEO Arijit Sengupta, the company arose from a Harvard Business School project he did with faculty adviser Clayton Christensen focused on how to remove humans from IT processes.

Sengupta wanted to make business analytics a push-button affair, and after eight years he finally thinks he and his team of mathematicians have accomplished that goal. Instead of making users find the needle in the haystack, he wanted to create software that can find the needle (and maybe a few other tiny household items) and present it to the user without ever being told what it’s looking for.

Humans are bad computers

Although Sengupta is quick to point out that the company’s flagship product, Lucid, is not machine learning software, the underlying thesis is similar to that of any company employing machine learning techniques: It doesn’t take long before human beings are overwhelmed by datasets and can’t possibly find all the relevant patterns and correlations. Humans don’t scale, he said during a recent phone call, so ”you cannot solve the big data … problem if humans are core to the process.”

This is the same reason Sengupta is not a big fan of most current BI tools or big data technologies, which he equates to trying to solve an exponential problem with a linear solution. The way most analysts work is they have to create dashboards and PowerPoints and try to prove there’s value in what they’ve found. And, of course, they’re responsible for actually uncovering those insights among the vast expanse of names, numbers and other values sitting in front of them. But if we can remove the human limitation from the equation — at least in the analysis stage — computers can just take over and solve the problem, Sengupta explained.

Once that’s done, he added, “[We can] get to a point where a business user feels like someone’s walking through the analytics and data and helping them find what’s valuable.” That’s exactly what BeyondCore claims Lucid can do.

Sit back and let this avatar take over

At about 1:51 into this video on Lucid, you can see what Sengupta is talking about. Once the user chooses from a pulldown menu what variable against which he wants to analyze the rest of the data, the software takes over and begins analyzing every combination of variables and then calculates which ones have the most-significant effect on the chosen value. At that point, the user has four options for how to proceed, although it’s the one called “Analyst Overview” that really shows what Lucid can do.

A screenshot of the Analyst Overview

A feature I suspect is either awesome or creepy depending on your personality, Analyst Overview actually brings up a presentation in which an animated analyst walks and talks users through the key findings on the analysis. It shows charts, highlights strong correlations and outliers, and generally gives the user a good idea of how to proceed and where to investigate.

After the presentation (or if they decide to skip it), Lucid users can enter Analyst Mode to begin looking at and experimenting with the software’s results. They can add or remove variables that might have led to false positives, maybe test a hypothesis (although, Sengupta is quick to point out, if there is any statistical significance to a hypothesis, Lucid will find it) and create new types of charts. They can add comments to data points or entire analyses, flagging key points or perhaps things that need a second set of eyes.

Statistician Mode

Anyone wanting to test Lucid’s math can enter Statistician Mode, where they’ll see scatter plots, R scores and other information showing how the software came to the results it did. “This is almost like showing our homework,” Sengupta joked, because there’s always someone in the room who doesn’t believe you’ve done the work.

But can it scale?

Update: Curiously, Lucid is one of the few big data products just now hitting the market that isn’t built atop Hadoop. It doesn’t even support Hadoop as a data source out of the gate,  except through partners, which might be a problem as more companies choose Hadoop as their primary data store for massive datasets. But Sengupta doesn’t think this limits Lucid’s effectiveness.

For one, he said, BeyondCore can add native Hadoop support if and when it’s necessary. And regardless where data is stored, as long Lucid can at least read it as key-value pairs, it can analyze it. The processing engine behind Lucid is also massively parallelized, Sengupta explained, so it can easily churn through large datasets like Hadoop MapReduce can. Lucid is primarily available as a cloud service hosted on the HP Cloud , but there’s also a laptop edition capable of handling up to 100 million rows of data.

At any rate, Sengupta noted, analyst activity in Lucid is plenty fast once the initial analysis is done, because all the calculations have already been done. At that point, whenever a user adds, subtracts or otherwise manipulates the results, the system is just pulling up the calculations it has already carried out rather than doing them anew.

One large BeyondCore beta customer in the IT industry used Lucid to analyze 58,000 customer invoices for discrepancies. It took just minutes to perform more than 500,000 calculations across 21,000 variable combinations, Sengupta said, and the software — without having any prior knowledge about what an “invoice discrepancy” was — discovered 30 critical insights that human analysts had never even considered.

In another instance, a large hospital company was interested in figuring out why some patients remained longer at some hospitals than they did at others. Lucid performed more than 900,000 calculations across about 534,000 variable combinations (city, procedure, insurer, line of services, etc.) and uncovered and 247,000 possible patient outcomes to uncover 35 critical insights. It also highlighted the outliers: For one procedure, patients in one city were 9.3 days later than average, while patients in another city were leaving 5.4 days earlier.

Is business ready for the future?

Assuming Lucid lives up to BeyondCore’s claims, an automated solution based on “pure math” is certain to turn a few heads from companies concerned with trimming the fat from their analytics efforts and making sure they’re not leaving anything behind. But getting them to abandon decades of decision-making process, job descriptions and IT investment won’t be easy. At least, it hasn’t been for other companies and whole industries promising world-changing technology products.

Still, Sengupta is understandably optimistic about the future of BeyondCore and how he thinks it can transform the analytics market. He even goes so far as to envision the possibility of combining Lucid with something like Siri to enable deep data analysis using nothing but a smartphone and the human voice. That’s an inspiring vision that seems entirely plausible in the very near future. But in the world of enterprise IT, at least, he might be getting ahead of himself.

Feature image courtesy of Shutterstock user phipatbig.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Health care and big data in 2012
• The importance of putting the U and I in visualization

The company, which specializes in real-time traffic data, powers a wide variety of in-vehicle navigation systems, mobile apps, commercial fleet management systems and even Google Maps. The secret behind Inrix’s success that it collects lots of data from lots of drivers in order to help everyone get where they’re going faster. But saving commuters driving time is just the beginning of the company’s plans. It thinks traffic data can help change a wide variety of industries, maybe even the world.

100 million devices and 1.8 million miles of road

That a traffic-data company could contribute to such macro-level change might seem laughable until you get a sense of Inrix’s scale. According to founder and CEO Bryan Mistele, 6 of the 8 auto companies with built-in navigations systems (including Ford, BMW and Audi) use and share Inrix data, as do 8 of the 12 top navigation apps in Apple’s App Store (including MapQuest, Garmin, Microsoft and Telenav). Many of the commercial trucks we see on the streets are sharing data with Inrix too, and even “dumb” phones without GPS and internet connections are sharing location data with the company through cell towers.

All told, the company counts more than 100 million endpoints as its data sources and covers more than 1.8 million miles of road worldwide, Mistele said. Its total volume of traffic data, which the company crunches through constantly to generate real-time information, is more than 500TB. It runs its own homemade big data infrastructure, Mistele says, because “there are no off-the-shelf packages [not even Hadoop] that in real time can process that amount of data.”

From real-time to predicting the future

And generating real-time traffic conditions is only a portion of what Inrix provides to the customers that pay for its services. The company also brings in, among other sources, weather data, accident data and sensor data in order to provide insights into how traffic is likely to shape up. By factoring in the location, number of cars involved and whether there are injuries, for example, Mistele said Inrix can predict how long an accident will hold up traffic at a given location.

Because it has so much historical information from such a broad set of sources, Inrix is also able discern reality from situations that might confuse models that are only concerned with whether vehicles are moving or stopped. Mistele said the system is smart enough to know that a car sitting at a stoplight on an arterial road is not akin to a car stuck in a traffic jam on a highway, or that taxi cabs and UPS trucks stopping and going are not signs of stop-and-go traffic conditions.

Disrupting industries, and urban sprawl

Aside from consumer navigation apps and those helping commercial drivers manage their delivery schedules, a number of state, regional municipal and even national governments use Inrix’s analytic services to help gauge a number of issues relating to road management. Mistele said the company has actually disrupted the media industry, too, by helping spur the end of traffic helicoptors and “traffic on the nines” on local radio stations. For customers like Clear Channel, it’s just a lot cheaper, easier and more effective to feed their on-air talent with real-time data and information that someone else has put together.

“This is a space where there has been a complete transformation [thanks to big data and crowd sourcing],” Mistele said.

As great as it is helping people get from Point A to Point B, though, being able to get a handle on traffic data could have even further-ranging effects. Insurance companies can use the data to determine more-accurate rates, and some hedge funds are using Inrix’s data as a means for determining economic health — more drivers during rush hour means more people working, Mistele explained. During the London Olympics, data from mobile devices helped officials monitor the movement of people, not traffic, throughout the city.

A shot of Xerox’s City Manager for parking

As cities continue to grow and congestion becomes an even bigger problem in terms of decreasing productivity and increasing pollution, it’s that kind of data from companies such as Inrix (or competitors Nokia Navteq and TomTom) that could help mitigate the effects. Xerox, for example, already uses Inrix data as part of its efforts to help cities improve urban planning around roads, mass transit and parking spaces. The more that city planners know about how, where and when their citizens move, the better they can plan transit systems that address those realities, or that can more easily respond when problems arise.

“Give people better data, give governments better data,” said Mistele, “and you can have a huge impact on one of the biggest of the biggest problems in our society.”

Feature image courtesy of Shutterstock user TonyV3112.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• CES 2012: a recap and analysis
• Dissecting the data: 5 issues for our digital future

According to a release from the MIT News Office, Associate Professor Devavrat Shah says his model has been 95 percent accurate during testing and has been predicting trends hours before they appear on Twitter’s list. The algorithm incorporates a new approach to machine learning that compares real-time data with historical data and predicts outcomes based on past events that most closely align with the current situation. So, rather than analyzing a topic’s chances of trending equally against the entire historical corpus of topics, it will assign more weight to topics whose paths followed similar trajectories up the ranks of top trends.

And Twitter is certainly interested in the research. A company spokesperson emailed me to point out that Shah’s graduate research assistant, Stanislav Nikolov, is a Twitter employee.

Imagine knowing these topics before Twitter does.

However, the algorithm’s level of accuracy and speed would have to translate to a much-larger and more-complex stage — Twitter’s real-life firehose and stockpile of historical tweets — if the company were to use its predictions to charge premiums for ads associated with certain topics, as Shah suggests. Advertisers might not be happy to pay premium rates for topics that fizzle out before ever becoming top trends (although a tiered rate system based on the model’s confidence or, perhaps, projected ranking among top trends could work). Thus far, the algorithm has been trained using a set of 400 topics, half of which trended and half of which did not.

Shah thinks it’s a great fit for Twitter data because the data is relatively clean and he has found a strong correlation between past and future activity. Other historical data sets might be more messy or have more noise than does Twitter’s data set, which would make it much more difficult to filter out extraneous data and discern the real factors that lead to a particular result. However, even Twitter has presented research showing, in the case of its search engine at least, how the sheer volume of data it receives and the speed at which it comes in can make it difficult to accurately predict what someone wants to see.

The good news, though, for anyone willing to give Shah’s algorithm a try is that it’s designed to process data in parallel across scale-out systems like those used by large web companies. Therefore, training it and then running it in production across a voluminous data set won’t run into the same obstacles traditionally faced by machine learning algorithms as data sizes increase. And there are potentially more lucrative and rewarding endeavors that could benefit from this type of predictive power: Shah suggests stock markets, movie ticket sales and public transportation as possibilities, but others might include combating cybercrime by identifying threats earlier or predicting the severity of disease outbreaks.

Feature image courtesy of Shutterstock user turtleteeth.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Examining the rise of crowd labor platforms in 2012
• Social third-quarter 2012: analysis and outlook

One company that wants to speed the evolution of job-searching is a startup called Entelo, which launched on Wednesday and has raised an undisclosed amount of funding from Battery Ventures and Menlo Ventures. Entelo aggregates public data from a variety of online sources that might do a better job of displaying high-tech skills than a resume ever could — Github, StackOverflow, Quora and LinkedIn among them — and creates profiles of potential candidates. After six months in private beta, Founder and CEO Jon Bischke told me, the company has created more than 300 million profiles and has more than 40 paying customers, including Box.net, LookOut, Kontagent and LevelUp.

Recruiters can search those profiles and filter by skills, education or a number of other factors, but the real beauty of Entelo is that it predicts for clients when the best talent might actually be willing to come work for them. Bischke said Entelo has a predictive model that might indicate when someone is in the market for a new job. One factor is the length of time he has been at a job in comparison to his historical tenures, but the most telling, Bischke said, are updates to social profiles.

Already, Entelo was able to help a client proactively reach out to and hire an engineer who had recently moved to Los Angeles from Boston and hadn’t even yet begun a job search in his new city. Entelo’s database of potential employees is a good service in its own right, but it’s just a prerequisite to doing prediction, Bischke said. “As far as anything we’ve seen in the market,” he added, “that’s the really differentiated piece.”

But the company also could deliver value outside the world of talent acquisition by spotting macro trends and reporting on them. For example, Bischke said, the company’s algorithm could spot — indeed, already has spotted — upticks in profile activity among employees of individual companies that could suggest an acquisition or perhaps a round of layoffs is coming. Of course, there’s value there for recruiters, too, who have had to manually keep track of things such as the dates when stock lockouts lift following an acquisition or IPO.

“We’re using big data to apply some automation to stuff good recruiters have done for many years,” Bischke said.

Although, he noted, Entelo has to be careful to take any business expansion slow and to first focus on getting the core business right. Part of the challenge for Entelo will be finding a middle ground between being a straight delivery mechanism for potential employees and being something more intelligent. Bischke doesn’t think it’s fair for Entelo to pre-judge individuals’ suitability for any given position aside from traditional means of search-engine relevancy, but Entelo is gathering a lot of data (e.g., number of followers on social platforms, number of code commits, education, etc.) that could help identify trends among the people who end up getting hired.

Another challenge, he said, is “trying to assess the probability a person is the kind of person you’re looking for.” While the guys who created Ruby, Github or any other popular technologies and platforms probably are very skilled and have accordingly large social followings, it’s highly unlikely someone will be able to hire them, so putting them at the top of search results — or surfacing them at all — isn’t really valuable for Entelo’s recruiter clientele.

And there’s still a lot of room for growth in the talent-acquisition space proper without trying to get too fancy. “The wind is at our back” in terms adding new data sources, he said, with services such as Github for specific fields already popping up. If Entelo is ambitious, it could move beyond its technology roots and start targeting fields such as law or medicine, too. ”There’s a huge amount of open sky,” he said.

Feature image courtesy of Shutterstock user maminez.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Facebook’s IPO filing: ideas and implications
• 12 tech leaders’ resolutions for 2012

Here are five sources of information that will help you talk intelligently about who’ll win without even forcing you to pick a side. Just the data, please.

FiveThirtyEight

This New York Times blog manned by expert statistician Nate Silver might be the gold standard for predicting elections. FiveThirtyEight is updated multiple times a week, usually tied to the release of poll data or economic numbers, and follows a consistent model for generating its forecast. The blog includes some easy-to-follow visualizations, including each candidate’s chances in each state. Also nice is that Silver gives some analysis of why the forecast is shaping up the way it is rather than just presenting the result. Latest prediction (July 31): Obama (69 percent chance of winning).

InTrade

Like the real stock market, event-prediction markets InTrade lets users buy stock in an outcome and can change in real time. Although it tends to be very accurate overall, it can vary greatly even during the course of a day as rumors or other information affect traders’ behavior. For contests with unknown variables, it’s probably best to get in on the InTrade action late: In January, it had Marco Rubio (R-FL) and New Jersey Governor Chris Christie as the most-likely Republican VP picks. Today, it has Rob Portman (R-OH) ahead, with Rubio down to third. Latest prediction (Aug. 1, 12:46 p.m. PT): Obama (58.2 percent chance of winning).

The Signal

The Signal is a Yahoo News blog maintained by data scientists David Rothschild (who also co-runs the PredictWise prediction site, which gives Obama a 59.2 percent chance of winning) and David Pennock. They accurately predicted Romney winning big on Super Tuesday in March, but weren’t so good at trying to pick Republican VP candidates. (In their defense, they were extrapolating from InTrade’s predictions and it was still early primary season). Their model is based on electoral votes and readers can explore various scenarios with an interactive map. Latest prediction (July 18): Obama (with 303 electoral votes).

PoliticIt

PoliticIt is a Utah-based startup that has its eyes set on selling big data software to political campaigns and is trying to prove its worth early on with a public site analyzing current races. The company generates an “It” score based on a candidate’s digital influence, which is comprised in part of the candidate’s digital footprint and voter sentiment toward the candidate. It’s kind of like Klout for politicians. As of June, PoliticIt had tracked more than 160 elections and claims candidates with higher “It” scores won 87 percent of the time. Latest It scores (May 31): Obama (48), Romney (34).

Twitter Political Index

Just launched on Wednesday, Twitter’s Political Index analyzes user sentiment about Barack Obama and Mitt Romney and generates a score from 0 to 100. It’s not so much a prediction service as it is a service to compare predictions and polls with voters’ actual feelings, letting users make their own predictions about what might happen come November. Keep in mind, though, that while Twitter analysis is popular for everything from stock activity to the Academy Awards, not everyone thinks it can accurately predict anything about elections. Latest index scores: Obama (34), Romney (25).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

What’s possible

Recent HIV research from Brown University is a great example of what’s possible. Researchers formulated a computer model based on numerous factors relating to drug use, sexual activity and the medical aspects of HIV infection. To ensure it was accurate, they calibrated the model until it could accurately reproduce known HIV infection rates in New York City from 1992 until 2002. They ran the model thousands of times on a supercomputer.

Credit: Brandon Marshall/Brown University

They found that the rate of of HIV infection among New York City injection drug users will be 2.1 in 1,000 by 2040 if current programs are left in place. Expanding needle exchange programs will decrease that rate by 34 percent; expanding HIV testing would only result in a 12 percent reduction. However, a comprehensive approach that includes these two programs as well as two others regarding the administration of medicine and antiretroviral therapy would drop the rate by more than 60 percent to .8 per 1,000.

Assuming their model is accurate, that’s a significant reduction — getting HIV rates among drug injectors near zero — and it’s all thanks to access to lots of data and lots of computing power. Recently, another group of researchers in Europe developed a computer model that found a strong correlation between web censorship and high violence rates during times of social unrest — a timely finding given the current state of world affairs.

Last week, I explained how Xerox is working to help Los Angeles and other cities get a better view of their traffic so they can try to make life more efficient and less congested for citizens, while simultaneously reducing pollution and optimizing budgetary resources. To achieve these goals, Xerox and other companies in this space are gathering data from everywhere — cars, mass-transit systems, traffic sensors, cell phones, weather databases — and developing complex machine learning models to determine how everything is connected.

Of course, these are just a handful of examples of what researchers and others are working on with regard to data. Pick an area of public concern — climate change, smart grid, crime rates, genetics, whatever –  and you’ll find someone with mountains of data running some seriously complex algorithms to make sense of it.

Anyone can do it

However, as anyone who reads GigaOM regularly probably knows, decision-makers don’t need in-house supercomputers or data scientists on staff to inform their policies with data (although the latter wouldn’t be a bad idea). All they really need is an internet connection. Data sets are available everywhere you look, including at data marketplaces such as Factual and Infochimps, at Data.gov, and even increasingly on news sites such as the Guardian (see disclosure). Thanks to cloud computing, the resources necessary to analyze this data are cheap and plentiful.

And with increasingly prevalent cloud services targeting low- to mid-level users who want to run some relatively simple analyses, there’s no excuse for politicians and others not to inform their decisions with — nay, base them on — data. Last week, with company at my house and two toddlers running around, I was able to sit down with my laptop and generate a predictive model for gun-related homicide rates using a service called BigML and data from the Guardian‘s Datablog. It’s nowhere near Brown’s model, but I was able to do it while sitting on my couch.

Lazy politicians need not even get their hands dirty with raw data because chances are some journalist or bureaucrat has already analyzed it for them. Data on gun ownership in the United States versus the rest of the world is everywhere this week, as is, already, data on the spike in gun sales after last week’s shootings in Colorado.

The Nevada state legislator I recently heard on the radio struggling to defend his proposed tax on junk food would have benefited from reading this study from the USDA. It’s the top result on Google when searching “junk food cheaper than healthy food.” There’s also this interesting study on the effectiveness of Mayor Bloomberg’s giant soda ban in New York.

Why we should listen to the data

Look at the state of the world right now. Droughts, deficits, civil wars, obesity epidemics. A skeptic would argue that the old methods of public policy decision-making, driven largely by political and economic concerns, haven’t worked out too well. Why not give data a chance to take the lead? In the wake of the great recession, smart businesses certainly have.

It’s a simple proposition: Choose an important issue, find relevant data on it, analyze the data (or trust someone else’s analysis), and go from there. It’s objective starting viewpoint about whether something might actually work, political pressures be damned. Who knows, a brave politician who plants a stake not on the left or the right, but with data analysis, might end up looking like a hero in the end.

Disclosure: Guardian News and Media Ltd., the parent company of the Guardian newspaper, is an investor in the parent company of this blog, Giga Omni Media.

Feature image courtesy of Shutterstock user MikeE.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Health care and big data in 2012
• Takeaways from the second quarter in cloud and data

Central to most of Xerox’s anti-congestion projects, including ExpressLanes,is the idea of dynamic pricing, which rises with demand in order to maintain some semblance of order. As Natesh Manikoth, Xerox’s chief technology officer for transportation solutions, explained to me, if a driver is paying to drive in the HOT (high-occupancy tolling) lane, he’s guaranteed a consistent speed of 45 miles per hour. If traffic starts backing up, prices for individual cars will rise to discourage them from entering, saving the lanes (which, before this program were high-occupancy-vehicle lanes) for high-occupany vehicles such as buses and those involved in carpools.

Xerox has another program in Los Angeles called ExpressPark, the goal of which is to let people know when they’re about to leave the house whether and where they might find parking, and how much it will cost. “It’s not enough to know how to set the price, you have to make sure that data gets to users in real time,” Manikoth said. Drivers also need to know parking spots will still be there when they arrive in 40 minutes. That’s a prediction problem.

The answers lie in big data, difficult data

The key to all of this, of course, is lots of data. ExpressLanes, Manikoth explained, works by sensing traffic flows in the HOT lanes as well as in the adjacent lanes and calculating travel times. Because a pre-defined algorithm won’t work, the model is designed to learn as it takes in more data about how any given set of conditions affect traffic flow. Xerox is just getting started with developing its model, Manikosh said, and he aknowledges it won’t be easy.

Traffic accidents, broken down cars and other unforeseen incidents can quickly make a mess of even the best models, especially because no one can predict how long an accident will take to clean up or how many lanes it will close down. And Los Angeles is a particularly unique beast among large cities because it lacks a strong city center, so traffic is relatively constant and in all directions. However, he said, “We stepped up, we’ll have to now prove it.”

View Larger Map

While the parking project depends heavily on prediction, those predictions rely heavily on history. Manikoth said his team can build a machine learning system that looks at historical data as it relates to the current price of parking to predict whether spots will remain available. But that’s easier said than done — or at least, done right — because parking behavior is affected by myriad hard-to-predict factors such as how long someone will be at a meter and how many drivers are willing to park illegally. The meters’ sensors and payment systems can track occupancy rates and what people pay, but not how individual people will act in any given situation.

Solve traffic, solve a lot more

Xerox isn’t alone in trying to help bring order to the chaos that is big-city driving, though. Companies such as IBM and Siemens are also working on the problem — and it’s all part of a larger effort to minimize the problems that cities — the economic engines of our society — experience as they grow. Drivers circling blocks looking for parking spots and commuters stuck in freeway gridlock contribute to pollution and generally lower the quality of life for everyone involved.

Manikoth said the real answer lies in combining information from other sources, such as mass-transit systems, toll highways, traffic sensors and weather data (all of which Xerox also collects) to paint a real-time picture of what traffic actually looks like. Armed with this type of information, city planners might be able to devise more-intelligent stoplights, bus routes and train schedules — maybe even dynamically — and commuters might be able to decide they’re better off just taking the subway today. By collecting diagnostic data from buses, he said, transportation authorities could spot potential issues that might otherwise result in a future breakdown that messes with schedules and people’s lives.

Of course, given how big and expansive a problem traffic management is, there’s monetary incentive for anyone who’s actually able to solve it. “We firmly believe that solving problems for cities is a good thing for society as a whole,” Manikoth said “but it’s also good business.”

Feature image courtesy of Shutterstock user Aaron Kohr.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data