Starting with 52 “root” tweets from users both famous and obscure, the researchers first analyzed the dynamics of retweeting, like the speed and spread of a tweet from a user to followers and then their followers. The researchers, from the University of Washington, MIT, and Penn, used the Twitter API to collect all the retweet information and found that most retweets occurred within one hour of the original tweet. Not surprisingly, they also found that root tweets are retweeted more than the retweets themselves.

They then plugged the important variables –- number of followers, retweet speed, retweets of other tweets –- into a Bayesian model, a statistical approach that uses prior evidence (the root tweets) to calculate how the retweet graph evolves. They experimented with feeding the model different amounts of prior evidence to see how much was needed to make an accurate prediction. Using only 10 percent of the retweets to guide the model, they were able to reasonably accurately predict retweet time and volume, and the error decreased the more retweet data they included. The average retweet time was only 4.4 minutes.

Throwing more information into the prediction engine (like whether a particular follower has a large numbers of followers of his or her own) could improve the accuracy. Their model was thrown off, it seems, by a few anomalous tweets with a very rapid onset and termination of retweets that didn’t follow the same pattern as the other tweets. (Though they don’t identify who sent those tweets, my bet is on @KimKardashian, whose followers’ actual and predicted retweet timecourse is pictured above.) The researchers didn’t even consider the time of day a tweet was posted, nor its content; there is likely huge potential to mine in those domains for what, and when, leads to trending.

With the abundance of the Twitterverse open to developers via API, this study represents just the tip of the iceberg in predicting tweeting behavior, something that startups like Blab are busily pursuing. It also shows that robust methods like Bayesian statistics can predict if a tweet has any retweet life left, and thus whether it can gather more eyeballs and clicks, something that is sure to prove very lucrative.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q3: Facebook remakes headlines in social media

However, if you’re like me, hearing anybody repeat that curly fries data point as fact likely sends shiver down your spine. It’s not that it’s not true — it very well might be — but that it’s nearly useless information without more background.

That’s right, the old correlation versus causation argument is front and center once again. In all the big data world, it’s probably the biggest fallacy there is, no matter how you look at it. No, getting value from big data always doesn’t require giving greater credence to correlation than causation. And, no, relying on correlation isn’t inherently some sort of an ethically or scientifically questionable practice.

Really, the choice between relying on correlation or striving to find causation probably depends on what you’re trying to do.

When there’s nothing at stake, correlate away

Let’s be honest: If all I’m concerned with doing is boosting clickthroughs, selling more products or predicting the movies you want to see, correlations probably will work just fine. I don’t really care why, for example, Mac users book more-expensive rooms on Orbitz — I just care that they do.

You visit my site, my system sees you’re using a Mac (or that you like curly fries, or any other attribute it can associate with you) and it shows you content that it thinks you’ll want to see. It’s not a perfect approach, but it’s probably a far cry better than the old method of just showing everybody the exact same content.

And when you’re collecting potentially petabytes of user data and trying to serve ads in near real time, strong correlations might be about the best things you can hope to find. It’s a volume-and-velocity business, and heavy examinations of why any two (or more) things are related to one another might not always provide a high return on investment.

A more extreme example of when correlations might suffice would be something like machine-to-machine systems that need to make decisions in real-time in order to prevent disasters. The people charged with running these systems might not know why a certain series of events often precedes a particular outcome, but it’s better safe than sorry.

You can’t make a difference — or real decisions — with correlations

But if you’re trying to use big data to make a meaningful difference in the word or to make decisions that can have significant real-world consequences, mere correlations probably won’t cut it. This is what Evgeny Morozov warns about in relation to crime in a recent New York Times column. It’s what Gourley had in mind when talking about data science versus data intelligence. It’s why the current discussion around machine learning almost always includes a human aspect, as well.

Many of the reasons for not acting on correlations alone are based on privacy and a whole collection of civil, constitutional and human rights. You simply can’t profile and then arrest, for example, people based on what their Likes suggest they might be. You probably shouldn’t make decisions about people’s financial, health or general well being based on mere correlations, either.

Heck, I wouldn’t even serve ads that delve into personal information such as health, sexual orientation or intelligence without a very strong reason to believe I was accurate (and express consent to serve those ads). And the Facebook-curly-fries study is full of correlations that could be potential landmines, a small portion of which are visible in the chart below.

More correlations from the “curly fries” study. Source: Proceedings of the National Academy of Sciences

But these are all situations where the fear of incorrectly profiling someone occasionally — and being sued as a result — might overpower the desire to do good most of the time. The data Darwinism that my colleague Om Malik wrote about recently extends beyond just peer reviews and social-media ratings, and one shouldn’t take the role of playing God (or catalyst for evolutionary change, to continue the Darwin metaphor) lightly.

Sometimes, though, correlations aren’t enough because you really want to solve a problem or perhaps build a great product. As Gourley explained at Structure: Data, even using correlative data to predict insurgent attacks in a place like Iraq is relatively easy, but predicting the likelihood of events doesn’t stop them. Stopping them requires really understanding and addressing the root causes of the attacks.

The same goes for stopping disease outbreaks, figuring out why programmers make more mistakes during certain seasons, stopping gun violence, or just capitalizing on that knowledge about curly fries or hotel-room bookers in order to build products that touch upon the deeper rationales for liking those things. You can fight the symptoms, so to speak, or you can cure the disease.

So feel free to try selling the next guy you see eating curly fries on a documentary about Dostoevsky, but don’t expect him to care. It might be that there’s some strong connection between curly fries and intelligence; of course, it might also be that intelligent people — entirely coincidentally — tend to live within walking distances of an Arby’s. But no one has asked about that.

Feature image courtesy of Shutterstock user Tobias Arhelger.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

OK, so your NCAA tournament bracket has officially been busted. Don’t feel so bad. ESPN college basketball analyst Jay Bilas, stat-geek superstar Nate Silver and even SAP’s vaunted predictive analytics software all missed the upsets, too. So did President Obama.

Three of the four correctly picked 11 of the Sweet 16 teams, while Bilas correctly chose 10. But despite the similariy in results between men and models, I’d follow Silver’s model-based forecast every time. Not only is it accurate, but it stands to make people a lot of money.

Just to be clear, though, Silver doesn’t actually pick winners and losers (at least not publicly, as far as I can tell). Rather, he uses a model that takes into account a number of variables — including a handful of popular computer rankings — and produces the probability of each team advancing through each round of the tournament. That’s what makes his forecast so effective if you’re a betting man: It’s easy enough to pick the winner and most of the final four by just choosing the top seeds (I’m looking at you, POTUS), but the way to accel past everyone else in points is to spot the Cinderellas.

If I were ESPN, I’d pay Silver a boatload of money to come on TV once a year and present his forecast to a March-Madness-obsessed nation. I’m fairly certain the network could extend the broadcast out to about three hours and charge Super-Bowl-like advertising rates. Here’s why.

It’s the probabilities, stupid

As I was saying, anyone, including Silver, can spot the best teams in the tournament by watching enough basketball, settling on some important data points to analyze or just following the NCAA’s seeding. Here are the seeds my experts, data analysts and the leader of the free world chose for the Sweet 16:

• Bilas: 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 5, 5
• Obama: 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5
• SAP: 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 4, 5, 5, 6, 7, 8
• Silver: 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5

Here are the actual seeds that advanced to the Sweet 16: 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 6, 9, 12, 13, 15.

SAP’s mid-seed-heavy bracket

The smart money is always on the higher seeds from a pure probability standpoint (although I have no idea how SAP built its model to get so many 5-8 seeds in the Sweet 16). But strange things can, and often do, happen in the NCAA tournament. This year, those strange things are called Wichita State (9-seed), Oregon (12-seed), LaSalle (13-seed) and Florida Gulf Coast University (15-seed).

So why am I so high on Silver if his Sweet 16 probabilities were just as off-base as the two non model-based human brackets and SAP’s model-based picks? Because if I were looking for a few upsets, he might have helped me spot them. Here some of his notable projections for lower-seeded teams most likely to advance:

• Arizona (6-seed): 38.1 percent of reaching the Sweet 16 — they made it (SAP picked this correctly)
• Florida Gulf Coast (15-seed): 3.3 percent chance of making the Sweet 16 — they made it
• Oregon (12-seed): 17.5 percent chance of making the Sweet 16 — they made it
• Minnesota (11-seed): 61.9 percent chance of winning its first game — it won (Bilas, SAP and Obama picked this, too)
• California (12-seed): 32.8 percent chance of winning its first game — it won (SAP picked this correctly)

And in my neck of the woods — Las Vegas — being smarter than the sportsbooks means big money. No. 12 Oregon and No. 13 LaSalle didn’t really sneak up on the oddsmakers (60-1 and 100-1 odds to make the Final Four, respectively), but No. 15 Florida Gulf Coast is paying out 1,000-1 should it reach the Final Four.

I wouldn’t count on that happening, though. Silver now gives those teams a 1 percent, 5.1 percent and 0.8 percent chance, respectively, of making the Final Four. Louisville, Florida and Indiana look like locks to make it, and one of them should win the tournament.

Men vs. models: Let’s call it a draw

If you’re looking at these selections as some sort of man-versus-machine competition, I don’t think you’ll find a clear winner. Although Silver comes out looking the best of the four brackets I analyzed, his projections aren’t that much different than Bilas’s picks. And although SAP’s picks fall apart in the end — two of its Final Four selections (including its national champion pick) are out — it did correctly pick a couple upsets. President Obama, well, he pretty much picked chalk. The jury is still out on SAP’s model, which has three Final Four picks alive but made what seems like a risky choice by choosing Michigan State over Louisville.

The better way to look at these results is probably as further evidence that man and machine need to work together more closely, something we highlighted heavily at our Structure: Data conference last week. Men create models, but men probably don’t crunch the numbers. And when there’s pride or money on the line, knowing which No. 15 seed has the best chances of making a run is probably what matters most.

Your chances of picking every upset without a little help: not good at all.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Helix Nebula and the future of Europe’s cloud
• Listening platforms: finding the value in social media data
• A near-term outlook for big data

You probably think I’m being sarcastic, but I’m not. Statistically speaking, football teams should go for it more often, they shouldn’t run on third and long and they’re almost certainly better off going for two-point conversions. The guys behind Statwing laid it all out in a blog post on Monday. What’s more, they’ve uploaded an entire data set of NFL statistics to their service that users can play around with for free to analyze a huge number of occurrences and correlations.

From the blog. One of countless analyses available with the data set.

It’s all about democratizing data

Statwing, you might recall, is one of the “data for dummies” tools I highlighted in a January post about advanced analytics tools so simple anyone can use them. Right now, it’s one of the simplest there is. Here’s how I described Statwing then — although it actually performs more types of analyses:

“You upload data, check the variables you’re concerned with, and it plots their relationship. (It also can describe the variables by highlighting the sample size, minimum, maximum, mean, median and standard deviation.) Graphs are accompanied by explanations as to how strong the correlation is based on various statistical metrics, as well as the results of a linear regression model.”

Greg Laughlin (courtesy of his Twitter profile).

The ease of use is by design, says Statwing co-founder Greg Laughlin. “There’s a general zeitgeist that people should care about data now,” he told me during a recent call, but they don’t always know to get started or really even see how all the hype around data relates to them. Early on its existence, Statwing is trying to answer both of those concerns by building an easy-to-use service that also happens to teach users about statistics, and by offering up some interesting data sets for people to play around with.

The latter part is easy, but valuable. Data sets like the NFL data or one about the Titanic’s passengers let other people into the data game and get them thinking statistically. They get people saying, “‘Oh, I grok that. I see how this interesting, I see how this is useful,’” Laughlin explained.

Building a data-analysis service that’s actually usable by mere mortals is a bit tougher. At its core, Statwing relies on a rules engine that considers the type of data uploaded and the types of variables (a maximum of two right now) a user wants to relate to each other. It can handle between 10 and 15 different analyses right now depending on how one defines them, Laughlin said, but at any rate they’re the ones used most often.

He credits Cloudera co-founder and Chief Scientist Jeff Hammerbacher (with whom, along with Greylock’s DJ Patil, I’ll be doing a fireside chat at Structure: Data on Thursday) with helping Statwing decide to make the rules engine the service’s core.

That has been a wise decision because it lets lay users get what they need out of the service without worrying about the underlying functions. Statwing has users that never click the “advanced” tab that shows the statistical breakdown, Laughlin said. They just use the service, essentially, as a faster way of making charts than using Microsoft Excel, and the headline stating whether or not there’s a statistically significant relationship is all the info they need.

“That’s really exciting for us,” Laughlin said. “… It’s giving them the power of stats without them having to think about it.”

Just one view of how Statwing presents results.

Paying the bills with bigger users

Of course, a startup can’t survive on free and unsophisticated users alone, so Statwing is ramping up its money-making efforts. For example, it has “just turned on the paywall in a really light way” by “maybe” charging really heavy users, Laughlin said. In the future, though, Statwing wants to add support for more variables and larger data sets (there’s a 5MB limit right now), and perhaps build in some predictive analytics.

“That kind of analysis is really powerful, really extensible,” he noted.

As the service grows, Laughlin sees the ideal paying user being someone who currently has to use statistical-analysis software like SPSS or R, but who doesn’t really go beyond the basic functions. That type of user has real business need for the software, he explained, but they don’t need all the complexity and arcane statistics dressing that comes along with that that type of product.

Some people don’t want advanced analytics democratized, Laughlin added, because they think people can’t ask the right questions. On the contrary, Statwing’s theory is that most people just struggle with the logistics of cleaning and formatting data and then knowing the terminology associated with the business questions they want to ask.

Back to football …

But forget business users — when will football coaches start caring about statistics?! Maybe not any time soon. Laughlin said a friend of his who works on the MIT Sloan Sports Analytics Conference sees a lot of interest in analytics from the higher levels in sports organizations, but noted that anecdotal evidence suggests most coaches aren’t too interested in letting data influence their decisions too heavily.

Think of a situation like fourth down and goal on the two-yard-line as akin to a CIO choosing between Oracle and some new whizbang database. Nobody ever got fired for buying Oracle, and nobody ever got fired for kicking a field goal.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

Jack Rivkin, an occasional blogger with deep investment experience, recently shared some of the best practices he was exposed to early in his career working on economic forecasts. He shared some sage suggestions for enterprises to bear in mind as they consider and implement big data strategies. Among his insights:

• Forecasting models can only be as good as the data inputs.
• Be skeptical and hedge when sharing the models by noting factors that could lead to different results.
• The less time it takes to process data, the more valuable it is.
• Constantly improve models and inputs.

Of course, big data isn’t wholly evolutionary — it does bring its own all-new opporunities and risks. Some of the world’s leading data scientists, IT executives and business users will address them at GigaOM’s Structure:Data conference in New York on March 20-21.

Feature image courtesy of Flickr user luckey_sun.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What’s driving the next phase of the e-commerce evolution
• GigaOM Research highs and lows from CES 2013
• How HR can make the case for workforce analytics

But this snafu is just a microcosm of a broader debate over how much stock we should put in web and social media data, and in what cases it’s most valid. It’s hard to figure out how much we should value speed and scale over quality of data. Millions of (presumably) younger people proactively searching or tweeting about a topic provides a huge and theoretically unbiased dataset, while traditional methods of phone calls or focus groups reach a smaller number of (presumably) older people who know they’re being observed, but who also are answering questions directly relevant to the research at hand.

Who’s more accurate: Google, Twitter or your neighbors?

The exact details of the discrepancy are explained in a Nature article published on Wednesday, but it appears to be a case of a lot of data that didn’t mean what Google thought it meant. Google’s search data covers almost the entirety of the web-surfing world and, in theory, can see outbreaks coming before they hit because it can watch the flu-related searches intensify in volume in real time. The Centers for Disease Control and Prevention says Google Flu Trends usually tracks very closely with its own data and can deliver results days faster, Nature writer Declan Butler reported.

Researchers think this year’s discrepancy might have something to do with hyped-up media reports leading to a volume of web searches for flu-related terms that was disproportionate — almost double, nationwide — to the actual number of cases. The CDC claims about 6 percent of the U.S. population was affected with flu-like symptoms during the peak period.

Google estimated more than 10 percent of the U.S. population had flu-like symtoms.

On the other hand, one project called Flu Near You, which relies on volunteers to report cases of flu among their friends and family, estimated a number closer to (albeit lower than) the CDC’s official statistics, perhaps because the data is based on clinical definitions of “influenza” and relies on people expressly reporting known cases. However, Flu Near You claims less than 45,000 participants and, according to Nature, covers only 70,000 people.

Flu Like You’s peak estimate was abour 4.5 percent of the population.

Responding to my inquiry about the discrepancy, a Google spokesperson sent the following statement:

“Flu Trends is meant to be a complementary tool to the surveillance systems used by the CDC. Since its initial launch in 2008 and through this flu season, Flu Trends has accurately predicted the start and peak time of flu season. However, this season our models estimated a higher influenza like illness rate than the Centers for Disease Control in some regions. As we do each year, we will be performing a model analysis and potential model update to improve the accuracy of the tool.”

And while Google’s predictions might be prone to the undue influence of a fear-mongering media environment, CDC researcher Lyn Finelli told Nature she’s even more skeptical of efforts to track flu outbreaks using Twitter data. She cites a low signal-to-noise ratio and a population of largely young-adult users that doesn’t align with the country’s overall demographic makeup.

To the contrary, however, Johns Hopkins University computer scientist Michael Paul told Nature that he’s a big believer in Twitter data, especially because it generates a large dataset that’s less susceptible to sample errors than smaller-scale projects such as Flu Near You. He claims to have developed a model that can accurately track the flu using Twitter, something a handful of other projects are already working on.

Pollsters struggle with the web, too

But flu statistics aside, questions over the validity of Twitter, Google and other web sites as data sources are nothing new. Last year, for example, I profiled a company called the Dachis Group that has devised a method for tracking companies’ presences, buzz and sentiment on social media. It claims its algorithms for ranking the buzz around Super Bowl XLVI advertisers were far more accurate — or at least yielded drastically different results — than USA Today‘s traditional AdMeter rankings of Super Bowl ads based on phone-based polling.

Although people appear generally willing to do away with phone surveys and other marketing-based polling efforts, there’s a lot more skepticism when it comes to using the web to predict political elections and gauge response to culturally popular events such as presidential debates or the Olympics. I covered both sides of the debate in October, as pre-election fever was in full force and many people were atwitter about Twitter’s tweets-per-minute counts during the presidential debates. What side experts fall on seems to depend on how much they trust the demographics, the subjects themselves, the sample size and how well someone can actually analyze sentiment in text.

Even on Google, politics has proven that interest doesn’t necessarily signify intent. Leading up to the presidential election in November, Mitt Romney was trending quite a bit higher than Barack Obama in search volume. Election night, however, was a different story, with Obama winning in a landslide.

Perhaps the best advice on how to deal with web data comes from Harvard epidemiologist John Brownstein, who told Nature, “You need to be constantly adapting these models, they don’t work in a vacuum. You need to recalibrate them every year.”

As web usage and users change along with the world around them, there’s really no guarantee that a single data point means the same thing or has the same effect from year to year. Even search is under attack by companies trying to proactively surface content for consumers before they know to look for it.

When accuracy is paramount, no place — Twitter, Google, the telephone or the wisdom of crowds — is the holy grail; they’ll all have to play a role.

Feature image courtesy of the Centers for Disease Control and Prevention.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• Listening platforms: finding the value in social media data

David Brooks’s recent column in the New York Times is a good example of this. He coined the term “data-ism” (which is quite apt) to describe a newfound penchant for reducing everything in our worlds into a number or statistic. Skeptical of this data worship, he is — rightfully — inclined to rebel.

But everything Brooks mentions in his article is really just statistics, the stuff academicians and businesspeople have been doing for years. It doesn’t take any revolutionary technological advances to measure the effect of political spending on campaign results or the idiosyncrasies in how a president speaks. At the best, these types of analyses are enlightening; at the worst, they’re overkill.

Data can be an unwelcome disinfectant

Like Brooks’ pointer to a study about whether there’s such a thing as hot hand in basketball. Or a recent debate (that got an incredible amount of undeserved digital ink from Deadspin and Nick Carr) about whether to adjust the points and frequency of Scrabble tiles based on what letters actually appear most in the English language. Right or wrong, who cares?

Unless you’re a professional gambler or in the sports business, sports are supposed to be fun; an escape from reality. Buying into things like hot hands, sweet spots, ancient curses and concern rays (thanks, Dave Barry) are part of the rooting experience. If it wasn’t for coaches’ insistence on punting on fourth down, I could watch an entire football game and not think about probabilites once.

As for Scrabble, well, it’s a game and it’s fun. People like it as it is. What’s next, lobbying to change the distribution of resource cards in Settlers of Catan to account for the relative value of each given recent drought conditions?

Likewise, while Max Levchin’s vision of the future recently had Nick Carr concerned about big brother (read my colleague Mathew Ingram’s take on it here) my takeaway from Carr’s blog post was more about the threat of a sterilized world. Human beings are not rational actors, and many of us don’t want to be — regardless of what the data says. We buy enormous sodas even though we don’t finish them, we demand all-you-can-eat data plans even though we don’t consume that much data and, directly addressing one of Levchin’s predictions, I bet many of us would willingly pay more for flat-rate auto insurance even if utility-style billing based on our real-time driving behavior would save us money.

Reducing the things we like — watching sports, eating, web surfing, driving — to data points ruins the experience of living carefree and exposes our optimistic anything-can-happen attitudes to a cold, surgical light. If I thought these were the pinnacle of data’s achievements, I’d rebel, too.

Data’s real promise is innovation

Thankfully, however, I’ve been lucky enough to spend my days speaking with some of the smartest data minds around and covering some truly revolutionary technologies. If there’s one thing I’ve learned, it’s that the real value of data isn’t just in uncovering statistical realities, but in finding methods for doing so where it was hitherto impossible and in creating entirely new products that change the way we interact with our world.

Big data is a technological revolution centered around collecting, storing and processing more data of more types than ever before. It’s also about doing all this stuff faster than ever before as data streams in from sensors, servers, Twitter, web surfing and however else we’re generating data. Data scientists are thinking up clever ways to stitch this data together, apply statistical techniques and do all sorts of things. They’re optimizing commerce, clearing traffic, insuring against inclement weather and even detecting genetic markers that might lead to a cure for cancer.

Climate Corporation’s policies are based on some incredible data science.

If you want to hear a lot more about what’s possible, come to our Structure: Data conference March 20-21 in New York.

Yes, there’s some value to what David Brooks calls data-ism — there’s a lot to be learned simply from monitoring new data sources, and a renewed focus on visualization means interesting data is now presented in ways that anyone can and might actually want to digest. But the real reason people are, or should be, excited about data is the promise of doing important things faster and better than previously possible (where those things were even possible before).

Talk to me when you’re able to predict a flu outbreak in real time based on automobile traffic patterns, smart grid data on heater usage and an uptick in illness references on Twitter. If you just wanna tell me that, statistically speaking, chicken soup doesn’t actually appear to affect the longevity of the common cold, well, I think I’ll pass. Chicken soup makes me feel better.

Feature image courtesy of Shutterstock user Jirsak.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Listening platforms: finding the value in social media data
• Cloud computing and trickle-down analytics

With that in mind, here are six of the best free tools I’ve come across for helping we mere mortals analyze our data without having to know too much about, well, anything (I’d keep an eye on the still-under-wraps Datahero, too). I’ve gathered some personal data and tracked down some interesting public data sets to help demonstrate what a novice can do with them. Someone with more skills can certainly do a lot more, and larger datasets will provide greater statistical significance.

BigML

BigML is to machine learning what Blue Moon is to Belgian ales: a simple approach to something generally more complex — but also rather accessible and good enough to do the job in a pinch. I explained the service more thoroughly in recent post about it being used to generate predictions of Kickstarter success, but here’s how it works, in a nutshell: Users upload and format data (which is actually pretty easy), BigML discovers the myriad relationships between the variables and creates a predictive model, and users enter hypothetical data and receive a prediction.

I’m pretty bad when it comes to entering my data into Fitbit (see disclosure), but I was relatively good for a month this summer as I prepped for the Warrior Dash, and that’s the data I used to demonstrate BigML. This prediction of how many calories I can expect to burn in a day would work a lot better if I had a bigger sample size and hadn’t occasionally forgotten to log calories and hours slept, but you get the point. The first image is the model the service generated; the second is the prediction interface.

Google Fusion Tables

The user interface for Google Fusion Tables  isn’t what I’d call pretty (“sparse” is probably a better description), but the still-in-experimental-mode visualization tool sure is easy if your data is nicely formatted. I created this interactive map simply by uploading a publicly available dataset about gun violence and clicking the button to create a map:

For this simple comparison of gun ownership and gun homicide rates, I just checked the countries by which I wanted to filter the chart. Easy.:

Infogram

If you have really simple data — like a few columns and a handful of rows — Infogram might be the easiest to use of the bunch. The company launched last year with a variety of infographic templates, but it has since expanded to include a large number of charts and graphs, too (including line, pie, pictorial, treemap and bubble). Furthermore, it gives sample data, which you can use as an example to enter your own or format the table you want to upload, and the interactive charts embed nicely into web pages (ours, at least).

Here are the top 10 things I ate during the time I was logging food via Fitbit, excluding copious amounts of beer, water, coffee and Diet Pepsi that I didn’t record.

In July, I made this chart with Infogram comparing infrastructure spending trends among internet companies.

And here’s a sample of the simplest chart in the world.

Many Eyes

Many Eyes is a free web service run by IBM that includes a wide variety of visualizations ranging from maps to pie charts to scatter plots. But what makes it stand apart from the others is the suite of text-analysis tools it offers — not only are they fairly novel, but all they require users to do is paste a page of plain text into the web interface and press a button to visualize it. I used it to analyze the last 15 posts I’ve written for GigaOM.

What did I find? For starters, I use the words “data,” “Facebook” and “users” a lot.

When it comes to two-word combinations, “big data,” “data centers” and “hard drives” are among the biggies.

This one is particularly interesting, showing how I tend to form phrases around certain words with common conjunctions, or just a space, in between.

Apparently, out of 10,013 words, I only used “cloud” 20 times. I usually followed it up with “provider,” “servers,” “computing,” “-based” and “providers.”

For fun, I also made a word cloud based on couple month’s worth of Fitbit food logs. It turns out, you can take the boy out of Wisconsin, but …

Statwing

Statwing might be my favorite of the bunch, if only because it’s so simple yet actually tries to teach users about statistics. You upload data, check the variables you’re concerned with, and it plots their relationship. (It also can describe the variables by highlighting the sample size, minimum, maximum, mean, median and standard deviation.) Graphs are accompanied by explanations as to how strong the correlation is based on various statistical metrics, as well as the results of a linear regression model.

To demonstrate Statwing, I went back to the Fitbit data. Of the variables that Fitbit tracks, some correlations are easy to predict (e.g., steps and calories burned), but I was kind of surprised to see that the 86 minutes a day I spent being fairly active really weren’t that good of an expenditure of my time.

Tableau Public

Tableau Public, the only free version of the popular business-intelligence software, was clearly designed with business users in mind. It expects a lot of structure in the data, and although you can edit almost every aspect of it within the application to get it into usable shape, the service doesn’t allow much guidance if you don’t speak the language of BI (it also requires Windows). But the software is very good at deciphering the characteristics of different variables, the drag-and-drop operation makes it kind of easy to experiment and the wide array of visualizations look really nice.

Using my Fitbit data (and here’s where you see how lax I am at data entry), I created a line graph comparing the calories I ate each day with the calories I burned. Assuming I didn’t go crazy eating on the days I forgot to make entries, the good news is I never ate more calories than I burned. (Note: Although these are static images, Tableau Public actually lets you embed interactive charts, which I’ve used in the past on several occasions, but they don’t always fit well within our pages.)

Here’s one I played around with a while back charting Amazon’s “Other” revenue againt the number of objects stored in Amazon S3.

Finally, here is my first-ever (I think) Tableau chart, which uses the raw data on government takedown requests that Google provided along with its Transparency Report in October 2011. You can read that post and play with the interactive version here.

There is, however, one disclaimer that applies to all of these tools: I didn’t get into cleaning and formatting data, which can be a somewhat arduous process. Many tools expect some sort of structure to the data — the X axis to be in columns and the Y axis in rows, measurements without units (e.g., grams), etc. — that just isn’t present if you’re downloading an Excel or CSV file rather than creating it yourself. Sometimes, with comprehensive datasets like your Fitbit Premium data, you’ll have to separate or combine the relevant data into new spreadsheet tables before uploading it to a service. But once you have the data ready to go, these tools can help you analyze it, visualize it and hopefully glean some insights from it.

Disclosure: Fitbit is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Health care and big data in 2012
• Cloud computing infrastructure: 2012 and beyond

On Thursday, Barnes & Noble VP Marc Parrish wrote a provocative guest post for The Atlantic explaining how big data technologies could help identify mass murderers such as James Holmes and Adam Lanza before they actually commit their heinous acts. As much I like to prescribe big data as the solution to various problems — and as much I wish Parrish’s solution was the right answer — his assessment is probably a bit fantastical.

There are a whole slew of reasons Parrish’s hypothesis might fall short, the most obvious of which was pointed out early and often by commenters to the post: There just aren’t enough incidents of gun-powered mass murder to draw strong assumptions about what types of behavior typically precede such an attack. Here’s an excerpt from the most-popular comment, from JLR84:

“The ‘patterns’ that you think of indicative of a spree-killer in the making are far more common than you think, meaning that the whole thing would be rife with false positives. So many that the authorities would never be able to follow up on them, and the system would quickly be ignored. … What you think is a ‘large amount of ammunition’ isn’t. … Spree shooters use relatively small quantities of ammunition compared to the average enthusiast, all things considered. Regular violent criminals, even less.”

Another strong argument has to do with ownership — who actually owns and purchases the guns used in mass murders, or any other homicide, for that matter? If a shooter steals guns or uses his father’s gun, for example, the shooter’s name might never find its way into a government database. Without other evidence linking the possession of guns with intent to do harm, trying to predict who’ll commit horrific crimes with guns might be a fruitless task.

But that doesn’t mean there isn’t a less glamorous way to use data as a means for curbing violence by guns. Perhaps — if someone were willing to undertake a massive data collection effort, carefully selecting, gathering and analyzing international data on topics such as poverty rates, mental health, gun laws, drug laws, violence in the media, known information about those who have committed murder, family composition, health care, etc. — we could actually identify commonalities or anomalies that shed some light on why certain countries have higher murder rates than others. It’s possible that Americans’ easy access to guns only facilitates a willingness to kill that has been cultivated by other factors and extends far beyond the small fraction of deaths attributable to mass murder.

By way of comparison, Canada has a rate of 1.6; the United Kingdom is 1.2. (Source: Wikipedia/UNODC)

Guns certainly make it easier to kill, but they probably don’t, by their mere presence, inspire violent tendencies. At 4.2 murders per 100,000 people, according to the United Nations Office on Drugs and Crime, we’re well above peers such as Canada, Australia and Western European countries — and even above many African, Middle Eastern, Asian and Eastern European countries. And although the percentage of homicides committed with guns is high in the United States (the UNODC says 68 percent, or 9,960 murders, in 2010, while the Guardian‘s Datablog uses data showing 60 percent), there’s no guarantee many of those murders wouldn’t have happened or have been attempted by other means.

Gun homicide rate per 100,000 people by country. Updated: the X axis now correctly reads “Guns/100 people.” (Source: KDnuggets)

Looking at statistics about guns alone does little to answer the question. Over at KDnuggets, an online community dedicated to data mining, there has been some discussion about the correlation between the number of guns in a country and the number of gun deaths. Excluding the United States — which easily tops the charts in terms of guns per capita 100 people and gun homicide rate (among countries with a per capita GDP of more than $20,000) — it’s hard to say with any statistical certainty that having more guns actually does lead to more murder by guns.

So maybe big data really can help solve America’s penchant for killing by helping us understand why, exactly, so many of our citizens feel so compelled to do so. Instead of trying to figure out when people are going to pull the trigger, let’s focus on answering why people are so willing to kill in a country that appears to have so much.

Feature image courtesy of Shutterstock user Sascha Burkard.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Listening platforms: finding the value in social media data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

It’s worth mentioning that Silver wasn’t the only statistician to perfectly predict the presidential race, either. In terms of Electoral College votes, Simon Jackman of Pollster did so, as did Josh Putnam of Davidson University. Save for Florida, Sam Wang of the Princeton Election Consortium fared very well, too, and actually nailed the popular vote split. Slate has a nice interactive chart showing how various statisticians and pundits fared in their predictions; there certainly are more predictions and models floating around that haven’t been included.

The important takeaway, however, is that the people who nailed the outcome didn’t achieve their results by cherry-picking data that served their political interests. They did it because they’re professional statisticians whose success depends on accurately predicting the outcomes of events, not on cheerleading for the outcome they might personally desire or that will drive the highest ratings. Even if the data they’re working with is somewhat biased — as some individuals and organizations suggested to me is the case — the science comes in being able to take the data sources for what they are and accurately weigh their relevancy.

Wrong. Data ALWAYS does, starting w/ collection | Data doesn’t play politics & says Obama win is.gd/piGdIb by @derrickharris

— Liberationtech (@Liberationtech) November 6, 2012

In business, this is the shift in thinking that’s driving the movement toward big data and advanced analytics. Forward-thinking companies want to use data to make the right decisions, not to back up their predetermined decisions based largely on gut instinct. But there’s an unprecedented amount of data at their disposal — some good, some bad — which is why data scientists who can figure out what sources to use and how to use them are in such high demand right now.

So in 2014 and and 2016, pollsters are going to keep polling, statisticians are going to keep analyzing those polls (and whatever other factors they choose to include) and, maybe, pundits and the media will pay some attention to what they’re saying. Probabilities aren’t promises etched in stone, and a vote either way can change the face of close elections like this one. But no one should be surprised when someone whose only job is to get it right does just that.

Feature image courtesy of Flickr user Carolyn Coles.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Listening platforms: finding the value in social media data
• The importance of putting the U and I in visualization