The organizers of the first Industrial Internet Quests have since received more than 3,000 submissions and were expecting to announce on Wednesday the contestants who will receive a total of $600,000. One submission for the flight competition has earned $100,000 for its developers, a five-person team from Singapore.

Kaggle has hosted data-science competitions for several other brand-name companies, from Facebook to Ford. Its publicly available leaderboards make data science a bit like a spectator sport, and open-source education on machine learning and natural-language processing makes it possible for lots of people to compete.

Demand is sky-high for data scientists and application developers, and farming out one-off projects is a common practice in all sorts of industries. That’s why it’s not surprising to see even big companies like GE turning to the crowd for data science solutions. And it’s why this sort of news could become more common in the future.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution
• A near-term outlook for big data

Everyone calls himself a data scientist now — and there’s a reason for that. The title “gets you 40 percent more money,” says Kaggle CEO Anthony Goldbloom. ”The problem is that it’s hard to know how good someone really is until six months down the road when you realize they haven’t done anything.”

His argument is that folks who have done well in Kaggle competitions over the past two years — insurance actuaries, mathematicians, students, chemists — have proven they have what it takes.

And Kaggle bona fides are becoming currency. This job posting for a New York Times data scientist lists participation in a Kaggle competition as a key criterion.

Connecting the right data scientists with the right problems

With Kaggle Connect, the company is making its two top tiers of competitors — it’s an invitation-only list — available to companies on an individual basis. “If Pfizer comes to us with a problem that is maybe not well specified enough and needs more iteration than a competition would allow, we can provide a data scientist that suits that problem,” Goldbloom said.

The customer pays a subscription cost of somewhere between $30,000 and $100,000 per month to gain access to appropriate data science resources. Kaggle gets a cut of that money and the data scientist gets the rest — although Kaggle is not breaking out the percentages.

In the interactive chart below, click on the map to bring up the name, picture and profile of the Kaggle Connect member.

What Kaggle brings to the table is a roster of people who have performed well in its competitions. What the companies provide is a juicy problem to solve and data to use in that quest. In some ways this is an extension of what Kaggle has already done with EMC’s Greenplum division, although that project required the use of Greenplum’s Chorus toolset.

The top two of eight total tiers of 80,000 contestants will initially serve as the invitation-only talent pool for Kaggle Connect. That’s about 1,500 Kagglers (if that’s a word). Kaggle began running data science competitions in early 2012 and started publishing its leaderboard of top big data problem solvers last September.

We’ll see how this all proves out, but if Kaggle success is really a predictor of big data chops writ large, expect to see a lot more Kaggle boasts on resumes going forward.

Feature photo courtesy of Shutterstock user Dirk Ercken.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Infrastructure Q1: Cloud and big data woo enterprises

Under the new alliance, Kaggle’s community of big data eggheads can use Greenplum’s Chorus big data application to solve real-world problems.

“We’ve had good adoption of Chorus and companies’ internal data workers are using it to do data science so they now have the tools, but honestly they don’t have all the people they need,” said Josh Klahr,  VP of product management for Greenplum. “Now you can search the Kaggle community based on rank, expertise, location and invite them to work on your challenge using Greenplum Chorus.”

Playing Yenta to big data players

Kaggle ranks its participants much the way the USTA ranks tennis players. And that community is growing fast — when Kaggle started fundraising in August, there were 11,000 members, now there are close to 60,000, said Anthony Goldbloom, CEO of Kaggle, who said this is the first such vendor partnership Kaggle has done. (GigaOM has worked with Kaggle and Splunk on the GigaOM WordPress Challenge: Splunk Innovation Prospect.)

Also on the partner ecosystem front, Greenplum inked a deal that gives Chorus users access to Gnip’s historical Twitter feeds and will let Chorus users import Twitter streams into their Chorus sandbox for analysis. And finally, Chorus is partnering with Tableau, the popular analytics tool so that users can provision Tableau workbooks from their Chorus data sources.

Big data is one area where building a broad ecosystem of data providers is incredibly important. Putting good data scientists together with great data sets is incredibly important, said Ben Woo, managing director of research firm Neuralytics. “Big data is awfully short on the kinds of people who’ve done this work before and, frankly, people who give a damn. This sort of matchmaking is valuable.”

Goal: Melding public and private data to spark new insights

This convergence of publicly available “sentiment” data from sources like Twitter and internal business data lets data scientists ask interesting questions or find interesting questions to ask. For example, a pharmaceutical company has lots of its own data on a new drug. What it may not have is the sort of information about unforeseen side effects that might surface on Twitter or blogs after the drug is released.  ”If you can match Twitter feeds and patient forums, you can find out unexpected things — see that maybe people are switching from your drug to another. Analyzing that discussion can be incredibly important,”  Goldbloom said.

In related news, Greenplum, as promised last spring, is open sourcing Chorus as the OpenChorus Project under the Apache 2.0 license.

Feature photo courtesy of Flickr user Kevin Krejci

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• A near-term outlook for big data
• The importance of putting the U and I in visualization

Much of this evidence comes via Kaggle, a platform where companies and organizations award prizes for the best solutions to their predictive-modeling needs. In September, for example, I covered a first-time Kaggle user and admitted data science neophyte named Carter S. who won a competition using a simple but effective method he dubbed “overkill analytics.”

Impressive, sure, but Carter builds insurance-industry risk models for a living. While he’s able to learn new techniques such as natural-language processing and social network analysis as he goes, he’s no stranger to a linear regression. But what if someone’s only formal experience with computer science was a single undergraduate programming course?

Ask Luis Tandalla. That was his case before he took a handful of free online classes last year on Coursera. Yet the University of New Orleans senior recently scored his first victory in a Kaggle competition hosted by the Hewlett Foundation where he had to devise a model for accurately grading short-answer questions on exams. Not bad for a college senior who didn’t really know what artificial intelligence and machine learning were before he signed up to learn them.

Once Tandalla got started, he told me, he got passionate about learning more. So he also took Coursera classes on natural-language processing and probabilistic models, began studying on his own outside the online lectures and even got active on Kaggle (this was his first victory in five competitions). He’ll receive his bachelor’s degree in mechanical engineering in May 2013, but now Tandalla says he wants to pursue a master’s degree in machine learning and start his own predictive-software company

The Coursera connection

Maybe Tandalla isn’t so unique after all. The second- and third-place finishers in the Heritage Foundation competition, it turns out, also learned machine learning on Coursera. The latter, Xavier Conort, is a 39-year-old actuary from Singapore who just decided to become a data scientist last year and is now Kaggle’s top-ranked competitor.

Andrew Ng

Stanford professor and Coursera co-founder Andrew Ng — who teaches the machine-learning class that all three top finishers took — doesn’t think their success is just coincidence. If you’re not trying to make the types of contacts students at top universities are after, and your goal isn’t to perform advanced research, he explained, online education platforms such as Coursera (and, I’ll add, Udacity and EdX), can be incredibly valuable.

In particular, Ng said, “Machine learning has matured to the point by where if you take one class you can actually become pretty good at applying it.” Familiarity with algebra and probabilities are certainly helpful, he added, but the only real prerequisite to his course is a basic understanding of programming.

And with machine learning becoming “one of the more highly sought-after skills in Silicon Valley,” Ng said, corporate recruiters say just completing a single course can significantly boost someone’s salary and job prospects at companies where such knowledge is still in short supply.

“I bet many students are going on to [do] great things because of these courses [even if we never hear about it],” Ng said.

Why it works, and why it could change the world

Ng thinks the current incarnation of online education platforms work so well because they’re essentially nurturing the already-talented students who seek them out. Some professionals, he explained, take courses to learn skills such as machine learning or iOS programming that weren’t in vogue or didn’t even exist when they earned their computer science degrees just a decade ago.

Furthermore, with students able to learn at their own pace, there’s a lot of valuable information disseminated in the discussion forums.

Free access to the best teachers around doesn’t hurt either. Ng said he couldn’t teach his course so well if he hadn’t spent so much time living in Silicon Valley learning best practices from some of the smartest computer scientists on the planet. That experience lets him spend less time teaching algorithms for the sake of algorithms and more time talking about how one might actually apply machine learning in the field.

Ng says that’s a more important than just understanding the algorithms in a vacuum. He compares it to learning how to write a computer program instead of just learning the syntax of a programming language but not being able to string commands together into something useful. This approach isn’t entirely unique among the new order of online educators: On Udacity, for example, Google VP and Stanford professor Sebastian Thrun, centers the Computer Science 101 curriculum around learning Python in the context of building a working search engine.

The value of this opportunity wasn’t lost on Tandalla. He said he can feel the passion that professors have even through the pre-recorded video lectures, and it feels good knowing you’re learning from the people who literally wrote the book on the subject you’re studying.

Who knows who’s the next Einstein

But ultimately, minting new data scientists — even Kaggle winners — is low-hanging fruit. Ng said we don’t yet know how much impact online educations platforms like Coursera can have. In all fields, there are talented people all over the world who just need an avenue to hone their skills and a chance to distinguish themselves.

“It makes me wonder,” Ng said, “if the next Albert Einstein is a little girl in Afghanistan who just needs [the opportunity to access quality education].”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• AWS Storage Gateway jolts cloud-storage ecosystem

The first contest in the series involves answering a question, albeit a difficult one. From the contest page:

“How can we make heterogeneous (dissimilar and incompatible) data sets homogeneous (uniformly accessible, compatible, able to be grouped and/or matched) so usable information can be extracted? How can information then be converted into real knowledge that can inform critical decisions and solve societal challenges?”

This a problem that’s magnified in government agencies, but that plagues companies of all types that try to get started with big data. While future big data strategies might mandate a particular data format or other standards, the past and, often, the present is a messy pile of stuff created by different divisions within different agencies or departments. The dream of creating spectacular algorithms, beautiful visualizations and uncovering hidden insights often only comes after untold man-hours spent cleaning and munging data (often with Hadoop) into formats that software systems can work with.

The registration deadline for the first contest is Oct. 13, and the submission deadline is Oct. 19. The 1st place prize is $1,000. Later contests will focus on more domain-specific fields such as energy, health care and earth science.

Although the possibility of influencing big data strategies within some of the country’s most advanced agencies might be novel, crowdsourcing solutions to these types of difficult problems is becoming rather common. TopCoder exists as a competitive platform for solving application development and design issues, and the Big Data Challenge is just the latest it’s hosting for NASA via the agency’s Center of Excellence for Collaborative Innovation and NASA Tournament Lab. There’s also general-purpose platform InnoCentive and wildly popular data science platform Kaggle.

Feature image courtesy of Shutterstock user Prokhorova Nadila.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining work in the digital age: an analysis by GigaOM Pro
• 12 tech leaders’ resolutions for 2012
• Examining the rise of crowd labor platforms in 2012

Not only was this Carter’s first win, it was also his first contest. You can read the detailed explanation of his victory on his blog, but the gist is that he didn’t get too involved with complex social graphing to determine relationships or natural language processing to determine topics readers liked. He figured out that most of what people liked came from blogs they’ve already read, and that the vast majority of posts people liked fell within a three-node radius on a simple social graph.

Statistically speaking, he did a generalized linear regression model, followed by a random forest model and averaged the results. “I’m not sure it’s a very unique technique,” he told me, “but it’s certainly a very powerful one.”

Source: Overkill Analytics

And therein lies the beauty of overkill analytics, a term that Carter might have coined, but that appears to be catching on — especially in the world of web companies and big data. Carter says he doesn’t want to spend a lot of time fine-tuning models, writing complex algorithms or pre-analyzing data to make it work for his purposes. Rather, he wants to utilize some simple models, reduce things to numbers and process the heck out of the data set on as much hardware as is possible.

It’s not about big data so much as it is about big computing power, he said. There’s still work to be done on smaller data sets like the majority of the world deals with, but Hadoop clusters and other architectural advances let you do more to that data in a faster time than was previously possible. Now, Carter said, as long as you account for the effects of overprocessing data, you can create a black-box-like system and run every combination of simple techniques on data until you get the most-accurate answer.

I wrote about the same general theory recently in explaining why Sparked.com’s Daniel Wiesenthal believes that big data (i.e., lots and lots of data combined with new storage and processing technologies) improves the practice of data science (i.e., the application of statistical techniques to data). The gist of his theory is that although complex models are great for small data sets, simple models can close the accuracy gap when applied to large data sets. Combine that with infrastructure that can process a lot of data relatively fast and support a wide variety of jobs, and you have a simpler, faster equally effective method.

Still, Carter said he didn’t get involved in Kaggle just to prove the effectiveness of overkill analytics. He does hope to get exposed to new data science techniques that haven’t yet caught on in the insurance industry, and he also wants to make a name for himself. When you work for a company with little turnover, he said, your professional network doesn’t grow too much, but doing Kaggle competitions is a great way to meet other data scientists — and winning is a great way to earn respect.

Ali Ahmad (username Xali) won the separate Splunk Innovation portion of the contest. According to a statement from Splunk, he “used Splunk’s built in statistical and visualization features to map out the relationship between blogs containing YouTube videos with those that are most likely to be viral, as measured by likes and shares. As a bonus, he fed the data into an app to view the YouTube videos most commonly liked and shared via WordPress blogs!”

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOm. Om Malik, founder of GigaOm, is also a venture partner at True.

Feature image courtesy of Shutterstock user nasirkhan.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

Kaggle hopes to change that with a twist on its predictive-modeling competition platform that makes public the competitors in invite-only private competitions. Think of it like watching a major tournament in golf or tennis, where you can watch the best in the world shoot it out to see whose algorithms are king. Kaggle’s tagline is “We’re making data science a sport.” Maybe now it can make data science a spectator sport.

Top five in the GigaOM/Splunk competition.

Actually, Kaggle has been running private competitions — in which customers’ generally remain anonymous and keep their challenge descriptions vague except for those invited to work on the data — since about the beginning of 2012. In the past, though, even the competitors remained a mystery. It also posts leaderboards for all public competitions and a cumulative leaderboard. But as any sports fan knows, there’s nothing quite like watching a tournament where only only the best of the best can play, and where the pressure is on.

Now, says Kaggle Founder and CEO Anthony Goldbloom, private competitions are more like running the U.S. Open in that others can watch the leaderboard and see how the invited data scientists are faring. It’s primarily a feature so other data scientists on the Kaggle platform can gauge their relative performance and get a little more motivation to step up their game and make it to the invitation-only competitions, but I think it could become geek spectator sport under the right conditions.

If you’re wondering which U.S. Open he’s talking about (golf or tennis), don’t fret — had Goldbloom been asked whether Kaggle is more like golf or tennis before it launched, even he might have guessed wrong. He’d probably have guessed tennis, in which certain players excel on certain types of courts, like Roger Federer on the grass court at Wimbledon, or Rafael Nadal on the clay court at the French Open. So, someone who works in biotech might naturally prevail in those competitions, while a natural-language processing specialist might do best in competitions with lots of text to mine.

It turns out Kaggle is more like golf, in which a dominant player like Tiger Woods can win on pretty much any course he plays. Newcomers can still win, especially because there are plenty of good data scientists still making their way to the nascent Kaggle platform, but, Goldbloom says, the really good ones will adapt their skill sets to whatever is necessary for any given competition.

The first private competition open to public viewership began on Wednesday, and is somewhat unique in that the sponsor is willing to share its name and its challenge. It’s insurance provider Allstate, and it’s trying to predict customer churn. According to Goldbloom, the prohibitive favorite is Jason Tigg, an Oxford physicist turned hedge fund manager, but Indianapolis actuary Shea Parkes and apparent mystery man Jonathan Peters are names to watch.

There you have it, sports fans. Place your bets accordingly.

Note: This story was updated to reflect that Oxford physicist Jason Tigg decided not to take part in the first private competition open to public viewership.

Feature image courtesy of Shutterstock user photofriday.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012

We’re hosting a competition on Kaggle’s data science platform to find the best models around likely readership across the WordPress (see disclosure) ecosystem of blogs. Here are the details:

The challenge is to predict whether a particular user will like a particular WordPress blog post.  The data consists of eight weeks of posts collected by WordPress, along with anonymized user responses to each post.  This challenge is an interesting mix of natural language processing (the raw blog posts) and metadata on the blogs and users. Contestants can download the data and submit prediction through the Kaggle platform, but a new feature for this competition is that they will also have free access to a Splunk server containing all the data, which they can employ for data exploration, visualization, feature extraction and modeling.

Aside from offering resources to work on the data, Splunk is also putting up $25,000 in prize money. The winning model will receive $10,000, second place $5,000, third place $3,000 and fourth place $2,000.

There’s also a $5,000 Splunk Innovation Prize for the most innovative use of data science, whether that comes in the form of a visualization, app, business model, you name it. Submissions for the latter track can be submitted through Kaggle’s new Prospect platform. Winners for both competitions will be announced at GigaOM Mobilize in September.

You can find out more about the competition here. Good luck!

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOM. Om Malik, founder of GigaOM, is also a venture partner at True.

Image courtesy of Shutterstock user sukiyaki.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Big data 2013: key trends and companies to watch

What Kaggle has come to realize, Co-Founder President and Chief Scientist Jeremy Howard told me, is that very few people in the world know more about machine learning than do the roughly 40,000 individuals who compete in Kaggle’s competitions. At the same time, the big data movement is putting the relatively complex but rewarding practice of machine learning on the radar of more companies, and they want to know how to leverage it for their businesses.

But without the right knowledge, Howard said, companies new to machine learning will likely either under-reach and lose out on possible benefits or overreach and risk overwhelming themselves. Who else better to educate them than the world’s largest (and, presumably, best) online data scientist community?

The way Prospect works is rather simple: Customers submit a sample of the data they want to better analyze, and competitors play around with the data and propose ideas on how to best use it. When the competition deadline comes, a winner is selected by some combination of community voting and customer selection. Customers pay their fee to Kaggle and the reward to the winner, and are free to do what they want with the advice.

“The worst case scenario is you spend a reasonably small amount of money and decide this isn’t a product for you,” Howard said, although he suspects most companies will end up wanting to take the next step and actually develop and implement models.

Inaugural Prospect customer Practice Fusion, at least, plans to do just that. The electronic health records provider is offering a $500 prize for the best idea on how to use its data, and then will conduct a $10,000 competition to actually develop a model based on that idea.

Howard said this process is part of the “analytics value chain” that Kaggle hopes to create as its own business model matures. Companies can use the platform to figure out how to best use their data, then build and train their models, figure out how to implement them in production and at scale, and finally maintain and improve them. “You have to get everything right in order to actually leverage your data,” he said, and Kaggle wants to help at every step.

Kaggle thinks it can become the go-to source for anyone needing help with machine-learning because its approach forces the platform to prove its worth. Fairly or not, Howard characterizes traditional analytics consulting firms as “having quite a lot of snake oil salesmen” that often don’t “have anything to back up their pretty pictures and words.” On the contrary, he says Kaggle’s competition approach “is by far the best method in the world for actually building and training [predictive] models.”

He acknowledges, however, that although he thinks the approach will continue to be the best one as Kaggle grows out its value chain, it might not. But Kaggle is a data science startup, he said, so it’s not about to act solely on his intuition: “We have to test these hypotheses.”

Feature image courtesy of Shutterstock user VLADGRIN.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization

Amid all that, you might think that guessing the winner would be hard. But one man predicted the result… sort of.

Meet Martin O’Leary, a glaciologist and data nerd who works at the University of Michigan. O’Leary, who calls himself a “recovering mathematician,” decided to use statistical analysis on Eurovision to try and understand which country would win:

Sweden’s going to win, unless it’s Malta, or maybe somebody else. If you average together the taste in pop music of all of Europe, you get a Hungarian. Don’t trust the scores on Saturday night, they’re just toying with your emotions.

And guess what? Sweden won!

You can read O’Leary’s entire series of posts to understand how he arrived at that conclusion, but here’s the quick version.

He did it by taking performing a Bayesian analysis on a wide range of previous Eurovision results, taking into account a few important factors with his model. First, the recognition that while Eurovision is a song competition, the results are not really based on the quality of song — although it can play a part. Then there’s the fact that there are semi-finals (held to whittle the number of contestants down) that allow some songs to be tested in public.

And then, most importantly, there’s the recognition that Eurovision is heavily influenced by transnational politics: which countries like which other countries plays a big part in voting. Entrants from the Balkans, for example, tend to trade votes with each other. Greece nearly always awards maximum points to Cyprus and vice versa. Big European powers like the U.K, France and Germany perform less well than smaller countries with lots of positive sentiment toward them.

But while O’Leary’s number-crunching enabled him to predict Sweden’s victory — and claim a victory for data modeling — it wasn’t infallible.

In particular, his prediction that Malta would be in the mixed seems to have caused some consternation. His guess was so exciting to the Maltese that it even made the newspapers, but in the end the country’s entry came in a measly 21st out of 26.

This was clearly upsetting to the Maltese, so he issued an apology:

This prediction caused quite a stir in Malta, with a story in the Times of Malta and over 16,000 pageviews from Malta1 on Saturday alone. Many took this as good evidence that Malta were going to do well in the contest, and some people were rather annoyed with me when they did not.

I’d like to apologise if I misled anyone. I didn’t expect anyone to take the model predictions particularly seriously, and if I had known, I would have included some more caveats and explanations of exactly what the model was predicting. Instead, I was fairly loose and jokey about the model results, and didn’t really talk about what they meant in real terms. Sorry, guys.

But will they forgive him?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• Helix Nebula and the future of Europe’s cloud
• The importance of putting the U and I in visualization