(c) 2012 Pinar Ozger. pinar@pinarozger.com

That’s why Huls and his team worked with Kaggle, a marketplace for data science competitions. Huls told the audience that the cooperation with Kaggle was the result of an interesting discovery: Some of Allstate’s own engineers participated in Kaggle competitions in their spare time. “My first reaction was: Clearly, we are not giving these people enough work,” he joked.

But then he realized that outside expertise could help Allstate build better predictions, which can directly impact Allstate’s bottom line. “How well you predict risk is the difference between being a successful and an unsuccessful insurance company,” explained Kaggle’s president and chief scientist Jeremy Howard.

However, there are also some risks in outsourcing this kind of data science. One of the biggest ones: Data scientists are smart folks, and they might figure out some things that you didn’t really want them to know, or include in their models. That can lead to data leakage and privacy issues, as Allstate experienced first-hand. The company’s data spanned across a number of years, and participants were able to figure out which customers had been driving which cars over a period of time. If you’re an insurance company, you might want to contain that data, admitted Huls: “In our competition, we didn’t do it very well, quite honestly.”

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Infrastructure Q4: Big data gets bigger and SaaS startups shine