Big Thoughts on Big Data: Infochimps

Infochimps is attempting to build a data market, and in doing so, the company is wading into some of the messiest and most unstructured data around, attempting to clean it up and put it up for sale. I spoke with Flip Kromer, co-founder, president and CTO of Infochimps, to get a sense of how the company has dealt with creating a market in such a nascent industry and how it handles the deluge of unstructured data.

He and I also spoke about the need for some sort of common data schema that would help identify and classify data neatly (a key element to making Infochimps scalable), how Infochimps is using Avro, and how the company is running almost 10 different data stores with the aid of Chef, the open-source configuration management code hosted by Opscode. Finally, we talked about what opportunities for startups Kromer sees in the big data realm (data visualization gets a vote). For the full story, watch the video, and make plans to come see Kromer at our Structure Big Data conference taking place on March 23 in New York City.

I’d also like to extend a thanks to AMS Pictures which let us use its studio in Austin, Texas for the shoot.

Related content from GigaOM Pro (subscription req’d):