The online future is personal, and that requires big data


Aaron Kimbell of WibiData at Structure:Data 2012

(c) 2012 Pinar Ozger.


The problem for many companies is that information about their users is spread across hundreds or even thousands of different fields in various databases, and it’s difficult to put it all together in real time. But doing that successfully is becoming more and more important as services become more personalized.

Kimball, whose company has been funded by Google chairman Eric Schmidt and Ron Conway’s SV Angel group, said that WiBiData’s solution is to put together a platform that combines big-data aggregation and analysis tools like Hadoop and Hbase and Apache’s Avro, which can pull in and make sense of data coming from multiple databases in one location, and then make it easy for companies to extract and make use of that information in real time. In addition to tools that combine those services, WiBiData also has a user interface and dashboard that allow companies to pick and choose which data to use quickly.

Online users in particular have grown used to services like Amazon and Netflix, which provide real-time recommendations based on what a user has done in the past or other elements of their user profile, Kimball said. This not only saves a user time, but exposes them to items they may not have even known they wanted to search for — and thus creates a better user experience.

So if your company is selling hardware and a user puts a hammer in their online shopping cart, you want to be able to suggest that they buy some nails as well. But this has to be done in milliseconds if it is to be useful, and that requires manipulating large amounts of data in real time to update or alter the user’s profile and pull in the new selection.

“We are now capable of collecting and analyzing orders of magnitude more information than at any other time in history,” thanks to the database tools we have, Kimball said, but “the one constrained resource is understanding.” Users of online services now expect to have things personalized in real time, and if your company can’t provide that then you are going to be behind the curve, he said.

Watch live streaming video from gigaombigdata at



You guys should stop regurgitating PR and do some research into alternatives such as Iconic OS that would solve these problems but, oops, place ownership of data back with the users and we can’t have that whilst there is money to be made from stealing it…


It seems to me that personalized data is to privacy as convenience is to security. In other words this won’t be easy and it can’t be bolted on, just ask MS.

Gizmodo in a longer piece, on what/why Google is doing this sort of thing:
The Case Against Google

Comments are closed.