Comments on: Forget your fancy data science, try overkill analytics

By: Theodore Van Rooy

Theodore Van Rooy — Fri, 28 Sep 2012 16:43:28 +0000

The downside: overkill-analytics often leads to overfit-analytics. As Carter himself points out: “as long as you account for the effects of overprocessing data”.

The more computation power you throw at the problem the more likely it is that your model happens (by chance) to work well … even measures like cross fold validation will not prevent this overfitting.

Not saying that it doesn’t work as an approach (it’s my favorite approach too) but that you should at least put some thought into the problem before you throw massive computational power at it.

By: mikeklawrence

mikeklawrence — Wed, 26 Sep 2012 22:15:28 +0000

Peter Norvig, Director of Research at Google, has a similar philosophy aimed at the problem of natural language processing. His talk (and paper) “the unreasonable effectiveness of data” is worth watching (it’s on youtube). The premise is that problems where complicated models were once needed, can now be solved by simple algorithms and lots of data.

By: mikeklawrence

mikeklawrence — Wed, 26 Sep 2012 22:12:57 +0000

Peter Norvig, Director of Research at Google, has a similar philosophy (aimed at natural language processing) that’s captured in a recent paper (and talk) called “The Unreasonable Effectiveness of Data”. In a nutshell, what others have tried to achieve with complicated models can be done using simple algorithms, and reams and reams of data. The talk can be found on youtube and is worth watching.

By: Jody Wynnyk

Jody Wynnyk — Tue, 25 Sep 2012 16:43:01 +0000

Great article – one of my challenges as a data analyst is producing reports that the business ‘gets’. They often think they want to see more information, but I find the more information they have, the more useless it becomes. This streamlined approach captures everything, but draws the focus where it needs to be. I’ll be integrating this technique into my reporting!