Well, well, well — maybe there’s a place for human beings in the new media landscape after all. Who would have guessed? The revelation comes courtesy of Gabe Rivera, the creator and programming genius behind a series of news aggregation sites including Memeorandum, Ballbug and Techmeme. He has just admitted that the algorithms he uses to find the latest hot news haven’t been working all that well, and that as a result, he has added the services of a (gasp!) human being to the mix to try and improve things. That would be Megan McCarthy, a former writer for Wired.com and the gossip site Valleywag.
Rivera posted a job ad on Craigslist recently, looking for what he variously described as a “news technician,” “news analyst” or “configuring editor.” As the young Techmeme founder noted in the ad, the type of job he had in mind — the one Megan McCarthy has apparently been hired to do — had never really existed before, but “will become increasingly important in the years ahead.” Anyone who has tried to follow the news coming from hundreds or even thousands of different sources, using a combination of overstuffed RSS readers and other tools, knows that Rivera is right — there is a fire hose of data gushing 24 hours a day, and filtering through it all is becoming harder and harder.
Not that long ago, it was assumed by many that algorithms would be the answer. Aggregators like Yahoo News (s yhoo) and Google News (s goog), along with more specialized (and in some cases short-lived) RSS “meme trackers” such as Techmeme and Tailrank, were seen as the killer app for information overload — automated news readers that would be able to sort through the headlines and figure out what was important. And some of them, including Memeorandum and Techmeme, manage to do that pretty well. But they are still missing, as Rivera describes in his frank and honest assessment of the flaws in his methods, an important ingredient: the ability to tell when a story or headline just doesn’t belong.
The hiring of a human being to tweak the selection of headlines at Techmeme and possibly some of Rivera’s other sites is just another sign that algorithms aren’t the final solution to the information overload problem, and that it’s going to take software and human beings working together to make sense of the information coming out of the fire hose. Even Google, the king of the algorithm — a company whose automated PageRank methods have helped to create more than $100 billion worth of stock market value in just a few years — has started allowing people to influence its search results directly, through its recently-launched Search Wiki features. Although the choices people make only directly affect the results they see, the search giant has suggested that in the future those choices could affect a site’s overall PageRank.
Looks like there might still be a few jobs left for us human beings after the robots take over.