Yesterday’s notice from Google (NSDQ: GOOG) that publishers would be able to block access to more than five stories in 24 hours set off a wave of claims that the change in the First Click Free program was a deal with critic News Corp (NYSE: NWS) and/or that the search engine was giving in to publisher concerns at the expense of users. The latest from Josh Cohen, the senior business product manager for Google News who explained the move yesterday on the official blog and during an FTC appearance, will probably set off another wave: Google is launching a new web crawler that gives publishers more blocking options in Google News.
Cohen admits the new options give publishers more control but reminds that the ability to block content isn’t new. Using the famous Robots Exclusion Protocol — trotted out whenever someone at Google or elsewhere wants to remind publishers that no one forces them to make their content available — already does that. But until now, applying exclusions to Google News took filling out a contact form. Now publishers will be able to do it themselves with the same abilities they have in Google:
“Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don’t even have to contact us – they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today. In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They’ll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP.”
That’s not giving in to News Corp. That’s saying, go ahead, find another excuse. Or as Cohen puts it: “We respect publishers’ wishes. If publishers don’t want their websites to appear in web search results or in Google News, we want to give them easy ways to remove it.”
As for the First Click Free changes, yes, it will be harder for people to game the system. But five stories a day for free ought to be enough for most casual readers. Anyone who needs more might consider supporting the publication.