Blog Post

Corrected: British Libraries Must Not Harvest Paywalled News Sites

» Clarification: This story has been superseded by a later story with more accurate information.


The UK government is abandoning plans that would have compelled publishers of content behind “paywalls” to make that content available for free through Britain’s main libraries.

It says the libraries, which had wanted to harvest both free and paid-for content in their archives, had not disproved some publishers’ claims that this would place onerous and expensive technical burdens on them.

Currently, the Legal Deposit Libraries Act 2003 grants the British Library, the National Libraries of Scotland and Wales, and the university libraries of Oxford, Cambridge and Trinity College, Dublin, the right to receive and store one printed copy of each printed work available in the UK.

Last September, the government, acting on advice from the Legal Deposit Advisory Panel, which advises government on the Act, proposed extending this provision to offline digital publications and online publications. The libraries would run harvesting algorithms to grab and store the content. But paid-access web systems make this more difficult.

The government had put the plans out to consultation in September.

But, in conclusion this week, it says: “In the light of the overall responses, and the lack of evidence from both libraries and publishers to support the case that the regulations do not impose a disproportionate burden, we do not believe that it is viable to go forward with the regulations as currently drafted unless we can find evidence of proportionality.

“The government is committed to delivering regulations that cover non-print content and therefore propose to develop the draft regulations to include only off-line content, and on-line content that can be obtained through a harvesting process.”

This is a victory for news publishers, who instead advocated libraries use the Newspaper Licensing Agency’s existing eClips service, from which publishers take a proportion of revenue.

Some news publishers operate their own pay-for digital archives. The Times’ dates back to 1785. After last year’s spat with James Murdoch over a related issue, the British Library already agreed to limit its scanning of printed papers.


Highlights from consultation responses follow…

A&N Media: “It is conceivable that these costs could run in excess of £100K per annum per publisher … We think there will be problems in principle and practice in harvesting newspaper websites … This may have been a meaningful concept two to three years ago but its feasibility has been long since overtaken by commercial and technological developments … a national archive needs to be built up sector by sector so that each type of online publication can be considered on its own and a suitable system developed.”

Guardian Media Group: “A random patch work of snap shots will not “plug the digital black hole” which the British Library (BL) states threatens the nation’s digital heritage … it poses a real threat to our ability to safeguard our commercial interests. The threat arises from the BL itself.”

IPC Media: “The cost of depositing a dynamic website could easily run into thousands of pounds per annum per website … many publishers are (or will) developing their own archives which could be referenced and indexed within libraries … we are concerned about the unquantified costs.”

News International: “Without substantial modification, the regulations will give rise to unacceptable commercial and legal risk … government should replace the Act with a new Act which is fully fit for purpose … . A legal deposit archive should be a lender of last resort … The law must recognise the primacy of the publishers” interests in this regard. While librarians may view their role as being to disseminate content as widely as possible it is, in our view, primarily the publishers” role to do so, unencumbered by competition from legal deposit libraries.”