On Wednesday the company announced the general availability of its product, Sqrrl Enterprise, which is a cleaned-up and more-functional version of the open source Accumulo software. That means users will get an experience a lot more similar to what NSA data analysts get than what the core database code allows.

How do we know this? Because Sqrrl’s co-founder and CTO Adam Fuchs helped build Accumulo and the applications that run on top of it during his previous life working for the spy agency. (If you want to know more about the history of Accumulo and the types of massive graph analyses the NSA is using it for, you can check out my coverage of the NSA citizen-spying scandal from two weeks ago (here and here).) So, instead of just downloading an open-source take on Google’s BigTable data store, Sqrrl users get things like built-in analytic functions and search; support for JSON data structures; and data encryption both at rest and in motion.

The Sqrrl architecture

It’s the latter features around security that Sqrrl Co-founder and VP of Business Development Ely Kahn said have many early Sqrrl users most excited. Health care companies, in particular, highlight an ideal use case for security features like those that Sqrrl provides. Because of its cell-level security and access control, Kahn explained, providers can try to do new things around data sharing while still keeping compliant with regulations such as HIPAA and the data requirements that come along with the Affordable Care Act.

But the applications of Accumulo and Sqrrl could be much broader across industries. Because it’s based on Hadoop, Sqrrl gives companies peace of mind when it comes to storing big data securely, Kahn said, which has been a big reason that many companies are afraid to do Hadoop in production. And Sqrrl’s analytic capabilities make it easier to analyze all that data, including log files and network data that could help a company track down the causes of any cyberattacks they might suffer.

At this point, said Kahn, who was previously director of cybersecurity strategy at the National Security Staff in the White House, that should be a major concern. For most organizations, he said, it’s not a question of whether they’ve been breached but “a question of whether they know that they know they’ve been breached.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud security market landscape, 2013–2017
• How data warehousing is now a cost-effective solution for businesses
• The importance of putting the U and I in visualization

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Forecasting the future cloud computing market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Snowden said that there wasn’t one specific moment where he decided that he was going to reveal top-secret information (if you’re just coming to this story, check out our omnibus post about what we know so far, which is being updated regularly). He said it just built up over time as he watched the agency collect more information via phone calls, emails, credit-card transactions, etc. He said he thought President Obama might change what was happening, but decided to leak the documents after the president “continued the policies of his predecessors.”

“The NSA has built an infrastructure that allows it to intercept almost everything. With this capability, the vast majority of human communications are automatically ingested without targeting. If I wanted to see your emails or your wife’s phone, all I have to do is use intercepts. I can get your emails, passwords, phone records, credit cards.”

NSA “routinely lies” to Congress

Snowden, who has since fled to Hong Kong and says even his family doesn’t know that he is involved in the leak, told the Guardian that he decided the NSA had overstepped its bounds and on top of that the agency “routinely lies” to Congress about the scope of its activities.

“I don’t want to live in a society that does these sort of things… I do not want to live in a world where everything I do and say is recorded. That is not something I am willing to support or live under… I can’t in good conscience allow the US government to destroy privacy, internet freedom and basic liberties for people around the world with this massive surveillance machine they’re secretly building.”

The former CIA technical assistant, who worked for contractor Booz Allen, said that the ability the National Security Agency has to pull in personal information about American citizens, track their location and even bug their computers is more far-reaching than many people know.

“You are not even aware of what is possible. The extent of their capabilities is horrifying. We can plant bugs in machines. Once you go on the network, I can identify your machine. You will never be safe whatever protections you put in place.”

Since the original story broke on Friday, there has been much debate about how the PRISM program works, and whether it allows the NSA to have “direct access” to the servers of companies such as Google, Yahoo and Facebook, as alleged in the slide presentation leaked by Snowden. The CEOs of those companies have denied any knowledge of such activity, but sources have told the Guardian, the Post and the New York Times that the NSA did in fact have direct access to their systems.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How intelligent networks address enterprise cloud issues
• Cloud and data first-quarter 2013: analysis and outlook
• How the mega data center is changing the hardware and data center markets

This story is moving so quickly that it is hard to keep a handle on all of the developments, not to mention trying to follow the denials and non-denials from those who are allegedly involved, and the threads that tie this particular story to the long and sordid history of the U.S. government’s surveillance of its own citizens. So we thought it would be useful to try and collect what we know so far in a single post, which will be updated as often as possible with new information.

The Guardian leak

Guardian blogger and former lawyer Glenn Greenwald reports that the NSA has gotten a secret order from the Foreign Intelligence Surveillance Court that allows it to collect data about phone calls made by “millions of customers” on the Verizon network: location data, time and other identifying info about the call — everything except the actual content of the calls themselves (the Guardian has a background piece about what kind of metadata is available with such an order).

“The National Security Agency is currently collecting the telephone records of millions of US customers of Verizon, one of America’s largest telecoms providers, under a top secret court order issued in April. The order… requires Verizon on an ‘ongoing, daily basis’ to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.”

The leak widens

Other stories that follow the Guardian report quote anonymous sources saying the Verizon court order is a renewal of an order that has been in place for some time, and add that other telecom companies such as AT&T are also involved in similar programs. Greenwald notes in his story that the NSA started a program of bulk collection of telephone, internet and email records in 2001 under President Bush and this later caused controversy when it was reported in 2006 that the NSA had been saving all of this information and was analyzing it to try and detect terrorism.

Information-security experts and other industry watchers note after Greenwald’s story is published that the NSA and other government agencies have had these kinds of abilities for years thanks to laws such as the Protect America Act and the FISA Amendments Act. ProPublica has a roundup of what the government can find out about you and your behavior without a search warrant, and security expert Bruce Schneier says that what we don’t know about the government’s surveillance programs is even more frightening than what we do know.

Meanwhile, our Stacey Higginbotham wonders whether the NSA story will be a wakeup call about the power of big data, while Derrick Harris looks at how the security agency and other government entities analyze the vast amounts of information that come from such programs.

Freelance journalist Joshua Foust argues that the NSA revelations won’t cause most people to change their behavior — including their habit of voting for politicians who enact the kind of legislation that permits such surveillance — because they simply don’t care enough about the issue. Some experts said the kind of data the NSA is getting can be very powerful when it comes to finding patterns of behavior, but research from the Cato Institute says that even mining large amounts of data can turn out to be not that helpful when it comes to catching terrorists.

The Wall Street Journal, meanwhile, said that the NSA’s surveillance program was “legal and necessary” and the furor over the disclosure of this program was misplaced:

“Nobody’s civil liberties are violated by tech companies or banks that constantly run the same kinds of data analysis. We bow to no one in our desire to limit government power, but data-mining is less intrusive on individuals than routine airport security. The data sweep is worth it if it prevents terror attacks that would lead politicians to endorse far greater harm to civil liberties.”

The Washington Post leak

Within hours of the Guardian story appearing, the Washington Post reports that it has been leaked an internal slide presentation from the NSA that describes a program it calls PRISM — which involves the collection of email and other personal data from internet companies including Google, Microsoft, Facebook, Apple and Yahoo. According to the Post report (and a subsequent Guardian report based on a similar leak), this program has been underway since at least 2007, and involves what one NSA slide refers to as “data collected directly from the servers” of the companies named.

All of the companies who are reportedly involved in PRISM (which refers to them as “partners”) deny any knowledge of such a program, and say they only provide data when forced to do so by court order, and that they have no “back door” systems that would allow the NSA to do what it claims to be doing. These denials are met by widespread skepticism, and many observers — including TechCrunch founder turned VC Michael Arrington — wonder why insiders working at the tech giants allegedly involved in the program wouldn’t have leaked the information earlier.

The ongoing fallout

Some tech-industry observers say the denials from internet companies may be true, because they aren’t convinced the companies in question would even have to know about the NSA’s collection practices in order for them to work. The original Washington Post story is updated early Friday to note that it’s not clear whether “direct access” to the servers of those companies would be required, and quotes from another leaked document that says the program allows NSA officers to send “content tasking instructions directly to equipment installed at company-controlled locations,” which could mean boxes installed at ISP switches.

Several sources note that former AT&T employee Mark Klein revealed in 2007 that he had come across documents that showed the telecom company installed equipment — using glass prisms as “splitters” — that allowed the NSA to make a copy of the data stream coming from the AT&T network and send it to data-storage centers operated by the security agency. This was alleged to be part of a larger program that stored telephone calls, emails and other internet activity for the government and had been underway for years.

Some network analysts speculate that the NSA may be making use of equipment installed at CDNs (content delivery networks), which handle much of the data traffic for companies like Google and Yahoo. Laws passed in the U.S. require equipment makers such as Cisco to build into their products a way for law enforcement officials to tap into the streams they carry, and the NSA could be searching those streams directly instead of copying or storing all the data itself (since the cost of the program is a relatively cheap-sounding $20 million, according to the Post leak).

In a statement about the leaks, the Office of the Director of National Intelligence said that it does its best to work “within the constraints of the law” to collect information related to national security, and that unauthorized leaks such as those to the Guardian and Post “threatens potentially long-lasting and irreversible harm to our ability to identify and respond to the many threats facing our nation.”

Trying to track down PRISM

A search for entities that might be involved in the NSA program turns up software from a relatively secretive startup called Palantir — which has been funded by the CIA through its investment arm — that happens to be named PRISM. According to descriptions of the software, it allows clients of Palantir to sift through massive amounts of data and find patterns quickly.

Others are skeptical, however, that the software described could be used to do what the NSA appears to be doing, and security-industry sources say the NSA usually builds its own products and doesn’t like to use those from third parties. On Friday afternoon, Palantir told The Verge: “Palantir’s Prism platform is completely unrelated to any US government program of the same name.”

Former Reuters social-media editor Matthew Keys said on Twitter that he had found several references to the PRISM program in classified job listings dating back to 2007:

Not wanting to be left out, the secretive activist group Anonymous released some classified documents that refer to Defense Department information technology — but they appear to be mostly jargon-filled descriptions of the department’s IT infrastructure, with little or no connection to PRISM or any NSA-related data collection practices.

The ripples spread outside the U.S.

As our man in Europe — David Meyer — noted in a couple of posts Friday morning, the repercussions from the PRISM and NSA revelations are being felt in Europe as well, with some critics calling for changes to the so-called “Safe Harbor” program, which allows data about EU citizens to be stored by non-EU companies. And the Guardian has reported that the U.K. government appears to have been getting information via the PRISM program, which was designed to focus on the communication activity of non-U.S. residents (since U.S. law still technically prevents the government from spying on its own citizens without a warrant).

Meanwhile, President Obama — whom many critics have accused of carrying on with surveillance programs started by his Republican predecessor, despite his disavowal of such methods while campaigning — said through a spokesman that he “welcomes the discussion” about privacy and security:

“The president welcomes the discussion of the trade-off between security and civil liberties. The close examination of some of these complicated issues could cause people to arrive at differing opinions… The president welcomes that debate.”

Late Friday, the Guardian posted another security-related scoop, publishing what it called a “secret presidential directive” that orders the U.S. government’s top national security and intelligence officials to draw up a list of potential overseas targets that the U.S. could hit with cyber-attacks. The story goes on to say this operation:

“can offer unique and unconventional capabilities to advance US national objectives around the world with little or no warning to the adversary or target and with potential effects ranging from subtle to severely damaging”

Google denial and Sir Tim Berners-Lee

The creator of the world wide web, Sir Tim Berners-Lee, posted a statement at the Web Foundation blog saying:

“Today’s revelations are deeply concerning. Unwarranted government surveillance is an intrusion on basic human rights that threatens the very foundations of a democratic society. I call on all Web users to demand better legal protection and due process safeguards for the privacy of their online communications, including their right to be informed when someone requests or stores their data.”

And Google co-founder Larry Page posted a response Friday afternoon to the accusations in the Guardian and Post stories, written with Chief Legal Officer David Drummond, saying the company does not provide the government with “back door” access to its servers, and had never heard of the PRISM program until Thursday:

“Press reports that suggest that Google is providing open-ended access to our users’ data are false, period… Any suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”

Zuckerberg denial

Facebook co-founder and CEO Mark Zuckerberg posted a statement about PRISM on his Facebook page late Friday, saying he wanted to respond personally to the “outrageous press reports” about his company’s involvement in the surveillance scheme. In language very similar to the Google denial, Zuckerberg said the network has not been part of any program to give the U.S. government “direct access” to its servers.

“Facebook is not and has never been part of any program to give the US or any other government direct access to our servers. We have never received a blanket request or court order from any government agency asking for information or metadata in bulk, like the one Verizon reportedly received. And if we did, we would fight it aggressively. We hadn’t even heard of PRISM before yesterday.”

Does the NSA even need a back door?

Christopher Mims at the Atlantic business site Quartz quotes NSA veteran and whistle-blower William Binney — who was part of a group that asked the Defense Department to investigate the NSA in 2002 — saying the security agency could probably get its hands on about 80 percent of the web traffic that passes through the U.S. without even having direct access to the servers of companies like Google. That’s because the NSA has access to at least one of the largest communications hubs on the continent, as described by the Electronic Frontier Foundation.

The Wall Street Journal posted a story that quoted unnamed security experts who said the tech companies mentioned in the PRISM presentation could be telling the truth about not providing “direct access” to their servers, but still have their data collected by the NSA. The Journal said U.S. officials told the paper that the NSA “receives copies of the data through a system they set up with a court order.”

“One industry executive familiar with the handling of data requests from U.S. intelligence agencies said companies have set up ways to cope with the volume of data by automating parts of the process. This method would allow data to be funneled to intelligence agencies without the need for manual steps by company employees.”

At The Daily Beast, writer Megan McArdle looked at the issue of whether tech company denials should be believed or not, and quoted privacy expert Julian Sanchez from the Cato Institute saying there are a number of ways that the NSA could get the data it wants without requiring direct access, including the “secret room” with splitter equipment that Mark Klein described at AT&T (mentioned above):

“Most likely… is that they’ve got something akin to the “Secret Room” that Mark Klein disclosed in AT&T hubs where traffic is being cloned (the companies would need to provide the relevant SSL encryption keys) split off into NSA’s own machines. It would be literally true, in that case, that the NSA does not have direct access to Google’s servers.”

How PRISM might work in practice

Late Friday, the New York Times posted a story that said some tech companies resisted the NSA’s demands to provide easier ways to get access to user data — including Twitter — but that some consented, opened up discussions with the security agency about developing methods to share that data, and even “changed their computer systems to do so.”

“In at least two cases, at Google and Facebook, one of the plans discussed was to build separate, secure portals, like a digital version of the secure physical rooms that have long existed for classified information, in some instances on company servers. Through these online rooms, the government would request data, companies would deposit it and the government would retrieve it.”

In other words, “companies were essentially asked to erect a locked mailbox and give the government the key” and Facebook actually built such a system, the NYT story said. Declan McCullagh at CNET explained in a post that according to his sources, all that the PRISM process does is automate something that is required under FISA (the Foreign Intelligence Surveillance Act) — so court orders are given to the tech companies and they have simply made the process of handing over that information easier.

Marc Ambinder, a security expert who writes for The Week, also described his understanding of how PRISM functions — in a nutshell, PRISM is just a piece of software that allows the NSA to collect and interpret data that is handed over under FISA. The actual software itself isn’t classified, which is why mentions of it show up online and in job postings. In McCullagh’s piece, a former NSA lawyer says that the slide presentation the Washington Post published is “suffused with a kind of hype that makes it sound more like a marketing pitch than a briefing.”

Meanwhile, for those trying to keep track at home, the Electronic Frontier Foundation has put together a comprehensive timeline of events related to NSA surveillance activity over the past decade:

It was for your own good

First tech companies claimed they didn’t know anything about PRISM and weren’t supplying data (or at least not direct access), and now the story some sources close to those companies are telling is that they set up portals or some other method of complying with FISA requests in order to “protect the innocent,” according to a post at TechCrunch.

“The NSA may have wanted full firehoses of data from Google, Facebook and other tech giants, but the companies attempted to protect innocent users from monitoring via compliance systems that created segregated data before securely handing it over as required by law.”

The Guardian has responded to criticisms of its original description of PRISM and the whole notion of “direct access” — as well as the repeated denials from Google executives and others that this has been taking place — by posting another slide from the leaked NSA presentation. While some have speculated (as mentioned above) that PRISM could mean simply sucking data from ISP equipment, the NSA slide contrasts this method of getting data with PRISM’s, which it describes again as “collection directly from the servers” of the companies mentioned.

The Director of National Intelligence released another statement on Saturday, calling the disclosures by the Guardian and Washington Post about NSA data collection “reckless” and filled with “significant misimpressions.” So DNI James Clapper said he had declassified some details about the program, published in a fact sheet (PDF link). Among other things, it says:

“PRISM is not an undisclosed collection or data mining program. It is an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act.”

The Washington Post published a follow-up story on Saturday that described the PRISM process in much the same way as earlier stories from the Guardian and the New York Times: as a system or software that allowed the NSA to process FISA requests for information more quickly — and the paper reiterated earlier statements that because the program was top secret, only a few individuals within those companies would even know about it, let alone be able to discuss it. According to the Post:

“Executives at some of the participating companies, who spoke on the condition of anonymity, acknowledged the system’s existence and said it was used to share information about foreign customers with the NSA and other parts of the nation’s intelligence community.”

Much of the criticism about the original Post story and the Guardian story has focused on the description of PRISM as allowing “direct access” to the servers of companies like Google, Facebook and Yahoo — something the leaders of those companies have strenuously denied providing. The most recent Post story suggests that at least some of the debate over this term is semantic, and that its sources say PRISM did allow the NSA to get data from those companies directly:

“Intelligence community sources said that this description, although inaccurate from a technical perspective, matches the experience of analysts at the NSA. From their workstations anywhere in the world, government employees cleared for PRISM access may ‘task’ the system and receive results from an Internet company without further interaction with the company’s staff.”

NSA whistle-blower reveals his identity

In another bombshell, the Guardian revealed the identity of the whistle-blower who sent them the leaked documents about PRISM and the NSA surveillance program: he is Edward Snowden, a 29-year-old former technical assistant at the Central Intelligence Agency, and he is now living in Hong Kong and expects he will “never see home again.” He said his family doesn’t know about his activities, and that he fully expects to be charged and potentially face jail time for his actions.

In an interview with the Guardian, Snowden says that he gradually became frustrated with what the NSA was doing and believed it was wrong — but originally held off on leaking anything because he thought Barack Obama would change those policies when he was elected president. But Snowden says the president continued with “the policies of his predecessor” and so he decided to come forward and let the American public know what was happening behind closed doors:

“I don’t want to live in a society that does these sort of things … I do not want to live in a world where everything I do and say is recorded. That is not something I am willing to support or live under.”

Snowden also said the documents he leaked clearly show that “the NSA routinely lies in response to Congressional inquiries about the scope of surveillance in America” and that the abilities that he had as a contractor with the CIA were beyond what most people can even imagine:

“You are not even aware of what is possible. The extent of their capabilities is horrifying. We can plant bugs in machines. Once you go on the network, I can identify your machine. You will never be safe whatever protections you put in place.”

The reaction

In a post written for The Atlantic magazine, James Fallows said that the most frightening and important part about PRISM and the rest of the NSA surveillance activity revealed by Snowden is that it is all legal under the Foreign Intelligence Surveillance Act and other legislation.

“That these programs are legal — unlike the Nixon “Plumbers” operation, unlike various CIA assassination programs, unlike other objects of whistle-blower revelations over the years — is the most important fact about them. They’re being carried out in “our” name, ours as Americans, even though most of us have had no idea of what they entailed.”

Fallows — and others such as Talking Points Memo founder Josh Marshall — raised some question marks about the wisdom of Snowden’s choice of Hong Kong, which is still part of China and therefore not particularly open to harboring whistle-blowers. However, according to some experts in the law, Hong Kong might be a good place to seek asylum because of a loophole that could allow Snowden to remain there indefinitely.

Icelandic MP Birgitta Jonsdottir, an early supporter of WikiLeaks and of freedom-of-information laws in general, told Forbes magazine that she plans to try and get her country to offer Snowden political asylum. But observers of the political scene in Iceland say this might be more difficult than it would have been in the past, since the new Conservative government is seen as more friendly to the Obama administration.

Daniel Ellsberg — the man who leaked the famous “Pentagon Papers” in 1971 and revealed that the government had been lying about the Vietnam War — said in a piece written for the Guardian that Snowden’s leaks give the United States a chance to “roll back what is tantamount to an executive coup against the U.S. constitution.” Ellsberg said that Snowden’s revelations were the most important leak in the history of the United States, including his own.

“Since 9/11, there has been, at first secretly but increasingly openly, a revocation of the bill of rights for which this country fought over 200 years ago. In particular, the fourth and fifth amendments of the US constitution, which safeguard citizens from unwarranted intrusion by the government into their private lives, have been virtually suspended.”

Meanwhile, David Kirkpatrick — author of the book “The Facebook Effect” — asked whether the secrecy and privacy invasions involved in the PRISM program might impair the growth of social networks and cloud services like Facebook.

“Do we really want to impair such powerful tools for spreading dialogue, political discourse, and U.S. values? Is it worthwhile to impair the extraordinary financial and commercial success of these great flagships for the American economy? Does Obama want Facebook et al just to be seen as tools of American power?”

Politico took a look at some of the things that we still don’t know about PRISM and the activity involved in the NSA’s surveillance program — including how much data the spy agency has been collecting from phone companies as well as tech companies like Google, whether this data collection has actually thwarted any specific terrorist attempts or not (something that is the subject of much debate) and how exactly the PRISM program works in practice.

Meanwhile, the Daily Beast has a piece that looks at the group within the U.S. intelligence apparatus that hunt down leakers like Snowden, a kind of internal police force called the Associate Directorate for Security and Counterintelligence — or the Q Group for short. And Salon magazine has a feature and interview with Laura Poitras, the documentary film-maker who was contacted by Snowden and later helped both the Post and the Guardian write their stories about the leak.

Got anything I am missing? Let me know at mathew@gigaom.com

Post and thumbnail images courtesy of Shutterstock / Lightspring and the Washington Post

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution
• How the mega data center is changing the hardware and data center markets

Eleven Paths is staffed with the former employees of respected Spanish security outfit Informática 64 and led by that company’s CEO, Chema Alonso. The idea is to have the new company operate like a startup under Telefónica’s umbrella — the telco calls this a “hothouse approach” — in order to speed up development of new security products around the cloud and, of course, mobile devices.

“Our customers are supposed to be big companies and individuals,” Alonso told me. “At Telefónica Digital we’re thinking all countries in the world are our customers. All our security products are going to be focusing worldwide.”

Unsurprisingly given Informática 64′s development of the FOCA tool, one of the first Eleven Paths products will be a penetration testing (pentest) tool, for simulating attacks on companies’ systems in order to find vulnerabilities.

“Right now, companies are doing tests every three months,” Alonso said. “We need a new way of thinking about how penetration tests must be done. We think information systems must be built to be pentested by design.”

On the mobile side, Alonso said, Eleven Paths is working on making security products easier to understand and use in the world of small form factors and bring-your-own-device concerns. “It’s about the speed of the bad guys creating security threats,” he said. “We want to modify the way in which people think about security.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Bluetooth to Feel Blue as Personal Area Network Battles Loom
• Mobile 2011: Data Consumption Will Explode
• HTML5’s a Game-Changer for Web Apps

The original report on the NSA’s surveillance effort came from Glenn Greenwald, who writes about politics for The Guardian, courtesy of a leaked document that confirmed the existence of an order signed by the ultra-secret Foreign Intelligence Surveillance Court. As the New York Times explains, even the existence of this kind of order is subject to the highest levels of U.S. government secrecy — much higher, in fact, than the diplomatic cables that former Army private Bradley Manning is accused of providing to WikiLeaks.

Update: Both the Guardian and the Washington Post are reporting that the federal government has also been getting personal data from a number of large internet players for years — including Google, Facebook and Apple.

The real story isn't just the spying itself: it's that we have this massive, ubiquitous Surillveance State, operating in total secrecy—
Glenn Greenwald (@ggreenwald) June 06, 2013

The government is cracking down on leaks

Even without the current Manning trial as a warning, the risks of a leak like the NSA court order would be abundantly obvious by now, given the Obama administration’s ongoing campaign against government leaks — a campaign that some argue has gone too far, with reporters being named as “co-conspirators” by the authorities (a charge that has echoes of the “aiding the enemy” accusations against Manning). Among other things, the government recently seized the phone records of Associated Press reporters.

This kind of cloak-and-dagger activity towards journalists — which some observers (including me, and former New York Times executive editor Bill Keller) have been warning might occur in the wake of the government’s pursuit of Manning and Assange — has likely thrown a substantial chill over the U.S. media when it comes to exposing government information. As if to reinforce that, within minutes of the Guardian story appearing online, sources said the Department of Justice would likely be investigating the leak and how it arrived in Greenwald’s inbox.

There have always been leaks, of course, and there would no doubt continue to be leaks even if WikiLeaks didn’t exist. The legendary Watergate investigation and the release of the famous Pentagon Papers both happened without WikiLeaks, or even the internet. But there’s also no question that having a repository for such documents that is both anonymous (or as close as it is possible to get) and largely stateless would make it easier for such leaks to occur.

Daniel Ellsberg, the former Defence Department official who leaked the Pentagon Papers in 1971 — and later became part of one of the most ground-breaking First Amendment trials in history — has said that Manning and WikiLeaks are carrying on the same tradition he was a part of: namely, the quest to hold the government accountable for its actions. Since the media seem reluctant to play the role they should in this effort, Ellsberg says, WikiLeaks becomes even more necessary.

WikiLeaks continues to struggle to stay alive

Meanwhile, WikiLeaks itself is struggling — in part because of Assange’s legal issues, as well as a lack of funding that was exacerbated when PayPal, Visa and MasterCard cut off the ability to donate to the organization, despite the fact that WikiLeaks hasn’t been accused of a crime. And viable alternatives have not yet emerged (a splinter group headed by a former WikiLeaks lieutenant tried to set up a competitor called OpenLeaks without much success, and the New Yorker recently launched its own effort called StrongBox).

Greenwald suggested in a comment on Twitter that the Obama administration’s behavior towards government leakers may have actually encouraged sources like his to become even bolder, as a way of defying what they see as an unreasonable attack on freedom of speech and whistle-blowing. But if WikiLeaks were stronger, sources would have another place they could go to reveal important information — and one that would be more difficult to attack.

To that end, a group called Freedom of the Press was formed earlier this year by Ellsberg and a number of other free-speech advocates, including BoingBoing’s Xeni Jardin and actor John Cusack, which is designed to help support WikiLeaks and a number of other entities via crowdfunding. Freedom of the Press has also crowdfunded a number of stenographers to take notes about the Manning trial, since very few media organizations were allowed to attend and most have had restrictions placed on what they can and can’t report.

Whether WikiLeaks can survive or prosper given the splintering of the organization (which appears to have been caused at least in part by Assange and his mercurial approach to running WikiLeaks) — remains to be seen. But having some kind of entity that performs the same function is a clear public good.

Post and thumbnail photos courtesy of Shutterstock / Mmaxer and Flickr user Carolina Georgatu

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• SOPA, OPEN and the fight for the Internet
• Connected world: the consumer technology revolution
• Bluetooth to Feel Blue as Personal Area Network Battles Loom

This is the latest episode in the long-running and global “Can we trust Huawei?” game. No-one has ever caught out the Chinese firm using its equipment to spy on people, nor has anyone even made such definitive allegations, but Huawei’s opaque relationship with the Chinese state has already led both the U.S. and Australia to ban the equipment from those countries’ broadband networks.

The U.K. doesn’t have that option – it’s now a decade since BT selected Huawei to supply kit for its 21st Century Network program (the upgrading of the country’s core networks), and today Huawei equipment can be found in the infrastructure of BT, O2, TalkTalk and EE. It’s not going anywhere anytime soon.

Second thoughts

That makes this a game of mitigating risk. To that end, in 2010 Huawei agreed to set up an equipment testing facility, known not at all ominously as the Cell, to reassure the U.K.’s spooks that nothing untoward was being snuck into the country’s critical national infrastructure.

So how’s that working out?

“Before seeking clarification, we assumed that Huawei funded the Cell but that it was run by GCHQ [the U.K.'s spy services],” the Intelligence and Security Committee wrote in its report (PDF warning), issued on Thursday. Nuh-uh – once they started digging, they realized that the Cell is staffed by Huawei people. Security-cleared Huawei people, but nonetheless not GCHQ, unless you count the director, who used to be at GCHQ but is now on Huawei’s payroll.

GCHQ has apparently argued that it’s better for the Cell to be staffed by Huawei people. The parliamentary committee did “not find this argument to be compelling”, it understated, pleading with the security services to at least get involved in staff selection.

And how’s this for reassuring?

“The Committee has been told that use of the Cell is voluntary: of the five [communications service providers] that use Huawei products, only three make use of the Cell’s facilities. The Government has told us that use of the Cell is encouraged only where a company is using Huawei equipment to provide services to the Government or as part of the [critical national infrastructure], or operating at a scale that could have a significant impact. Nevertheless, we question its assessment that the two major broadband providers which do not use the Cell are not operating at a scale where using its services would provide extra mitigation.”

Everything’s fine

In response to all this, the British ambassador to China, Sebastian Wood, issued a statement highlighting how Huawei is a “long-term valued investor in the U.K. with a business that is growing and creating jobs in Britain.”

“Our work with Huawei and their U.K. customers gives us confidence that the networks in the U.K. that use Huawei equipment are safe and secure,” he added.

Then the Chancellor of the Exchequer, George Osborne, chipped in with this: “I am pleased that next week Huawei is opening a flagship office in Reading as part of its plan to invest £1.3bn into its UK business over the next five years, generating a further 700 jobs.”

Nice to know the government is confident, but what the parliamentary committee found amounts to this: Huawei’s much-vaunted testing facility turns out to be a self-regulating Huawei affair, and one that still isn’t fully operational at that. What’s more, it only started to be set up several years after Huawei began supplying equipment for Britain’s critical infrastructure. The committee also slammed the civil servants of a decade ago for not alerting ministers when Huawei was being proposed as a major BT supplier.

This isn’t to say the U.K.’s communications nervous system is riddled with security-busting backdoors. But the oversight to stop this from being the case appears to have been lacking, to say the least. And there’s not a lot anyone can do about it now.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How intelligent networks address enterprise cloud issues
• CES 2012: a recap and analysis
• Facebook’s tactical retreat on privacy

Some day our bodies — or at least the clothing or accessories that adorn them — could become key network nodes in the internet of things. European researchers think that sensors and transmitters on our bodies can be used to form cooperative ad hoc networks that could be used for group indoor navigation, crowd-motion capture, health monitoring on a massive scale and especially collaborative communications. Last week, French institute CEA-Leti and three French universities have launched the Cormoran project, which aims to explore the use of such cooperative interpersonal networks.

The concept of wireless body area networks (WBANs) isn’t a new one. WBANs could be used to sever the cord between patients and their monitoring equipment. Companies like Apple and Heapslylon are exploring the possibility of connected clothes with embedded sensors. We’ve already begun embracing a new era of wearables, such as Google Glass to Fitbit (see disclosure), designed to become extensions of our senses and movements.

All of these devices will become key end-points in the internet of things, but what Cormoran proposes to make them pull double duty. Rather than just remain terminuses, they could route bits to and relay data from each other, becoming a distributed ad hoc network that constantly morphs as we move through physical space.

Better living through distributed networking

Why would you want this kind of network? For one, there is an inherent inefficiency in the point-to-multipoint transmissions that dominate mobile data communications today. Wearable tech usually connects via Bluetooth to a smartphone, which then transmits its info to some distant cell tower. Many medical and connected home devices use proprietary technologies requiring their own dedicated wireless gateways.

Assuming your device can even get a connection to the internet, it’s often using an expensive, power-hungry and highly suboptimal means to transmit tiny specs of data. A distributed wireless network, however, could aggregate data from hundreds if not thousands of nearby devices and then find the most efficient link to offload that collective data to the internet at large. This kind of collaboration is the same principle proposed by mesh-networking outfits Open Garden and the Open Technology Institute as a means of optimizing wireless systems – if everyone shares their connections and relays each others’ data, then everyone benefits.

But there’s an additional benefit to this kind of collaborative communication: by linking to one another, body area networks could create new useful data about users’ surroundings and location. By measuring the signal strength of nearby connections, the network could determine the precise location of every node, or person, within it.

You can imagine some of the possible applications for such technology. In a busy airport or train station, proximal location-based services could route departing passengers en masse to their proper gates or trains or arriving passengers to the proper baggage claim. City planners could use the technology to track and manage the flow of pedestrian traffic, and emergency agencies could use it to coordinate the evacuation of a building. Sociologists could use it to study group behavior, and game designers and movie CGI could use it to digital map crowd movement.

The big “what if?”

On the flip side, though, creating such collaborative networks has ominous security implications. Our own notions of individual privacy suffer if we know every transmitter in a hundred-foot radius is talking to our devices and even helping to carry our personal data back to the cloud.

There are a lot of similarities between collaborative body area networks and the vehicle-to-vehicle connected car technologies pursued by the automotive. If all cars on the highway could talk to one another, they could coordinate their activities, preventing accidents and getting drivers to their destinations faster. But the danger is that these networks would get hacked. Personal information about a car’s driver could get in malicious hands, or data intended to prevent accidents could be falsified to actually cause them.

It’s an exiting project, but Cormoran is going to have deal with similar questions. It will have to not only create the protocols that will allow our body area networks to coordinate, but ensure that the data they relay remains secure and most of the information they share remains anonymous.

Disclosure: Fitbit is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

Networking diagram courtesy of Cormoran. Privacy image courtesy of Shutterstock user Johan Swanepoel.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Analyzing the wearable computing market
• Opportunities and challenges for mobile deals

On the company’s blog, CrackPassword, Katalov writes of how he and his team were able to access a user’s backups (including photos) and documents, and were able to restore an iCloud backup onto a new Apple device without being asked for the second mode of security, the four-digit passcode, even with two-factor authentication turned on:

In its current implementation, Apple’s two-factor authentication does not prevent anyone from restoring an iOS backup onto a new (not trusted) device. In addition, and this is much more of an issue, Apple’s implementation does not apply to iCloud backups, allowing anyone and everyone knowing the user’s Apple ID and password to download and access information stored in the iCloud.

The Elcomsoft team used their own Phone Password Breaker software to sign into the targeted user’s iCloud account with the Apple ID and password. Then, to look at that data, they say they just used software that can browse and analyze offline iTunes backups.

They were then able to restore an entire backup of the user’s device and iCloud data to a new iPhone without ever being asked for secondary security information — again, even though they say two-factor authentication was turned on.

The one way the unsuspecting user whose account is being targeted would know this was happening is via an automatically generated email from Apple letting them know that their Apple ID was used to sign onto a new device.

An Apple spokeswoman declined to comment on the article’s findings.

Obviously this is concerning for Apple users who assumed far more security from Apple’s recently introduced system. But the weaknesses, as Katalov points out, tend to come at the expense of convenience. Why aren’t you asked for your passcode when setting up a brand new device? Presumably so the purchase of new phones or replacement devices at Apple Stores can happen a faster and with fewer hiccups.

He points out that Apple isn’t promising more than it’s delivering, but concludes the company has much further to go to offer real protection for users from targeted hacking.

Updated at 11:52 a.m. PT with response from Apple.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• Analyzing the wearable computing market
• The evolution of consumer-media cloud storage

The feature, known as “two-factor” authentication, is already used by companies like Google and Apple and works by sending a pin code via text message to a user’s cell phone. Twitter has details and a tutorial video here.

The decision to add an extra security feature comes after hackers have repeatedly gained control of high profile Twitter feeds. The most prominent example occurred last month when hackers used the Associated Press’s account to say bombs had injured President Obama. The fake tweet roiled financial markets and led to calls for Twitter to improve its security features.

Attackers have also targeted CBS, the BBC and the Onion. The latter offered a candid account of how the hackers phished employees accounts and induced some of them, including a person with control over social media passwords to share log-in information.

Two factor authentication would likely have prevented those attacks because the attackers would have had to enter a password sent to the employee’s cell phone.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Survey: How apps can solve photo management
• Sector RoadMap: Social customer service in 2013