Whatever the case may be, agreeing to turn over data to the government might have seemed like a good idea at the time, but the potential downside risks of that particular slippery slope are fairly overwhelming.

The popular response to the NSA revelations may lie somewhere between mild disinterest and outright apathy, according to surveys like the one done by the Pew Center — in part because we seem to have gotten used to the idea that tech companies are monitoring our every move. But being seen as co-operating with the spy agency is still a fairly huge risk for cloud-based services. Not only that, but co-operating in even a small way makes those companies look like easy targets for further government pressure.

Who co-operated and to what extent is unclear

At this point, the actual truth of what is involved in the NSA’s so-called PRISM program remains a rapidly shifting target. The documents first published by the Guardian and the Washington Post a week ago seemed pretty cut and dried in their description of a system that allowed the spy agency “direct access” to the servers of Google, Yahoo, Facebook and about half a dozen other companies — something the Post originally said was provided voluntarily and gave the NSA broad access to information about user behavior.

Almost immediately, however, the details started to blur: not only did those companies deny providing “direct access” to their servers, but some sources said the data was only provided under duress, because of secret court orders related to the Foreign Intelligence Surveillance Act. As the days went on, other reports quoted anonymous sources saying the whole system (the one those companies had denied any knowledge of) was just an attempt to automate the processing of those legitimate FISA requests.

One report quoted anonymous staffers at several of the companies saying they only agreed to co-operate with the NSA because they were afraid if they didn’t do so, the government would demand even more of their data and that wouldn’t be fair to users. And finally, on Friday, the New York Times reported — using some conveniently leaked documents — that Yahoo had tried to resist the NSA’s attempts to compel it to provide user data, but was ultimately unsuccessful and was ordered by the court to comply.

User trust is a precious commodity

So what we have now are a broad range of conflicting reports about who did what — including semantic debates about what the term “direct access” actually means, as well as how much access was provided voluntarily vs. how much was provided under duress. So far, the only company that seems to have emerged unscathed is Twitter, which reportedly fought the government’s attempts to enrol the company in the PRISM program and succeeded, a tale that has burnished Twitter’s claim to be the “free-speech wing of the free-speech party”).

That said, however, there seems to be little doubt that many companies co-operated with the NSA, and may have set up “lockbox” or “clean room”-style facilities for providing data — and there are even suggestions that this group could go far beyond just Google and Yahoo and Facebook, and could include hundreds of other technology providers that have co-operated to some extent with the spy agency and given the NSA details about their equipment and/or products that could help its surveillance program.

These companies may have convinced themselves that co-operation was inevitable, or that they needed to do something to help the government catch terrorists, or that by automating the legally legitimate FISA process they could save themselves a lot of trouble and expense, or some combination of all the above. But in reality, they have not only shown themselves to be weak — which will encourage the NSA to pressure them even further because they know they can win — but also fundamentally untrustworthy, and that could cause them a lot more problems with users than they ever contemplated.

Post and thumbnail photos courtesy of Shutterstock / Luis Louro, Shutterstock / Lightspring and Flickr user Thomas Leuthard

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

The Yahoo work is actually pretty important. Among the features Hadoop users have been demanding from the platform is a transition from batch-processing-only mode into something that can actually deal with data in real time. The reason for the demand is quite simple: Although being able to analyze or transform data minutes to hours after it’s generated is helpful for certain analytic tasks, it’s not too helpful if you want an application to be able to act on data as it hits the system.

A service like Twitter is a prime example of where Storm can be valuable. Twitter uses Storm to handle tweets so users’ Timelines are up to date and do things like real-time analytics and spotting emerging trends. In fact, it was Twitter that open sourced Storm in 2011 after buying Storm creator Backtype in order to get access to the technology and its developers.

Among web companies, Storm has become quite popular as a stream-processing complement to Hadoop since then. And now Yahoo has made possible a much tighter integration between the two — even to the point that Storm can borrow cycles from batch-processing nodes if it needs some extra juice. That’s a valuable feature — just last week I heard Twitter engineer Krishna Gade bemoan Storm’s auto-scaling limitations during a talk at Facebook’s Analytics @ Web Scale event.

Krishna Gade talking Storm at the Facebook event.

The Storm-on-Hadoop work is among the first of many promised improvements to come thanks to YARN, a major update to the Apache Hadoop 2.0 code that lets Hadoop clusters run multiple processing frameworks simultaneously. Twitter has been using the open source Mesos resource manager to achieve the same general capabilities, but Gade’s colleague Dmitriy Ryaboy said during the same talk that the company plans to begin using YARN for some big data workloads when it upgrades to Hadoop 2.0. He expects — probably correctly — much more community effort will go toward continuously improving its capabilities and building applications for YARN.

Facebook’s Wormhole project isn’t open source (as far as I can tell), but its lessons are still valuable (and LinkedIn has open sourced a similar technologies named Kafka and Databus). It’s what’s called a publish-subscribe system, which is essentially a concise way of saying that it manages communications between applications that publish information (e.g., updates to a database) and subscribe to the information their fellow applications are publishing. At Facebook, for example, Wormhole sends changes to Facebook’s master user database to Graph Search so that search results are as up to date as possible, or to its Hadoop environment so analytics jobs have the newest data.

Of course, like all things Facebook (its new Presto interactive query engine comes to mind), Wormhole is built to scale. Latency is in the low milliseconds and, blog post author Laurent Demailly notes

“Wormhole processes over 1 trillion messages every day (significantly more than 10 million messages every second). Like any system at Facebook’s scale, Wormhole is engineered to deal with failure of individual components, integrate with monitoring systems, perform automatic remediation, enable capacity planning, automate provisioning and adapt to sudden changes in usage pattern.”

Although they were developed within separate companies, there’s actually a tie that binds Yahoo’s Storm-in-Hadoop work and Facebook’s Wormhole. As web companies grow from their initial applications into sprawling business composed of numerous applications and services, so too do their infrastructures. To address the differing needs of their various systems at the data level, the companies have begun breaking them down by their latency requirements (i.e., real-time, near real-time and batch, however they choose to word them) and then building tools such as Storm and Wormhole to manage to flow of data between the systems.

We’ve previously explained in some detail how LinkedIn and Netflix have built their data architectures around these principles, and we’ll hear a lot more about how they and other web companies are tackling this situation at Structure next week. Among the speakers are senior engineers and technology executives from Facebook, Google, LinkedIn, Box, Netflix and Amazon.

Update: This post was updated at 1:46 p.m. to clarify that Twitter is not eliminating Mesos for all its workloads. 

Feature image courtesy of Shutterstock user agsandrew.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Altiscale is in some ways a manifestation of Stata’s seven years of experience helping turn Hadoop from a cute little project into a production system running across 42,000 nodes. It might not be not pretty, but it gets the job done. And, thanks to the handful of former senior Yahoo, Google and LinkedIn engineers that joined Stata (who’s the company’s CEO) at Altiscale, the company knows Hadoop cold.

Team Altiscale (State is middle row, second from left). Source: Altiscale

The deep knowledge of Hadoop shows itself in the product design and business model. The company is “all Hadoop, all the time,” he explained, and everything — including the hardware and the network — is optimized for particular aspects of Hadoop workloads and operations. Essentially, Stata told me, Altiscale wants to be companies’ Hadoop dial tone — when users need to run a job, the service should just be there ready to do it.

So, although Altiscale is a hosted service, it’s not exactly a cloud service as many people would define it. Rather than charge by the hour, for example, Stata’s experience suggests Hadoop services are best charged based on a monthly baseline usage with room even built in for reasonable overages. This is because companies familiar with Hadoop usually understand their baseline requirements, give or take a handful of additional jobs, and would prefer to be able to budget for that each month.

He compares traditional hourly cloud billing to cell-phone billing in the 1990s: “At the end of the month,” he joked, “you were typically surprised on the wrong side.” Altiscale is more like a wireless plan with a maximum amount of minutes per month and some rollover minutes included. In fact, Stata said,  ”We’re pretty forgiving in terms of the limits. … As long as you’re not abusive, you don’t get charged more for it.”

And unlike many other Hadoop services, Altiscale isn’t immediately going after developers who want to try their hand at big data or deal with data through a wizbang interface. Rather, its initial audience is current Hadoop users — companies and data scientists — who know how the technology works but just want a better way to consume it. Right now, users access Altiscale by SSHing into a “desktop” environment (that’s actually hosted on Amazon Web Services) that gives them access to their favorite Hadoop tools such as MapReduce, Hive, Pig and Flume, as well as to data science tools such as R.

“We call that the scaling down problem,” Stata said.

What that means is that it takes a lot of effort to build a true self-service model that greenhorn Hadoop users can dive right into, and Altiscale would be irrelevant if waited to launch until it had figured that out. Part of that is a design problem, and part of that is a matter of Hadoop being designed to run better at scale. Plus, Stata added, the folks who got to first or second gear with Hadoop and then got stuck are way underserved right now.

However, although Altiscale might be about serving experienced Hadoop users with a more-managed experience, it’s not about serving legacy workloads. A lot of companies are using Hadoop today to somehow perform traditional enterprise data warehouse tasks or tie tightly into existing IT environments, he explained, but “we go after what I call ‘new data problems.’” That means online advertising and any workloads — servers log analysis, smart grid data, logistics, etc. — relying heavily on lots of sensor- or machine-generated data that can stream right into Hadoop.

Stata acknowledges it won’t be easy trying to win customer away from established Hadoop vendors such as Cloudera, MapR and Hortonworks (which many of Stata’s former Yahoo comrades founded), but, he told me a few months ago, he thinks its very doable. That’s because no matter how easy they make it to manage Hadoop, there’s a class of customers that’s just better served with a cloud service rather than trying to scale their operations staff and energy bill along with their Hadoop cluster.

“Self-managed Hadoop, essentially, is [those vendors'] ultimate goal,” Stata said. “Our goal is to to just take on the management responsibility, to take on all those management things the Yahoos and Googles do under the covers and just run Hadoop as a managed service. The winds of change are in our favor.”

If you want to hear more about where Hadoop is head, stop by our Structure conference next week, where I’ll be discussing that topic with Google Fellow and MapReduce creator Jeff Dean. Other webscale speakers include Facebook VP of Engineering Jay Parikh, Box VP of Engineering Sam Schillace and Amazon CTO Werner Vogels.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• 2012: The Hadoop infrastructure market booms

No prizes for guessing where the Toronto-based GhostBird folks are heading – as they wrote in a blog post on Wednesday, they’re off to help the revitalized Flickr team make new iOS and Android apps. KitCam and PhotoForge 2 won’t see any further updates, and you won’t be able to download them from the App Store anymore.

As for Rondee, well, this one is a bit more puzzling. It’s a San Diego, CA.-based free conference calling service that obviously caters to the enterprise market – or at least, it will be until July 12. The company’s website recommends that customers should flee to rival service Instant Conference.

Meanwhile, the Rondee team will join Yahoo’s Small Business unit. At the moment, there is no Yahoo conference call service for small businesses – that division only deals with web hosting, domain registration, ecommerce and marketing services and business email. That may change.

Neither the Rondee nor GhostBird takeovers came with disclosed terms, but I think it’s fair to say we’re not in Tumblr territory here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social networks will displace business processes, not socialize them
• Social first-quarter 2013: analysis and outlook
• Siri: Say hello to the coming “invisible interface”

One of the most tangible signs of this apathy (if that’s what it is) came from a Pew study that was released Tuesday of people’s attitudes towards the monitoring of their activity by the government. As described by my colleague Barb Darrow, over half of the Americans surveyed believed that the NSA’s behavior was “an acceptable price to pay to stop terrorism.” And almost half of those who participated agreed the NSA should be able to “monitor everyone’s email and other online activities if officials say this might prevent future terrorist attacks.”

A collective shrug of the shoulders

I confess that the Pew survey didn’t surprise me at all. Within hours of the Guardian and Post stories, there were responses flying through my Twitter stream and on other social platforms that amounted to a collective shrug of the shoulders: We knew this was happening, at least in general terms, most said — so why make such a big deal out of it? After all, the Patriot Act has been around since shortly after September 11 of 2001, and it authorizes a fairly wide variety of surveillance under certain conditions (although even the author of that Act says the NSA’s recent behavior over-reaches what was intended).

Another possible cause of the somewhat apathetic response is that so little is known about what the NSA is doing exactly. The initial reports said that the spy agency was provided with “direct access” to the servers of companies like Google, Facebook and Yahoo and could get whatever data it wanted — but then the qualifications started, and pretty soon it wasn’t clear what exactly “direct access” meant, or what kinds of data the NSA was permitted to get.

One thing PRISM makes clear is that all of those tin-foil hat wearing people knew what they were talking about.—
James Kendrick (@jkendrick) June 11, 2013

After some follow-up reporting from the Post, the New York Times and the Guardian, it appeared that the term “direct access” could refer to a system set up by the tech companies in question that would essentially automate official requests under the Foreign Intelligence Surveillance Act, providing a kind of secure area or “lock-box” to which only the NSA would have the keys. In other words, something that amounts to direct access by another name.

So one popular response boils down to: “Well, this is all legal and above-board then, so who cares?” — even though others pointed out that the court that approves the FISA orders is secret and has reportedly approved 99.9 percent of all requests made to it (according to other reports, the NSA and others can surveill wide ranges of data for weeks before they even need to get a court order, and other kinds of data such as chats may not even require an order).

Don't be distracted. The big story isn't Edward Snowden nor intelligence leaks. It's government's disregard for privacy and civil liberties.—
Robert Reich (@RBReich) June 11, 2013

We’re used to being snooped on

Part of the problem could be that, while leaker Edward Snowden’s dramatic revelations have gotten a lot of attention, there have been plenty of previous reports about similar activity over the past decade and very few of those have prompted much outrage at all. Former AT&T employee Mark Klein provided some pretty damning evidence in 2007 about how the NSA had set up a “secret room” filled with equipment that the spy agency could use to duplicate whatever data it wanted (using prisms, coincidentally enough) and that story eventually died.

Have views about personal privacy changed so dramatically? And if they have, are Google and Facebook and other “cloud” services part of the problem — or at least part of the reason — for those changes? It’s possible that we’ve become so used to the idea that our online lives are an open book, whether we like it or not, that there’s no longer any point in fighting it. If Google and Facebook are tracking all of our data and metadata in order to serve us ads, what difference does it make whether the National Security Agency is also doing it?

David Sirota at Salon makes a persuasive case for why the NSA’s actions aren’t even remotely the same as what Google and Facebook do — primarily because one is at least notionally voluntary. But that may not make a big difference to many users once the heat of the recent headlines has dissipated.

@mathewi I think it's that our networks already keep everyone informed by the second. We're all being "spied" on by Facebook, Twitter, et al—
Jack Narcotta (@JackN_TBR) June 11, 2013

What is there to be done?

The other complicating aspect is that, even if we were to get outraged about the NSA surveillance, it’s not clear what anyone could do about it. As more than one person has pointed out, it’s difficult to have the kind of “open debate” that President Obama said he wants to have about privacy and security when all of the actions of the NSA and PRISM are classified top secret, as are the court orders they use, and the secret court that produces them. Google isn’t even allowed to say that it gets FISA orders, although it is now trying to get approval to at least mention them.

There are some efforts aimed at pushing the debate forward, including a project called StopWatching.us, which is backed by a coalition of companies and entities — including the open-source Mozilla project and the Electronic Frontier Foundation. But all it plans to do is send a letter to Congress asking for changes to the Patriot Act and FISA (which some members of Congress are also asking for). There’s no real chance of a “Stop SOPA” type of movement because Google and Yahoo and Facebook are all involved in what is being protested about.

And the final death knell for broader interest in such matters is that the news cycle inevitably moves on. The NSA story is complicated, information about it is fragmented and contradictory, and most of the answers are top-secret and therefore unlikely to ever be confirmed. Not a great recipe for something that’s going to hold the attention of the average reader.

Post and thumbnail photos courtesy of Flickr user Thomas Leuthard

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• How the mega data center is changing the hardware and data center markets
• A near-term outlook for big data

This is the second billion dollar exit where Facebook lost out. In case of Tumblr, Facebook’s Mark Zuckerberg offered around $500 million to David Karp, but Marissa Mayer charmed Karp and, of course, more than doubled the money she was willing to pay for Tumblr. It was rumored that Facebook were in the running to buy Waze, but couldn’t pull the trigger.

Actually, selling to Google (or anyone else) was actually the only outcome for this company — even though it had tens of millions of people using the software in dozens of countries worldwide, it would have been pretty hard for them to turn social commuting into a real business. Google, on the other hand, can simply layer this on its maps and try and use the data to drive more real world transactions.

As I pointed out in a post about the new Google Maps, Google will ultimately create more natural advertising formats for maps-driven interfaces and Waze helps them towards that objective. That said, it is a great exhale for Waze’s investors, who were facing the prospect of building a real business — a much harder proposition than most in Silicon Valley understand or are willing to admit.

The big winners in this deal are investors that include Magma Ventures, Blue Run Ventures and Vertex Ventures, who were earliest backers of the company. Sources say they each made well north of $100 million from the deal. The surprise (and ironic) winner might be Microsoft, which is rumored to have invested in the company as a strategic investor.

The company had raised $12 million in its Series A funding in early 2008 and was valued at around $20 million at that time. It snagged another $25 million Series B funding in November 2010 and was valued at just under $100 million after that round of funding. The company received cash from Qualcomm Ventures and Microsoft in addition to other internal investors.

Horizon Ventures and Kleiner Perkins Caufield & Byers were the last money into the company, but they too have made a nice chunk of change on this deal. In October 2011, the company received another $30 million in funding from Horizon Ventures and Kleiner Perkins Caufield Byers at a pretty hefty valuation – around $250 million, according to sources.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s IPO filing: ideas and implications
• Sector RoadMap: Content personalization in 2013
• Social first-quarter 2013: analysis and outlook

This tidbit comes courtesy of app developers, who of course are now able to play with the preview of iOS 7 ahead of its consumer launch later this year. A Yandex spokesman subsequently confirmed the inclusion to me.

Here are tweets from devs in Russia:

Яндекс в iOS7 http://t.co/wnJMn2bqWr—
ʎɐsʇɐd ʎdɯnɹƃ (@alexmak) June 11, 2013

… And Turkey:

iOS 7 supports yandex search on Safari

@YandexComTr @yandex @YandexDestek http://t.co/0zCwTU5Oj2—
Bahaeddin Nakiboglu (@bahaeddin) June 10, 2013

Turkey is a particularly big win for Yandex, which is pushing hard into that country. As for the company’s more traditional markets, the inclusion of Yandex search as an option in Safari could even be seen as belated – the company has a majority share of the Russian search market, and a few months ago it even ranked higher than Microsoft’s Bing for numbers of searches on a global basis.

Yandex has already been supplying data for local users of Apple Maps since September last year.

Around the world, Google is the default search option for Apple’s customers. Bing and Yahoo are also global options. Not many local players get to join that list – although China’s Baidu is a notable exception in that country.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• Siri: Say hello to the coming “invisible interface”
• Takeaways from mobile’s second quarter

Companies such as Google, Facebook and Apple all claimed this week that they had never heard of a National Security Agency program called PRISM that reportedly gave intelligence analysts access to all kinds of personal data stored on the servers of the largest internet companies on the planet. But while the NSA may not have shared the code name for the project, first disclosed in reports from The Washington Post and the Guardian Thursday, “the companies were essentially asked to erect a locked mailbox and give the government the key,” the Times reported, citing “people briefed on the negotiations” between tech companies and the government. And while companies “bristled” at the request, they acquiesced.

Facebook — whose founder, Mark Zuckerberg, slammed what he called “outrageous press reports” in a statement Friday — actually built a system for the NSA that allowed it to securely transmit data to the government if requested under the FISA law, according to the report. It’s not clear what any of the other companies named as having negotiated with the government — Apple, AOL, Google, Microsoft, Paltalk and Yahoo — may have done to facilitate access to user data.

The companies’ carefully worded denials were likely constructed because the employees tasked with carrying out these systems “are not allowed to discuss the details even with others at the company, and in some cases have national security clearance,” the report said.

Twitter, which many noted was conspicuous in its absence from the list of companies named in the original reports, reportedly declined to provide the federal government with any assistance in getting access to user data in response to legal requests. “While handing over data in response to a legitimate FISA request is a legal requirement, making it easier for the government to get the information is not, which is why Twitter could decline to do so,” the report said.

Our tracking page on the unfolding PRISM story has a thorough explanation of the background to this report.

This post was updated at 11:43pm with the correct name of the NSA, the National Security Agency.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The state of cross-platform media measurement
• Q4 Wrap-up: SOPA and the future of digital content
• Connected world: the consumer technology revolution

This story is moving so quickly that it is hard to keep a handle on all of the developments, not to mention trying to follow the denials and non-denials from those who are allegedly involved, and the threads that tie this particular story to the long and sordid history of the U.S. government’s surveillance of its own citizens. So we thought it would be useful to try and collect what we know so far in a single post, which will be updated as often as possible with new information.

The Guardian leak

Guardian blogger and former lawyer Glenn Greenwald reports that the NSA has gotten a secret order from the Foreign Intelligence Surveillance Court that allows it to collect data about phone calls made by “millions of customers” on the Verizon network: location data, time and other identifying info about the call — everything except the actual content of the calls themselves (the Guardian has a background piece about what kind of metadata is available with such an order).

“The National Security Agency is currently collecting the telephone records of millions of US customers of Verizon, one of America’s largest telecoms providers, under a top secret court order issued in April. The order… requires Verizon on an ‘ongoing, daily basis’ to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.”

The leak widens

Other stories that follow the Guardian report quote anonymous sources saying the Verizon court order is a renewal of an order that has been in place for some time, and add that other telecom companies such as AT&T are also involved in similar programs. Greenwald notes in his story that the NSA started a program of bulk collection of telephone, internet and email records in 2001 under President Bush and this later caused controversy when it was reported in 2006 that the NSA had been saving all of this information and was analyzing it to try and detect terrorism.

Information-security experts and other industry watchers note after Greenwald’s story is published that the NSA and other government agencies have had these kinds of abilities for years thanks to laws such as the Protect America Act and the FISA Amendments Act. ProPublica has a roundup of what the government can find out about you and your behavior without a search warrant, and security expert Bruce Schneier says that what we don’t know about the government’s surveillance programs is even more frightening than what we do know.

Meanwhile, our Stacey Higginbotham wonders whether the NSA story will be a wakeup call about the power of big data, while Derrick Harris looks at how the security agency and other government entities analyze the vast amounts of information that come from such programs.

Freelance journalist Joshua Foust argues that the NSA revelations won’t cause most people to change their behavior — including their habit of voting for politicians who enact the kind of legislation that permits such surveillance — because they simply don’t care enough about the issue. Some experts said the kind of data the NSA is getting can be very powerful when it comes to finding patterns of behavior, but research from the Cato Institute says that even mining large amounts of data can turn out to be not that helpful when it comes to catching terrorists.

The Wall Street Journal, meanwhile, said that the NSA’s surveillance program was “legal and necessary” and the furor over the disclosure of this program was misplaced:

“Nobody’s civil liberties are violated by tech companies or banks that constantly run the same kinds of data analysis. We bow to no one in our desire to limit government power, but data-mining is less intrusive on individuals than routine airport security. The data sweep is worth it if it prevents terror attacks that would lead politicians to endorse far greater harm to civil liberties.”

The Washington Post leak

Within hours of the Guardian story appearing, the Washington Post reports that it has been leaked an internal slide presentation from the NSA that describes a program it calls PRISM — which involves the collection of email and other personal data from internet companies including Google, Microsoft, Facebook, Apple and Yahoo. According to the Post report (and a subsequent Guardian report based on a similar leak), this program has been underway since at least 2007, and involves what one NSA slide refers to as “data collected directly from the servers” of the companies named.

All of the companies who are reportedly involved in PRISM (which refers to them as “partners”) deny any knowledge of such a program, and say they only provide data when forced to do so by court order, and that they have no “back door” systems that would allow the NSA to do what it claims to be doing. These denials are met by widespread skepticism, and many observers — including TechCrunch founder turned VC Michael Arrington — wonder why insiders working at the tech giants allegedly involved in the program wouldn’t have leaked the information earlier.

The ongoing fallout

Some tech-industry observers say the denials from internet companies may be true, because they aren’t convinced the companies in question would even have to know about the NSA’s collection practices in order for them to work. The original Washington Post story is updated early Friday to note that it’s not clear whether “direct access” to the servers of those companies would be required, and quotes from another leaked document that says the program allows NSA officers to send “content tasking instructions directly to equipment installed at company-controlled locations,” which could mean boxes installed at ISP switches.

Several sources note that former AT&T employee Mark Klein revealed in 2007 that he had come across documents that showed the telecom company installed equipment — using glass prisms as “splitters” — that allowed the NSA to make a copy of the data stream coming from the AT&T network and send it to data-storage centers operated by the security agency. This was alleged to be part of a larger program that stored telephone calls, emails and other internet activity for the government and had been underway for years.

Some network analysts speculate that the NSA may be making use of equipment installed at CDNs (content delivery networks), which handle much of the data traffic for companies like Google and Yahoo. Laws passed in the U.S. require equipment makers such as Cisco to build into their products a way for law enforcement officials to tap into the streams they carry, and the NSA could be searching those streams directly instead of copying or storing all the data itself (since the cost of the program is a relatively cheap-sounding $20 million, according to the Post leak).

In a statement about the leaks, the Office of the Director of National Intelligence said that it does its best to work “within the constraints of the law” to collect information related to national security, and that unauthorized leaks such as those to the Guardian and Post “threatens potentially long-lasting and irreversible harm to our ability to identify and respond to the many threats facing our nation.”

Trying to track down PRISM

A search for entities that might be involved in the NSA program turns up software from a relatively secretive startup called Palantir — which has been funded by the CIA through its investment arm — that happens to be named PRISM. According to descriptions of the software, it allows clients of Palantir to sift through massive amounts of data and find patterns quickly.

Others are skeptical, however, that the software described could be used to do what the NSA appears to be doing, and security-industry sources say the NSA usually builds its own products and doesn’t like to use those from third parties. On Friday afternoon, Palantir told The Verge: “Palantir’s Prism platform is completely unrelated to any US government program of the same name.”

Former Reuters social-media editor Matthew Keys said on Twitter that he had found several references to the PRISM program in classified job listings dating back to 2007:

Not wanting to be left out, the secretive activist group Anonymous released some classified documents that refer to Defense Department information technology — but they appear to be mostly jargon-filled descriptions of the department’s IT infrastructure, with little or no connection to PRISM or any NSA-related data collection practices.

The ripples spread outside the U.S.

As our man in Europe — David Meyer — noted in a couple of posts Friday morning, the repercussions from the PRISM and NSA revelations are being felt in Europe as well, with some critics calling for changes to the so-called “Safe Harbor” program, which allows data about EU citizens to be stored by non-EU companies. And the Guardian has reported that the U.K. government appears to have been getting information via the PRISM program, which was designed to focus on the communication activity of non-U.S. residents (since U.S. law still technically prevents the government from spying on its own citizens without a warrant).

Meanwhile, President Obama — whom many critics have accused of carrying on with surveillance programs started by his Republican predecessor, despite his disavowal of such methods while campaigning — said through a spokesman that he “welcomes the discussion” about privacy and security:

“The president welcomes the discussion of the trade-off between security and civil liberties. The close examination of some of these complicated issues could cause people to arrive at differing opinions… The president welcomes that debate.”

Late Friday, the Guardian posted another security-related scoop, publishing what it called a “secret presidential directive” that orders the U.S. government’s top national security and intelligence officials to draw up a list of potential overseas targets that the U.S. could hit with cyber-attacks. The story goes on to say this operation:

“can offer unique and unconventional capabilities to advance US national objectives around the world with little or no warning to the adversary or target and with potential effects ranging from subtle to severely damaging”

Google denial and Sir Tim Berners-Lee

The creator of the world wide web, Sir Tim Berners-Lee, posted a statement at the Web Foundation blog saying:

“Today’s revelations are deeply concerning. Unwarranted government surveillance is an intrusion on basic human rights that threatens the very foundations of a democratic society. I call on all Web users to demand better legal protection and due process safeguards for the privacy of their online communications, including their right to be informed when someone requests or stores their data.”

And Google co-founder Larry Page posted a response Friday afternoon to the accusations in the Guardian and Post stories, written with Chief Legal Officer David Drummond, saying the company does not provide the government with “back door” access to its servers, and had never heard of the PRISM program until Thursday:

“Press reports that suggest that Google is providing open-ended access to our users’ data are false, period… Any suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”

Zuckerberg denial

Facebook co-founder and CEO Mark Zuckerberg posted a statement about PRISM on his Facebook page late Friday, saying he wanted to respond personally to the “outrageous press reports” about his company’s involvement in the surveillance scheme. In language very similar to the Google denial, Zuckerberg said the network has not been part of any program to give the U.S. government “direct access” to its servers.

“Facebook is not and has never been part of any program to give the US or any other government direct access to our servers. We have never received a blanket request or court order from any government agency asking for information or metadata in bulk, like the one Verizon reportedly received. And if we did, we would fight it aggressively. We hadn’t even heard of PRISM before yesterday.”

Does the NSA even need a back door?

Christopher Mims at the Atlantic business site Quartz quotes NSA veteran and whistle-blower William Binney — who was part of a group that asked the Defense Department to investigate the NSA in 2002 — saying the security agency could probably get its hands on about 80 percent of the web traffic that passes through the U.S. without even having direct access to the servers of companies like Google. That’s because the NSA has access to at least one of the largest communications hubs on the continent, as described by the Electronic Frontier Foundation.

The Wall Street Journal posted a story that quoted unnamed security experts who said the tech companies mentioned in the PRISM presentation could be telling the truth about not providing “direct access” to their servers, but still have their data collected by the NSA. The Journal said U.S. officials told the paper that the NSA “receives copies of the data through a system they set up with a court order.”

“One industry executive familiar with the handling of data requests from U.S. intelligence agencies said companies have set up ways to cope with the volume of data by automating parts of the process. This method would allow data to be funneled to intelligence agencies without the need for manual steps by company employees.”

At The Daily Beast, writer Megan McArdle looked at the issue of whether tech company denials should be believed or not, and quoted privacy expert Julian Sanchez from the Cato Institute saying there are a number of ways that the NSA could get the data it wants without requiring direct access, including the “secret room” with splitter equipment that Mark Klein described at AT&T (mentioned above):

“Most likely… is that they’ve got something akin to the “Secret Room” that Mark Klein disclosed in AT&T hubs where traffic is being cloned (the companies would need to provide the relevant SSL encryption keys) split off into NSA’s own machines. It would be literally true, in that case, that the NSA does not have direct access to Google’s servers.”

How PRISM might work in practice

Late Friday, the New York Times posted a story that said some tech companies resisted the NSA’s demands to provide easier ways to get access to user data — including Twitter — but that some consented, opened up discussions with the security agency about developing methods to share that data, and even “changed their computer systems to do so.”

“In at least two cases, at Google and Facebook, one of the plans discussed was to build separate, secure portals, like a digital version of the secure physical rooms that have long existed for classified information, in some instances on company servers. Through these online rooms, the government would request data, companies would deposit it and the government would retrieve it.”

In other words, “companies were essentially asked to erect a locked mailbox and give the government the key” and Facebook actually built such a system, the NYT story said. Declan McCullagh at CNET explained in a post that according to his sources, all that the PRISM process does is automate something that is required under FISA (the Foreign Intelligence Surveillance Act) — so court orders are given to the tech companies and they have simply made the process of handing over that information easier.

Marc Ambinder, a security expert who writes for The Week, also described his understanding of how PRISM functions — in a nutshell, PRISM is just a piece of software that allows the NSA to collect and interpret data that is handed over under FISA. The actual software itself isn’t classified, which is why mentions of it show up online and in job postings. In McCullagh’s piece, a former NSA lawyer says that the slide presentation the Washington Post published is “suffused with a kind of hype that makes it sound more like a marketing pitch than a briefing.”

Meanwhile, for those trying to keep track at home, the Electronic Frontier Foundation has put together a comprehensive timeline of events related to NSA surveillance activity over the past decade:

It was for your own good

First tech companies claimed they didn’t know anything about PRISM and weren’t supplying data (or at least not direct access), and now the story some sources close to those companies are telling is that they set up portals or some other method of complying with FISA requests in order to “protect the innocent,” according to a post at TechCrunch.

“The NSA may have wanted full firehoses of data from Google, Facebook and other tech giants, but the companies attempted to protect innocent users from monitoring via compliance systems that created segregated data before securely handing it over as required by law.”

The Guardian has responded to criticisms of its original description of PRISM and the whole notion of “direct access” — as well as the repeated denials from Google executives and others that this has been taking place — by posting another slide from the leaked NSA presentation. While some have speculated (as mentioned above) that PRISM could mean simply sucking data from ISP equipment, the NSA slide contrasts this method of getting data with PRISM’s, which it describes again as “collection directly from the servers” of the companies mentioned.

The Director of National Intelligence released another statement on Saturday, calling the disclosures by the Guardian and Washington Post about NSA data collection “reckless” and filled with “significant misimpressions.” So DNI James Clapper said he had declassified some details about the program, published in a fact sheet (PDF link). Among other things, it says:

“PRISM is not an undisclosed collection or data mining program. It is an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act.”

The Washington Post published a follow-up story on Saturday that described the PRISM process in much the same way as earlier stories from the Guardian and the New York Times: as a system or software that allowed the NSA to process FISA requests for information more quickly — and the paper reiterated earlier statements that because the program was top secret, only a few individuals within those companies would even know about it, let alone be able to discuss it. According to the Post:

“Executives at some of the participating companies, who spoke on the condition of anonymity, acknowledged the system’s existence and said it was used to share information about foreign customers with the NSA and other parts of the nation’s intelligence community.”

Much of the criticism about the original Post story and the Guardian story has focused on the description of PRISM as allowing “direct access” to the servers of companies like Google, Facebook and Yahoo — something the leaders of those companies have strenuously denied providing. The most recent Post story suggests that at least some of the debate over this term is semantic, and that its sources say PRISM did allow the NSA to get data from those companies directly:

“Intelligence community sources said that this description, although inaccurate from a technical perspective, matches the experience of analysts at the NSA. From their workstations anywhere in the world, government employees cleared for PRISM access may ‘task’ the system and receive results from an Internet company without further interaction with the company’s staff.”

NSA whistle-blower reveals his identity

In another bombshell, the Guardian revealed the identity of the whistle-blower who sent them the leaked documents about PRISM and the NSA surveillance program: he is Edward Snowden, a 29-year-old former technical assistant at the Central Intelligence Agency, and he is now living in Hong Kong and expects he will “never see home again.” He said his family doesn’t know about his activities, and that he fully expects to be charged and potentially face jail time for his actions.

In an interview with the Guardian, Snowden says that he gradually became frustrated with what the NSA was doing and believed it was wrong — but originally held off on leaking anything because he thought Barack Obama would change those policies when he was elected president. But Snowden says the president continued with “the policies of his predecessor” and so he decided to come forward and let the American public know what was happening behind closed doors:

“I don’t want to live in a society that does these sort of things … I do not want to live in a world where everything I do and say is recorded. That is not something I am willing to support or live under.”

Snowden also said the documents he leaked clearly show that “the NSA routinely lies in response to Congressional inquiries about the scope of surveillance in America” and that the abilities that he had as a contractor with the CIA were beyond what most people can even imagine:

“You are not even aware of what is possible. The extent of their capabilities is horrifying. We can plant bugs in machines. Once you go on the network, I can identify your machine. You will never be safe whatever protections you put in place.”

The reaction

In a post written for The Atlantic magazine, James Fallows said that the most frightening and important part about PRISM and the rest of the NSA surveillance activity revealed by Snowden is that it is all legal under the Foreign Intelligence Surveillance Act and other legislation.

“That these programs are legal — unlike the Nixon “Plumbers” operation, unlike various CIA assassination programs, unlike other objects of whistle-blower revelations over the years — is the most important fact about them. They’re being carried out in “our” name, ours as Americans, even though most of us have had no idea of what they entailed.”

Fallows — and others such as Talking Points Memo founder Josh Marshall — raised some question marks about the wisdom of Snowden’s choice of Hong Kong, which is still part of China and therefore not particularly open to harboring whistle-blowers. However, according to some experts in the law, Hong Kong might be a good place to seek asylum because of a loophole that could allow Snowden to remain there indefinitely.

Icelandic MP Birgitta Jonsdottir, an early supporter of WikiLeaks and of freedom-of-information laws in general, told Forbes magazine that she plans to try and get her country to offer Snowden political asylum. But observers of the political scene in Iceland say this might be more difficult than it would have been in the past, since the new Conservative government is seen as more friendly to the Obama administration.

Daniel Ellsberg — the man who leaked the famous “Pentagon Papers” in 1971 and revealed that the government had been lying about the Vietnam War — said in a piece written for the Guardian that Snowden’s leaks give the United States a chance to “roll back what is tantamount to an executive coup against the U.S. constitution.” Ellsberg said that Snowden’s revelations were the most important leak in the history of the United States, including his own.

“Since 9/11, there has been, at first secretly but increasingly openly, a revocation of the bill of rights for which this country fought over 200 years ago. In particular, the fourth and fifth amendments of the US constitution, which safeguard citizens from unwarranted intrusion by the government into their private lives, have been virtually suspended.”

Meanwhile, David Kirkpatrick — author of the book “The Facebook Effect” — asked whether the secrecy and privacy invasions involved in the PRISM program might impair the growth of social networks and cloud services like Facebook.

“Do we really want to impair such powerful tools for spreading dialogue, political discourse, and U.S. values? Is it worthwhile to impair the extraordinary financial and commercial success of these great flagships for the American economy? Does Obama want Facebook et al just to be seen as tools of American power?”

Politico took a look at some of the things that we still don’t know about PRISM and the activity involved in the NSA’s surveillance program — including how much data the spy agency has been collecting from phone companies as well as tech companies like Google, whether this data collection has actually thwarted any specific terrorist attempts or not (something that is the subject of much debate) and how exactly the PRISM program works in practice.

Meanwhile, the Daily Beast has a piece that looks at the group within the U.S. intelligence apparatus that hunt down leakers like Snowden, a kind of internal police force called the Associate Directorate for Security and Counterintelligence — or the Q Group for short. And Salon magazine has a feature and interview with Laura Poitras, the documentary film-maker who was contacted by Snowden and later helped both the Post and the Guardian write their stories about the leak.

Got anything I am missing? Let me know at mathew@gigaom.com

Post and thumbnail images courtesy of Shutterstock / Lightspring and the Washington Post

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution
• How the mega data center is changing the hardware and data center markets

Facebook’s data center in Prineville, Ore

They all denied — in carefully worded ways — that they provided direct access to customer information. The source of the original story — ostensibly leaked NSA slides obtained by the Washington Post and The Guardian — indicated that the National Security Agency tapped directly into these company servers to get at customer meta data.

In a radio interview Friday on WBUR, Post reporter Barton Gellman said it’s more likely that the slide was poorly worded and that the NSA placed its own “black boxes” on vendor property next to the servers in question. Those black boxes could mirror the server and be queried as a proxy while giving those vendors plausible deniability if asked whether their own servers had been accessed.

Obviously, if that is the case it’s hard for vendors to plead ignorance to what was going on. But there are ways the government could harvest people’s Google and Facebook and other data without those vendors knowing.

First, they could eavesdrop on the HTTP traffic flowing over the internet — which is not usually encrypted. Or there could be a covert back door into these services themselves, something that Jon Oltsik, senior principal analyst at Enterprise Strategy Group finds hard to believe.

And, there have been reports that government agencies are indeed collecting data provided from internet service providers and telcos.  On Friday, The Wall Street Journal said that the NSA’s gathering data on Verizon customers, is just the tip of the iceberg. According to the Journal:

“… people familiar with the NSA’s operations said the initiative also encompasses phone-call data from AT&T Inc. and Sprint Nextel records from Internet-service providers and purchase information from credit-card providers.

Update: One security expert who did not want to be named becuase he does work for government agencies said if the NSA is doing what it does best — which is traffic analysis. This is stuff like who is talking to whom and for how long and when. “It is a kind of social network mapping … questions arise if there is a radio in an uninhabited jungle [or] when I call Djokar Tsarnaev at 1 a.m. or when one call comes in and one call goes out continguously,” he said via email.

If this is the sort of traffic analysis the NSA is doing — the default assumption — then there is “no requirement for the cooperation of the endpoints, only the carriers,” he said. In all internet-based TCP/IP situations, the communications are all multi-hop, and thus there is no need to surveil all possible paths, just the “must go through” paths,” he continued. “If I can listen one hop outside your firewall, then there is nothing you can do about it, you won’t know I am doing it, and to the extent that traffic analysis is sufficient for the surveillance team, the job is done.”

This is really key stuff. When you post updates to your Facebook page or Google Drive, that data typically flows unencrypted over the web. That data-in-transit could, in theory, also be intercepted at the routers directing traffic or at Content Delivery Network (CDN) points that optimize traffic flow. We just don’t know, because security agencies won’t say. But the upshot is, if the government is collecting that traffic, it truly does have a ton of information about everything you do, or at least everything you say. That is truly a sobering use of big data.

As we learned last year, it may well be compiling a full dossier on you (and everyone) which it’s storing in the no-longer-quite-so-top-secret giant NSA data center in Bluffdale Bluffton, Utah,

Beyond that, we probably won’t know the truth of what Google, Microsoft, et al. knew and if or how much they participated in snooping for years to come.

This story was updated at 4 p.m. PDT to correct the name of the town where the NSA data center is located and again at 9:14 a.m. PDT on June 8 to add additional comment and context from a security expert.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• A near-term outlook for big data
• Connected world: the consumer technology revolution