• A single editor, programmed by Serge Wroclawski, idenfitied and properly standardized road names across the country. In six months, the program fixed 4,156,347 street names to a nicer looking long-form style.
• Roughly 20 percent of changes made within OpenStreetMap are done by more than 99 percent of the user population. The remaining one percent is in charge of the rest.
• In addition to millions of miles of road, the program has a database of 78 million buildings.

And those are just some of the numbers involved. The more important pieces are done in helpful, striking graphics that show the advances the program has made city by city. A before and after presentation of downtown Chicago shows off the hundreds of buildings put in place via a single governmental database of the city. There are “heat maps” of updates to major cities such as New York City, Tokyo, and Sydney that show where users have made the most changes over a period of time. There are even glimpses into the most densely-mapped data in the world — compiled from an independent list and largely featuring areas in France, the U.S., and Cameroon — that show extra markers like walking trails, hills and even individual trees. But perhaps the greatest thing the report shows is the consistent hard work put forth within the crowd that keeps OpenStreetMap running and updated. There’s perhaps no better example of the value communities can create than  Google’s acquisition of Waze today for an undisclosed sum, and it’s easy to see why: The power of many minds can lead to great work and amazingly gorgeous data. Interested in getting in on the action? OpenStreetMap is an ongoing project, and its new user orientation can be found here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining work in the digital age: an analysis by GigaOM Pro
• Connected consumer first-quarter 2013: Analysis and outlook
• Examining the rise of crowd labor platforms in 2012

This story is moving so quickly that it is hard to keep a handle on all of the developments, not to mention trying to follow the denials and non-denials from those who are allegedly involved, and the threads that tie this particular story to the long and sordid history of the U.S. government’s surveillance of its own citizens. So we thought it would be useful to try and collect what we know so far in a single post, which will be updated as often as possible with new information.

The Guardian leak

Guardian blogger and former lawyer Glenn Greenwald reports that the NSA has gotten a secret order from the Foreign Intelligence Surveillance Court that allows it to collect data about phone calls made by “millions of customers” on the Verizon network: location data, time and other identifying info about the call — everything except the actual content of the calls themselves (the Guardian has a background piece about what kind of metadata is available with such an order).

“The National Security Agency is currently collecting the telephone records of millions of US customers of Verizon, one of America’s largest telecoms providers, under a top secret court order issued in April. The order… requires Verizon on an ‘ongoing, daily basis’ to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.”

The leak widens

Other stories that follow the Guardian report quote anonymous sources saying the Verizon court order is a renewal of an order that has been in place for some time, and add that other telecom companies such as AT&T are also involved in similar programs. Greenwald notes in his story that the NSA started a program of bulk collection of telephone, internet and email records in 2001 under President Bush and this later caused controversy when it was reported in 2006 that the NSA had been saving all of this information and was analyzing it to try and detect terrorism.

Information-security experts and other industry watchers note after Greenwald’s story is published that the NSA and other government agencies have had these kinds of abilities for years thanks to laws such as the Protect America Act and the FISA Amendments Act. ProPublica has a roundup of what the government can find out about you and your behavior without a search warrant, and security expert Bruce Schneier says that what we don’t know about the government’s surveillance programs is even more frightening than what we do know.

Meanwhile, our Stacey Higginbotham wonders whether the NSA story will be a wakeup call about the power of big data, while Derrick Harris looks at how the security agency and other government entities analyze the vast amounts of information that come from such programs.

Freelance journalist Joshua Foust argues that the NSA revelations won’t cause most people to change their behavior — including their habit of voting for politicians who enact the kind of legislation that permits such surveillance — because they simply don’t care enough about the issue. Some experts said the kind of data the NSA is getting can be very powerful when it comes to finding patterns of behavior, but research from the Cato Institute says that even mining large amounts of data can turn out to be not that helpful when it comes to catching terrorists.

The Wall Street Journal, meanwhile, said that the NSA’s surveillance program was “legal and necessary” and the furor over the disclosure of this program was misplaced:

“Nobody’s civil liberties are violated by tech companies or banks that constantly run the same kinds of data analysis. We bow to no one in our desire to limit government power, but data-mining is less intrusive on individuals than routine airport security. The data sweep is worth it if it prevents terror attacks that would lead politicians to endorse far greater harm to civil liberties.”

The Washington Post leak

Within hours of the Guardian story appearing, the Washington Post reports that it has been leaked an internal slide presentation from the NSA that describes a program it calls PRISM — which involves the collection of email and other personal data from internet companies including Google, Microsoft, Facebook, Apple and Yahoo. According to the Post report (and a subsequent Guardian report based on a similar leak), this program has been underway since at least 2007, and involves what one NSA slide refers to as “data collected directly from the servers” of the companies named.

All of the companies who are reportedly involved in PRISM (which refers to them as “partners”) deny any knowledge of such a program, and say they only provide data when forced to do so by court order, and that they have no “back door” systems that would allow the NSA to do what it claims to be doing. These denials are met by widespread skepticism, and many observers — including TechCrunch founder turned VC Michael Arrington — wonder why insiders working at the tech giants allegedly involved in the program wouldn’t have leaked the information earlier.

The ongoing fallout

Some tech-industry observers say the denials from internet companies may be true, because they aren’t convinced the companies in question would even have to know about the NSA’s collection practices in order for them to work. The original Washington Post story is updated early Friday to note that it’s not clear whether “direct access” to the servers of those companies would be required, and quotes from another leaked document that says the program allows NSA officers to send “content tasking instructions directly to equipment installed at company-controlled locations,” which could mean boxes installed at ISP switches.

Several sources note that former AT&T employee Mark Klein revealed in 2007 that he had come across documents that showed the telecom company installed equipment — using glass prisms as “splitters” — that allowed the NSA to make a copy of the data stream coming from the AT&T network and send it to data-storage centers operated by the security agency. This was alleged to be part of a larger program that stored telephone calls, emails and other internet activity for the government and had been underway for years.

Some network analysts speculate that the NSA may be making use of equipment installed at CDNs (content delivery networks), which handle much of the data traffic for companies like Google and Yahoo. Laws passed in the U.S. require equipment makers such as Cisco to build into their products a way for law enforcement officials to tap into the streams they carry, and the NSA could be searching those streams directly instead of copying or storing all the data itself (since the cost of the program is a relatively cheap-sounding $20 million, according to the Post leak).

In a statement about the leaks, the Office of the Director of National Intelligence said that it does its best to work “within the constraints of the law” to collect information related to national security, and that unauthorized leaks such as those to the Guardian and Post “threatens potentially long-lasting and irreversible harm to our ability to identify and respond to the many threats facing our nation.”

Trying to track down PRISM

A search for entities that might be involved in the NSA program turns up software from a relatively secretive startup called Palantir — which has been funded by the CIA through its investment arm — that happens to be named PRISM. According to descriptions of the software, it allows clients of Palantir to sift through massive amounts of data and find patterns quickly.

Others are skeptical, however, that the software described could be used to do what the NSA appears to be doing, and security-industry sources say the NSA usually builds its own products and doesn’t like to use those from third parties. On Friday afternoon, Palantir told The Verge: “Palantir’s Prism platform is completely unrelated to any US government program of the same name.”

Former Reuters social-media editor Matthew Keys said on Twitter that he had found several references to the PRISM program in classified job listings dating back to 2007:

Not wanting to be left out, the secretive activist group Anonymous released some classified documents that refer to Defense Department information technology — but they appear to be mostly jargon-filled descriptions of the department’s IT infrastructure, with little or no connection to PRISM or any NSA-related data collection practices.

The ripples spread outside the U.S.

As our man in Europe — David Meyer — noted in a couple of posts Friday morning, the repercussions from the PRISM and NSA revelations are being felt in Europe as well, with some critics calling for changes to the so-called “Safe Harbor” program, which allows data about EU citizens to be stored by non-EU companies. And the Guardian has reported that the U.K. government appears to have been getting information via the PRISM program, which was designed to focus on the communication activity of non-U.S. residents (since U.S. law still technically prevents the government from spying on its own citizens without a warrant).

Meanwhile, President Obama — whom many critics have accused of carrying on with surveillance programs started by his Republican predecessor, despite his disavowal of such methods while campaigning — said through a spokesman that he “welcomes the discussion” about privacy and security:

“The president welcomes the discussion of the trade-off between security and civil liberties. The close examination of some of these complicated issues could cause people to arrive at differing opinions… The president welcomes that debate.”

Late Friday, the Guardian posted another security-related scoop, publishing what it called a “secret presidential directive” that orders the U.S. government’s top national security and intelligence officials to draw up a list of potential overseas targets that the U.S. could hit with cyber-attacks. The story goes on to say this operation:

“can offer unique and unconventional capabilities to advance US national objectives around the world with little or no warning to the adversary or target and with potential effects ranging from subtle to severely damaging”

Google denial and Sir Tim Berners-Lee

The creator of the world wide web, Sir Tim Berners-Lee, posted a statement at the Web Foundation blog saying:

“Today’s revelations are deeply concerning. Unwarranted government surveillance is an intrusion on basic human rights that threatens the very foundations of a democratic society. I call on all Web users to demand better legal protection and due process safeguards for the privacy of their online communications, including their right to be informed when someone requests or stores their data.”

And Google co-founder Larry Page posted a response Friday afternoon to the accusations in the Guardian and Post stories, written with Chief Legal Officer David Drummond, saying the company does not provide the government with “back door” access to its servers, and had never heard of the PRISM program until Thursday:

“Press reports that suggest that Google is providing open-ended access to our users’ data are false, period… Any suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”

Zuckerberg denial

Facebook co-founder and CEO Mark Zuckerberg posted a statement about PRISM on his Facebook page late Friday, saying he wanted to respond personally to the “outrageous press reports” about his company’s involvement in the surveillance scheme. In language very similar to the Google denial, Zuckerberg said the network has not been part of any program to give the U.S. government “direct access” to its servers.

“Facebook is not and has never been part of any program to give the US or any other government direct access to our servers. We have never received a blanket request or court order from any government agency asking for information or metadata in bulk, like the one Verizon reportedly received. And if we did, we would fight it aggressively. We hadn’t even heard of PRISM before yesterday.”

Does the NSA even need a back door?

Christopher Mims at the Atlantic business site Quartz quotes NSA veteran and whistle-blower William Binney — who was part of a group that asked the Defense Department to investigate the NSA in 2002 — saying the security agency could probably get its hands on about 80 percent of the web traffic that passes through the U.S. without even having direct access to the servers of companies like Google. That’s because the NSA has access to at least one of the largest communications hubs on the continent, as described by the Electronic Frontier Foundation.

The Wall Street Journal posted a story that quoted unnamed security experts who said the tech companies mentioned in the PRISM presentation could be telling the truth about not providing “direct access” to their servers, but still have their data collected by the NSA. The Journal said U.S. officials told the paper that the NSA “receives copies of the data through a system they set up with a court order.”

“One industry executive familiar with the handling of data requests from U.S. intelligence agencies said companies have set up ways to cope with the volume of data by automating parts of the process. This method would allow data to be funneled to intelligence agencies without the need for manual steps by company employees.”

At The Daily Beast, writer Megan McArdle looked at the issue of whether tech company denials should be believed or not, and quoted privacy expert Julian Sanchez from the Cato Institute saying there are a number of ways that the NSA could get the data it wants without requiring direct access, including the “secret room” with splitter equipment that Mark Klein described at AT&T (mentioned above):

“Most likely… is that they’ve got something akin to the “Secret Room” that Mark Klein disclosed in AT&T hubs where traffic is being cloned (the companies would need to provide the relevant SSL encryption keys) split off into NSA’s own machines. It would be literally true, in that case, that the NSA does not have direct access to Google’s servers.”

How PRISM might work in practice

Late Friday, the New York Times posted a story that said some tech companies resisted the NSA’s demands to provide easier ways to get access to user data — including Twitter — but that some consented, opened up discussions with the security agency about developing methods to share that data, and even “changed their computer systems to do so.”

“In at least two cases, at Google and Facebook, one of the plans discussed was to build separate, secure portals, like a digital version of the secure physical rooms that have long existed for classified information, in some instances on company servers. Through these online rooms, the government would request data, companies would deposit it and the government would retrieve it.”

In other words, “companies were essentially asked to erect a locked mailbox and give the government the key” and Facebook actually built such a system, the NYT story said. Declan McCullagh at CNET explained in a post that according to his sources, all that the PRISM process does is automate something that is required under FISA (the Foreign Intelligence Surveillance Act) — so court orders are given to the tech companies and they have simply made the process of handing over that information easier.

Marc Ambinder, a security expert who writes for The Week, also described his understanding of how PRISM functions — in a nutshell, PRISM is just a piece of software that allows the NSA to collect and interpret data that is handed over under FISA. The actual software itself isn’t classified, which is why mentions of it show up online and in job postings. In McCullagh’s piece, a former NSA lawyer says that the slide presentation the Washington Post published is “suffused with a kind of hype that makes it sound more like a marketing pitch than a briefing.”

Meanwhile, for those trying to keep track at home, the Electronic Frontier Foundation has put together a comprehensive timeline of events related to NSA surveillance activity over the past decade:

It was for your own good

First tech companies claimed they didn’t know anything about PRISM and weren’t supplying data (or at least not direct access), and now the story some sources close to those companies are telling is that they set up portals or some other method of complying with FISA requests in order to “protect the innocent,” according to a post at TechCrunch.

“The NSA may have wanted full firehoses of data from Google, Facebook and other tech giants, but the companies attempted to protect innocent users from monitoring via compliance systems that created segregated data before securely handing it over as required by law.”

The Guardian has responded to criticisms of its original description of PRISM and the whole notion of “direct access” — as well as the repeated denials from Google executives and others that this has been taking place — by posting another slide from the leaked NSA presentation. While some have speculated (as mentioned above) that PRISM could mean simply sucking data from ISP equipment, the NSA slide contrasts this method of getting data with PRISM’s, which it describes again as “collection directly from the servers” of the companies mentioned.

The Director of National Intelligence released another statement on Saturday, calling the disclosures by the Guardian and Washington Post about NSA data collection “reckless” and filled with “significant misimpressions.” So DNI James Clapper said he had declassified some details about the program, published in a fact sheet (PDF link). Among other things, it says:

“PRISM is not an undisclosed collection or data mining program. It is an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act.”

The Washington Post published a follow-up story on Saturday that described the PRISM process in much the same way as earlier stories from the Guardian and the New York Times: as a system or software that allowed the NSA to process FISA requests for information more quickly — and the paper reiterated earlier statements that because the program was top secret, only a few individuals within those companies would even know about it, let alone be able to discuss it. According to the Post:

“Executives at some of the participating companies, who spoke on the condition of anonymity, acknowledged the system’s existence and said it was used to share information about foreign customers with the NSA and other parts of the nation’s intelligence community.”

Much of the criticism about the original Post story and the Guardian story has focused on the description of PRISM as allowing “direct access” to the servers of companies like Google, Facebook and Yahoo — something the leaders of those companies have strenuously denied providing. The most recent Post story suggests that at least some of the debate over this term is semantic, and that its sources say PRISM did allow the NSA to get data from those companies directly:

“Intelligence community sources said that this description, although inaccurate from a technical perspective, matches the experience of analysts at the NSA. From their workstations anywhere in the world, government employees cleared for PRISM access may ‘task’ the system and receive results from an Internet company without further interaction with the company’s staff.”

NSA whistle-blower reveals his identity

In another bombshell, the Guardian revealed the identity of the whistle-blower who sent them the leaked documents about PRISM and the NSA surveillance program: he is Edward Snowden, a 29-year-old former technical assistant at the Central Intelligence Agency, and he is now living in Hong Kong and expects he will “never see home again.” He said his family doesn’t know about his activities, and that he fully expects to be charged and potentially face jail time for his actions.

In an interview with the Guardian, Snowden says that he gradually became frustrated with what the NSA was doing and believed it was wrong — but originally held off on leaking anything because he thought Barack Obama would change those policies when he was elected president. But Snowden says the president continued with “the policies of his predecessor” and so he decided to come forward and let the American public know what was happening behind closed doors:

“I don’t want to live in a society that does these sort of things … I do not want to live in a world where everything I do and say is recorded. That is not something I am willing to support or live under.”

Snowden also said the documents he leaked clearly show that “the NSA routinely lies in response to Congressional inquiries about the scope of surveillance in America” and that the abilities that he had as a contractor with the CIA were beyond what most people can even imagine:

“You are not even aware of what is possible. The extent of their capabilities is horrifying. We can plant bugs in machines. Once you go on the network, I can identify your machine. You will never be safe whatever protections you put in place.”

The reaction

In a post written for The Atlantic magazine, James Fallows said that the most frightening and important part about PRISM and the rest of the NSA surveillance activity revealed by Snowden is that it is all legal under the Foreign Intelligence Surveillance Act and other legislation.

“That these programs are legal — unlike the Nixon “Plumbers” operation, unlike various CIA assassination programs, unlike other objects of whistle-blower revelations over the years — is the most important fact about them. They’re being carried out in “our” name, ours as Americans, even though most of us have had no idea of what they entailed.”

Fallows — and others such as Talking Points Memo founder Josh Marshall — raised some question marks about the wisdom of Snowden’s choice of Hong Kong, which is still part of China and therefore not particularly open to harboring whistle-blowers. However, according to some experts in the law, Hong Kong might be a good place to seek asylum because of a loophole that could allow Snowden to remain there indefinitely.

Icelandic MP Birgitta Jonsdottir, an early supporter of WikiLeaks and of freedom-of-information laws in general, told Forbes magazine that she plans to try and get her country to offer Snowden political asylum. But observers of the political scene in Iceland say this might be more difficult than it would have been in the past, since the new Conservative government is seen as more friendly to the Obama administration.

Daniel Ellsberg — the man who leaked the famous “Pentagon Papers” in 1971 and revealed that the government had been lying about the Vietnam War — said in a piece written for the Guardian that Snowden’s leaks give the United States a chance to “roll back what is tantamount to an executive coup against the U.S. constitution.” Ellsberg said that Snowden’s revelations were the most important leak in the history of the United States, including his own.

“Since 9/11, there has been, at first secretly but increasingly openly, a revocation of the bill of rights for which this country fought over 200 years ago. In particular, the fourth and fifth amendments of the US constitution, which safeguard citizens from unwarranted intrusion by the government into their private lives, have been virtually suspended.”

Meanwhile, David Kirkpatrick — author of the book “The Facebook Effect” — asked whether the secrecy and privacy invasions involved in the PRISM program might impair the growth of social networks and cloud services like Facebook.

“Do we really want to impair such powerful tools for spreading dialogue, political discourse, and U.S. values? Is it worthwhile to impair the extraordinary financial and commercial success of these great flagships for the American economy? Does Obama want Facebook et al just to be seen as tools of American power?”

Politico took a look at some of the things that we still don’t know about PRISM and the activity involved in the NSA’s surveillance program — including how much data the spy agency has been collecting from phone companies as well as tech companies like Google, whether this data collection has actually thwarted any specific terrorist attempts or not (something that is the subject of much debate) and how exactly the PRISM program works in practice.

Meanwhile, the Daily Beast has a piece that looks at the group within the U.S. intelligence apparatus that hunt down leakers like Snowden, a kind of internal police force called the Associate Directorate for Security and Counterintelligence — or the Q Group for short. And Salon magazine has a feature and interview with Laura Poitras, the documentary film-maker who was contacted by Snowden and later helped both the Post and the Guardian write their stories about the leak.

Got anything I am missing? Let me know at mathew@gigaom.com

Post and thumbnail images courtesy of Shutterstock / Lightspring and the Washington Post

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution
• How the mega data center is changing the hardware and data center markets

Can it be over already?

Since both devices stepped on the block, there has been endless comparisons between iPhone and Android. Researchers conduct countless “head-to-head” data analyses, polls and measurements to find out which is the better phone. But, especially within the last year, a clear (if unfavorable) result has emerged: iPhone and Android are different phones with different approaches. As such, comparing the two is fruitless.

A study by mobile company Flurry shows what months of studies have proven: iPhone and Android are inherently better in different ways. Android leads slightly in global device share, and Apple commands a stronger share of apps both in volume and time spent.  The company certainly points to valid reasons why these statistics are the way that they are, including the fragmented nature of Android app development, greater dissemination of Android phones in general, and the immense push that Apple has made in advertising premium apps. But it doesn’t tell  us anything new – it just casually avoids the idea that comparing the two is moot.

Android devices and iPhones aren’t in the same race, and it’s impossible to even tell if they’re on similar tracks. Android has achieved the great feat of widespread penetration all across the world, and its unique hardware options mean that low-cost devices can successfully navigate rising markets. Meanwhile, Apple has entered the rising global hunger for smartphones with little momentum, although its recent foray into financial subsidies has worked well in India.

Android’s diversified system and widespread market appeal simply cannot be compared against the more concentrated pockets of iPhone users, and sometimes these statistics don’t actually prove dominance in a particular market. Sure, Android may get more downloads overall, but Apple’s revenue from the App Store remains unmatched. Other times, they’re different depending on what time, place and particular region of the world.

In a lot of ways, it comes down to the semantics of how we view this debate: holding a head to head between two different phones with two different aims and environments is just impossible. It discounts the strengths and weaknesses both have, and it eliminates burgeoning competition from Windows Phone and Blackberry.

So let’s agree that this “head-to-head” nonsense just doesn’t work anymore.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• CES 2012: a recap and analysis
• Connected world: the consumer technology revolution
• The state of cross-platform media measurement

What the MIT researchers found, according to a press release (the actual study is behind the Nature paywall), is that their model turns out to be an accurate predictor of cities’ productivity based on gross domestic product and patent rates. They developed their model based on whatever information was available about the American and European cities they studied, based on everything from the rate of simultaneous Foursquare checkins to the contagion rates of sexually transmitted diseases — all things that suggest people being in close physical proximity.

However, lead researcher Wei Pan noted, more people doesn’t necessarily mean more social-tie density. He pointed to Beijing as an example of a city that’s growing like crazy but that’s nearly impossible to traverse from one side to the other because it keeps growing outward and there aren’t adequate options for transportation. As a result, people pretty much see the people they live near and hardly anybody else in any meaningful manner.

A sprawling Beijing from Fragrant Hill. Source: Derrick Harris

In stark contrast is Zurich, a city where many people outside the urban center but where a massive investment in public transportation means about 60 percent of them make their way inward during the work day. It has experienced an increase in social-tie density and productivity and even a reduction in crime.

The obligatory caveat here is that the correlations these researchers found don’t necessarily prove their theory. Other factors not analyzed could certainly be at play.

But their theory certainly seems feasible when one considers other systems where entities are forced to interact. As much as I enjoy working from home, for example, it’s easy to get caught up interacting only with my editorial colleagues (confession: we have a lot of employees on the business side, and I don’t know many of their names). I don’t think I’m alone, and I do think a lot of potentially good ideas never get a chance to formulate because our paths and our thoughts rarely cross.

There’s no doubt that companies in Silicon Valley feel this way. Massive corporate campuses are designed to maximize interactions, and even those lowly startups and private companies tend to favor open floor plans that don’t hide people behind walls or cubicles. Mayer’s mandate might suck for employees accustomed to their home-office lifestyle but — from a productivity and creativity standpoint — it never seemed too crazy to me. Earlier research from MIT suggests she’s right, too.

Inside the Googleplex. Source: Google

Or look at the concept of big data, which is as much about breaking down siloes as it is about anything else. No matter how it’s accomplished from an infrastructure perspective (e.g., throwing everything in Hadoop), taking different types of data collected for different purposes on different systems and then analyzing them against each other is one of the ways companies hope to get more from their data than just the sum of its parts. Data that doesn’t interact with other data just does its job and nothing else.

In fact, one of the major themes at our Structure conference June 19 and 20 will be the importance of building infrastructure designed to handle the ever-expanding boundaries of IT operations. How do you keep a central view of data and activity spread across numerous services and endpoints, and then how do you ultimately make sure the data and systems that need to talk to each other are able to do so?

And if you think that urban planning, corporate campuses and big data don’t have anything to do with one another, think again. The MIT research is actually a follow up to a somewhat well-known 2010 study from the Santa Fe Institute that shows how cities experience super-linear growth in everything — productiviy and patents, but also crime, disease and pollution — as they grow in population. In part, the MIT researchers wanted to figure out what was causing the increases in productivity that accompany the growth.

If there are ways to design cities that can increase productivity while keeping crime and disease in check, that would be a huge deal. If the companies and institutions that exist within these cities and that we rely on for innovation can design themselves and their infrastructure in ways that encourage productivity — and, actually, creativity (something the Santa Fe Institute’s Geoffrey West has suggested is only possible up to a point) — perhaps we can actually help solve things like crime, disease and pollution before they spiral out of control.

Feature image courtesy of Shutterstock user pio3.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Key technologies for the smart city
• Social networks will displace business processes, not socialize them
• Social first-quarter 2013: analysis and outlook

“The way we create wealth is by creating platforms,” Schmidt said. For a data platform, application programming interfaces (APIs) can be core ingredients. That means data coming off of countless connected devices can be continuously pushed into to a common place, presumably onto something as accessible and intuitive as Google.

But Google might not end up hosting this platform. “The back-end services will get standardized, probably, by some companies that have not yet been founded,” Schmidt said.

The trick is to ensure that personal data remains anonymous. The nature of the company that attempts to launch a data platform could dictate what sorts of privacy issues crop up as a result. “A brand new company nobody knows is perfectly happy to take your address book and spam everybody to death,” Schmidt said. “Once they get to a certain size, they (might overstep and) apologize and stop doing it.” Such a company might not have viewed the data use as problematic, or maybe it was a matter of doing it to be competitive, Schmidt said. Larger companies, on the other hand, have to stand up to consumer and regulatory scrutiny, he said.

Google is well acquainted with this sort of blowback, most recently regarding the access of personal data through Google Street View cars, and also over sharing Google Buzz user data.

While there seems to be an opportunity when it comes to growing a widely accepted data platform, doing it while guaranteeing privacy could be a tall order, and anyway, plenty of startups are still wrestling with whether to open up their data for external uses. Progress will have to be made there, too, for a platform to be truly valuable.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The evolution of consumer-media cloud storage
• A near-term outlook for big data
• Connected world: the consumer technology revolution

I won’t waste your time recapping BigML — I’ve done it in detail before — but will note that the service does have some new features since the last time I played around with it. Among them is a new sunburst visualization to complement the classic tree one.

However, if you’re new to Quandl (like I am), it’s pretty cool. It’s a free service offering up more than 6 million financial, economic and social datasets that are neatly formatted and ready for consumption. Even better is that most (maybe all) of the datasets are organized by time, and Quandl automatically brings up an embeddable and interactive line chart when you click on the link to open the dataset.

Even better than that is the service’s “Supersets” feature, which lets you add columns from multiple datasets — in one click, mind you — together to form one big dataset comprised of a bunch of disparate variables. Someone interested in analyzing the unemployment rate in Nevada, for example, could create a Superset that compares it against other factors such as currency exchange rates, the U.S. Misery Index (the national unemployment rate plus the inflation rate) and residential energy consumption in California. I did just that, and the result looks like this (absent the energy consumption variable):

The table looks like this:

These could, of course, be completely unrelated variables — thus making any correlations all but meaningless — but my assumption is that Nevada depends heavily on tourism, so things going on nationally, internationally and in neighboring states could affect it. I might have chosen other variables, but one of the drawbacks of Quandl right now is that even though it has 6 million datasets, they’re not all super useful. Hopefully, that will change over time.

That aside, though, I think it’s pretty easy to see the value of a service like this.

Once you download the dataset as a CSV file and upload it to BigML, you can start making predictions. Here’s how all this data looks as a sunburst:

Here’s the prediction interface, which lets you adjust each variable using a slider:

This would work better with a larger dataset from which to derive the predictions (and probably if I had better data skills), but the bigger picture is this: I was able to do this in about an hour, and after a long day of working and parenting. As more datasets become publicly available, and as consumers begin getting deluged with their own data from activity trackers, health apps, Google, and any number of data-ownership or data-liberation efforts, they’ll probably want some way to start making sense of it all. None of the myriad services available for doing so are perfect, but we’re headed down the right track.

To prove that point, here are the Nevada unemployment and the Misery Index charted against each other using Datahero. It has improved quite a bit since entering public beta in April, including getting the export feature to work. This took about 2 minutes.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Sector RoadMap: Social customer service in 2013
• Connected consumer first-quarter 2013: Analysis and outlook

Bensen is a veteran of the remote access and networking world, wo brings his experience building voice over IP and remote terminal access technologies to investing in internet of things companies. His believe is that between 500 and 5,000 companies will be created (of course not all will succeed) to take advantage of the opportunity, likening it to the networking boom of the nineties. Unlike that boom, however he sees the eventual standards for IoT coming not just from networking giants, but healthcare, insurance and other companies that have traditionally held themselves aloof from broad tech industry standards setting. At the end of the podcast he also shares his favorite use case for his own connected devices.

Download this episode

Internet of Things Show RSS Feed

Subscribe to this show in iTunes

Listen on Stitcher

Show notes:
Host: Stacey Higginbotham
Guest: Woody Bensen, a general partner with Prism VentureWorks and a managing director of IoTWorks

• How big is the internet of things, and what kind of infrastructure does it need?
• We need systems integrators for building connected devices and services. But no one knows what features matter
• Securing the internet of things will happen at the gateway and needs a lightweight solution.
• Sharing use cases and swapping IFTTT recipes.

PREVIOUS IoT PODCASTS:

Podcast: How to design a connected device that isn’t a jerk, plus IoT’s recipe for success

Podcast: The history of the internet of things includes a Swedish hockey team and LEGOs

IoT Podcast: Where self-milking cows graze fields of data gold

Podcast: Power to the people — and all their connected devices

What you really need to know before buying connected devices

How the internet of things may make parents less worried but more neurotic

Shark Week for the internet of things

What the Internet of Things can learn from Minecraft and Lemmings

Podcast: How IBM uses chaos theory, data and the internet of things to fix traffic

Electric Imp aims to make the Internet of Things devilishly simple

When devices can talk, will they conspire against you?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Internet of things: creating tomorrow’s health care
• Siri: Say hello to the coming “invisible interface”
• The connected planet: Smartphones aren’t the only player

Scientists are hard at work creating robots that are able to sense and predict human actions, which should make them better performing tasks and look more natural while doing so. Researchers at Cornell University have trained a robot to recognize objects in its line of sight, as well as certain human actions, and then assign probabilities to the next set of possible actions. The video below (ignore the campiness of it) shows the robot in action trying to refill a cup of coffee — it must recognize the book and coffee cup on a person’s desk, predict which one he’s going to pick up, predict him taking a drink, then predict him setting the cup down for long enough for a pour to occur.

It’s kind of like IBM’s Watson system, only it’s predicting human actions instead of the correct answers to questions. In fact, both rely on databases full of relevant information — human movements and objects in the case of the robot — in order to make predictions. And instead of accepting natural-language queries like Watson does, the Cornell robot uses a Microsoft Kinect camera to visually detect what’s going on. (Interesting side note: Microsoft researchers used a Kinect and machine learning to train an elevator in the company’s research building to detect the difference between someone who intends to get on and someone who’s stopping to chat in front of it.)

Teaching robots to predict human gestures isn’t just about saving us a trip to the coffee pot, refrigerator or elevator control panel, though. As Stacey Higginbotham explained last week when covering a similar experiment by Disney Research, the more human-like the robot, the more comfortable people will be interacting with it. Important if you’re in the theme-park business, yes, but also if you’re trying to automate some of the caregiving functions that aging baby boomers will require in the next couple decades.

Feature image courtesy of Shutterstock user Ociacia.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• GigaOM Research highs and lows from CES 2013
• The Internet of things: creating tomorrow’s health care
• Connected world: the consumer technology revolution

But thanks to smarter connected devices, crowdsourcing trends and better medical data analytics and algorithms such clinical trials may become a thing of the past — or at least less of a burden. The launch this week of a crowdfunding campaign for the Scout, a home monitoring device that tracks, pulse, respiratory rate, blood pressure, temperature and other vitals, offers a perfect example of how clinical trials may change. As part of the company’s Indiegogo campaign for the Scout it’s inviting participants to opt into a what will become the usability study it submits to the U.S. Food and drug Administration for approval.

From the campaign web site:

Before Scanadu Scout™ can become a medical device it will have to go through the FDA (Food and Drug Administration) approval process and this is where your help comes in. With the Scanadu Scout™ you will help us by Scouting yourself and giving us feedback to refine the Scanadu Scout™.

This will happen in the framework of official clinical studies in which you will be invited to partake, ONLY IF YOU OPT-IN. For each study, some of you will be contacted and will have to sign an Informed Consent form. With your help we can put Scanadu Scout™ through FDA to become an over-the-counter consumer-grade diagnostic tool.

Scanadu, the company behind the Scout, isn’t the first or only company to recognize the power of connected devices, crowds and data, Transparency Life Sciences, a New York company started by a refugee from the pharmaceutical industry is also tackling the problem of slow, large and expensive clinical trials with crowdsourced data. TLS has built an online tool to collect information from researchers, physicians and patients that will then take their input to design an FDA-approved protocol for a drug study.

The FDA in December approved the first TLS protocol for a study on the effectiveness of a drug for Multiple Sclerosis patients. An industry consultant wrote that TLS took a process that takes 6 months and completed it in 6 weeks thanks to its ability to gather data from multiple sources into its tool that then formats the data properly. The study relies on remote patient monitoring to cut down on office visits, something that the Scout may one day be able to help with.

Remote monitoring cuts costs but also increases compliance and participation in the study, because it reduces doctor visits. Patients in the study still have access to a nurse or doctor and regular check ins, but they no longer have to spend a chunk of their day traveling to an office and waiting. While TLS and Scanadu are using the web to help speed up the FDA trial process, other doctors are thinking about using crowdsourced data to eliminate some of them, instead relying on crowdsourced data to monitor the efficacy of certain drugs and medical devices after their initial approval. And there are tons of startups out there thinking about finding and cataloguing patient data outside of formal trials, such as PatientsLikeMe, Medify and others in this story.

Much as mobile connectivity has changed the way people hail cabs, book tables at restaurants and share photos, the internet of things and the resulting data from consumer devices may soon change the way we test the efficacy and safety of our drugs. Figuring out the right balance of oversight and self-reporting in this new paradigm will be crucial, but it’s something that should happen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• 4 iPad apps to help wrangle data
• Dissecting the data: 5 issues for our digital future

President and COO George Mathew says the company’s mission is to be a one-stop shop for statistical analysis. It wants to be the one place where analysts and data scientists can blend their data, model it on it and then visualize it. Often, he noted, that same process might require two or three separate products.

Another feature that Alteryx hopes will set it apart is its collection of prebuilt models in what the company calls an analytics gallery. Users can share their own work or find models others have built for tackling similar issues. Alteryx also offers up its own pre-formatted datasets for analysis, often public data such as the U.S. census.

“The canvas for creating an analytics application should never be blank for the analyst when they’re getting started,” Mathew explained. They often need to understand external data as well as their internal data, so Alteryx’s software gives them easy access to it.

Because it’s based on the R statistical-programming language, heavy R user Walmart has been able to transition some workloads to Alteryx when employees need an easier user experience. McDonald’s uses it to analyze data about franchisees and about its growth strategy in China, and Bloomin’ Brands (parent of company of Outback Steakhouse and other restaurants) is using it to help build menus that take into account what diners in various parts of the country prefer to eat. Nine of the 10 leading top wireless providers providers are also users, Mathew said, trying to blend actual call data with traditional sources such as customer service data.

Mathew compares Alteryx’s current growth as analogous to software-as-a-service applications like Salesforce.com in the CRM space, or even Tableau in the traditional business-intelligence space. In a business world increasingly driven by at least the idea of big data, one might expect any vendor pushing a more consumer-like purchase and consumption experience to get interest from companies tired of dealing with legacy software or never wanting to experience it in the first place.

“The disruption that’s happening is creating a new space for ourselves,” Mathew said, “without having to go head to head, frankly, with the a status quo out there.”

Feature image courtesy of Shutterstock user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Social customer service in 2013
• The Internet of things: creating tomorrow’s health care
• Dissecting the data: 5 issues for our digital future