The new research is the latest in a number of similar initiatives that seek to mine web data to predict all kinds of events. Recorded Future, for instance, analyzes news, blogs and social media to “help identify predictive signals” for a variety of industries, including financial services and defense. Researchers are also using Twitter and Google to track flu outbreaks.

from “Mining the Web to Predict Future Events,” Horvitz and Radinsky, http://research.microsoft.com/en-us/um/people/horvitz/future_news_wsdm.pdf

Eric Horvitz of Microsoft Research and Kira Radinsky of the Technion-Israel Institute describe their work in a newly released paper, “Mining the Web to Predict Future Events” (PDF). For example, they examined the way that news about natural disasters like storms and droughts could be used to predict cholera outbreaks in Angola. Following those weather events, “alerts about a downstream risk of cholera could have been issued nearly a year in advance,” they write.

Horvitz and Radinsky acknowledge that epidemiologists look at some of the same relationships, but “such studies are typically few in number, employ heuristic assessments, and are frequently retrospective analyses, rather than aimed at generating predictions for guiding near-term action.” They outline the advantages that software has over humans in this area:

• Learning: Software “has the ability to learn patterns from large amounts of data, can monitor numerous information sources, can learn new probabilistic associations over time, and can continue to do real-time monitoring, prediction, and alerting on increases in the likelihoods of forthcoming concerning events.”
• Tireless researching: Software, with its “long tentacles into historical corpora and real-time feeds,” can dig up data that humans might never find because they’re too focused on “knowledge that is easily discovered in studies or available from experts.”
• Lack of bias: Software can assist “when inferences from data run counter to expert expectations,” or when “there is a significantly lower likelihood of an event than expected by experts based on the large set of observations and feeds being considered in an automated manner.”
• Greater access to news: “A system monitoring likelihoods of concerning future events typically will have faster and more comprehensive access to news stories that may seem less important on the surface (e.g., a story about a funeral published in a local newspaper that does not reach the main headlines), but that might provide valuable evidence in the evolution of larger, more important stories (e.g., massive riots).”

One of the problems that the researchers faced in developing their software model is the fact that tragic events in poor African countries are often not widely reported. So they taught the software to generalize somewhat: “Instead of considering only ‘Rwanda cholera outbreak,’ an event with a small number of historical cases, we consider more general events of the form: “[Country in Africa] cholera outbreak.” We turn to world knowledge available on the Web…[that] maps Rwanda to the following concepts: Republics, African countries, Land- locked countries, Bantu countries, etc.”

Horvitz and Radinsky also taught the software what to ignore: It “was able to recognize that the drought experienced in New York City on March 1989, published in the NYT under the title: ‘Emergency is declared over drought’ would not be associated with a disease outbreak…The system estimates that, for droughts to cause cholera with high probability, the drought needs to happen in dense populations (such as the refugee camps in Angola and Bangladesh) located in underdeveloped countries that are proximal to bodies of water.”

“I truly view this as a foreshadowing of what’s to come,” Horvitz told the MIT Technology Review. “Eventually this kind of work will start to have an influence on how things go for people.” He said Microsoft isn’t commercializing the research yet, but that it will continue, and he wants to get more “data further back in time.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The authors, Antonio A. Casilli and Paola Tubaro, detail their findings in a paper titled “Social Media Censorship in Times of Political Unrest – A Social Simulation Experiment with the UK Riots,” which appears in the July issue of the Bulletin of Sociological Methodology (it’s not yet available online, but an advance version is available here).

The research is especially timely given the attention social media has received during the revolutions and violent protests that have occurred worldwide over the past couple years. As the authors note when discussing the U.K. government’s response to riots in August 2011, “[T]he same information technologies that had been presented as tools of liberation in the height of the Arab Spring, have been portrayed as threats to the very values of freedom and peace that Western governments allegedly stand for.”

The authors attribute their findings (albeit computer-generated) largely to the idea of “vision,” which plays a pivotal role in sociological experiments trying to determine how individuals act during times of protest or rioting. Put simply, less censorship means more vision, so citizens (called “agents” in the computer model) know what’s going on around them and can act in more uniform and rational manners. More censorship means less vision, so citizens are less aware of their surroundings and tend to act randomly.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q2: Google closes the quarter with a bang
• Finding the Value in Social Media Data
• What Enterprise Software Vendors Could Learn from the Consumer Space

If you’re concerned about the rise of the “surveillance society,” in which the authorities use cameras and other means to snoop on your activities, the past week or so has probably added even more fuel to that fire. The British intelligence service is doing its best to crack encrypted BlackBerry instant messages  to identify rioters — and the police are using facial recognition to do the same — while some departments are setting up social-media observation units to track Twitter and Facebook, and others are using algorithms to try to predict where crimes will occur. In a world where our online activities are increasingly public, the bottom line is that governments have even more ability to observe our behavior, whether we like it or not.

Crawling through Facebook pages and Twitter feeds looking for dissidents is something that we’ve come to associate with repressive and totalitarian regimes like Egypt’s former dictatorship or the Chinese government, both of which have reportedly used Facebook’s “real name” policy — along with other methods such as geo-targeting — to identify disruptive elements via social media. Western governments have also proven to be interested in these kinds of technologies, however, particularly in the wake of events such as the recent riots in London, which have led the British prime minister to discuss potentially banning certain people from using social networks.

As I noted in a recent post, it’s not just Britain’s PM who is interested in taking these kinds of steps. A prominent British MP said shutting down social networks would be no different than closing a road during an emergency, and the Bay Area Rapid Transit authority actually shut down cellular networks in some of its stations in advance of a protest because it was afraid demonstrators would use cellphones to organize — a controversial move that the FCC is reportedly looking into.

Block them, or use them for surveillance?

But shutting down or blocking access to social media and social networks is one thing; the flip side of that is using these networks and tools to snoop on users who the police or other agencies believe need to be surveilled. There are brute-force attempts such as MI5′s plan to try to crack the encryption used by Research In Motion’s BlackBerry Messenger service, which was allegedly used by some London rioters to coordinate their activities, and then there are the various attempts at using facial-recognition software to identify rioters who posted photos to Facebook or elsewhere.

Interestingly enough, a civilian volunteer effort to do something similar using publicly available software and a Facebook app was recently abandoned, because the founders said the software’s ability to identify people was simply not good enough — raising the prospect of potential “false positives,” which could lead to innocent people being targeted by the authorities. And a number of observers have noted that even London’s much-criticized network of closed-circuit security cameras, which led some to call Britain the first modern “surveillance society,” didn’t have much effect on stopping or even quelling the recent riots.

Meanwhile, the New York Police Department has launched an official social-media monitoring branch, whose job it will be to track Twitter and Facebook for information that might lead to charges involving everything from disturbing the peace to gang violence (hopefully this will result in more serious charges than the recent arrest by British police of a man who planned a water-pistol fight using Facebook).

Could social media be used to predict crime?

In another recent effort that reminded some of the movie Minority Report, the Santa Cruz police department is experimenting with an algorithm-driven program that tries to identify where crimes might occur based on patterns from past arrests in the city. While the software doesn’t take into account posts from Twitter or Facebook, it’s easy to see how it could — in the same way that some people are trying to predict the movement of stocks and markets based on what people are posting to Twitter.

Although some (including me) have argued that the crackdown on social media being considered by Britain is as wrongheaded as a shutdown of communications services such as cellphones, there is one big difference between telephone conversations and Twitter or Facebook: namely, that one is private and the others are effectively public. That means while the police or the federal authorities would — in most cases, at least — have to get a warrant of some kind to tap your phone or eavesdrop on your computer, they can follow you on Twitter whenever they wish, and create profiles based on your Facebook activity or any other social-networking platform.

Doing this may well be beyond the abilities of most police forces, who are already stretched in dealing with the existing crimes they already know about — but it is certainly not beyond the abilities of MI5 or Scotland Yard or other Western intelligence services, many of whom are already using sophisticated data-collection methods to track suspicious activity on a number of communications networks including the Internet. How long until social media becomes part of that, if it isn’t already?

Post and thumbnail photos courtesy of Flickr users Jim Sher and Jennie Moo

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Social customer service in 2013
• Sector RoadMap: Content personalization in 2013
• Frenemy mine: The pros and cons of social partnerships for online media companies

The subway operator said it was to guarantee passengers’ safety, but others don’t see it that way, with groups like the Electronic Frontier Foundation calling it “a chilling strike against free speech.” The timing is also rather uncanny: The attempt to disrupt protesters by shutting down their ability to communicate by mobile phone comes right as the British government is discussing how to quell the modes of communication between rioters wreaking havoc in the UK.

BART shut off power nodes on the several platforms during busy commute times (between 4 p.m. and 7 p.m. on Thursday) and told carriers that operate service in the area about it after the fact, CNET reports. The subway system defended its actions in a statement on Friday, saying that it knew that organizers of the protest — which regarded the fatal shooting of a man by BART police in July — would be using mobile devices to coordinate their actions and relay the whereabouts of BART police. BART said the action was “one of many tactics to ensure the safety of everyone on the platform.”

Cellular service was not disrupted outside of Embarcadero, Montgomery, Powell and Civic Center stations, according to BART. “In addition, numerous BART Police officers and other BART personnel with radios were present during the planned protest, and train intercoms and white courtesy telephones remained available for customers seeking assistance or reporting suspicious activity,” the subway system’s statement reads.

BART’s actions sound similar to tactics UK Prime Minister David Cameron has been talking about in the wake of widespread rioting in his country that have alarmed free-speech activists – shutting down access to Twitter, Facebook and other social media sites and even BlackBerry Messenger when used “for ill.” The Guardian quoted Cameron on Thursday as saying:

And when people are using social media for violence we need to stop them. So we are working with the police, the intelligence services and industry to look at whether it would be right to stop people communicating via these websites and services when we know they are plotting violence, disorder and criminality.

“I have also asked the police if they need any other new powers.

“Police were facing a new circumstance where rioters were using the BlackBerry Messenger service, a closed network, to organise riots,” he said. “We’ve got to examine that and work out how to get ahead of them.”

Of course this is not lost on many how eerily similar both the measures employed by SF BART on Thursday and the kind the UK government is discussing are to the decision to cut Internet and cell phone service in Egypt during demonstrations against the government in early January.

Image courtesy of Flickr user ol slambert

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Survey: How apps can solve photo management
• What the Google-Motorola deal means for Android, Microsoft and the mobile industry

It seems totalitarian states like Egypt and Libya aren’t the only ones struggling with the impact of social media and the desire to muzzle services like Twitter and Facebook. In the wake of the riots in London, the British government says it’s considering shutting down access to social networks — as well as Research In Motion’s BlackBerry messenger service — and is asking the companies involved to help. Prime Minister David Cameron said not only is his government considering banning individuals from social media if they are suspected of causing disorder, but it has asked Twitter and other providers to take down posts that are contributing to “unrest.”

The British PM also said he has asked the police whether they need any new powers to stop the violence, including the ability to shut down social networks or communications services if they believe these tools are being used to incite unrest. Police in Britain have reportedly already begun arresting people based on their use of social media such as Twitter and Facebook, charging them with suspicion of inciting violence and/or disorder for posting tweets, and photos. In his statement, Cameron said:

[W]e are working with the police, the intelligence services and industry to look at whether it would be right to stop people communicating via these websites and services when we know they are plotting violence, disorder and criminality.

It’s not clear how the British government plans to identify who is “plotting” to commit violence or criminal acts using social media, however. Would posting a photo of a burning car be enough? Would retweeting someone who admitted to causing violence get a user’s account shut down, or result in questioning by the authorities? The prime minister didn’t say. Meanwhile, British Home Secretary Theresa May is reportedly meeting with Twitter, Facebook and BlackBerry to discuss their “responsibilities” during such events.

Twitter says it won’t remove posts or users

For its part, Twitter has said it has no intention of blocking any users’ accounts or removing their posts. A spokesman> who talked to The Telegraph about the issue referred to a Twitter blog post from earlier this year entitled “The Tweets Must Flow,” in which co-founder Biz Stone and Twitter’s general counsel Alex Macgillivray said: “We don’t always agree with the things people choose to tweet, but we keep the information flowing irrespective of any view we may have about the content.”

Those comments were made during the popular uprisings in Tahrir Square in Egypt, when the Egyptian government shut down access not just to social networks and mobile communications networks, but eventually to the entire Internet: an attempt to smother dissent that ultimately failed, and may have actually accelerated the revolution in that country. Britain’s prime minister and his government would no doubt argue that there is a world of difference between what they are doing and what Egyptian dictator Hosni Mubarak did, but free-speech advocates aren’t likely to agree.

As we’ve pointed out before, the role that Twitter and other social tools have played in the London riots is identical to the role they played in the uprisings and demonstrations in Tunisia, Egypt and other countries. In other words, they have allowed people to connect with each other and distribute information quickly and easily to tens of thousands of users of these services. That’s the power of real-time networks. Obviously, that has allowed some to spread misinformation and plan acts of violence — but it has also allowed others to correct that information, and to coordinate positive moves as well, such as planning a cleanup detail in the wake of the London riots.

As author and journalism professor Jeff Jarvis has noted in his response to the British prime minister’s comments, democratic governments have to be very careful in making moves that curtail free speech, even if they think their motivation is justified. And as others have pointed out, Britain is already on what many believe is the wrong side of the freedom of speech issue in other ways — including its support for so-called “super-injunctions” that restrict the publication of certain information about court cases in that country, and in some cases have resulted in bans on using social media.

Why not crack down on telephones too?

If social-media tools such as Twitter and Facebook hadn’t been invented yet, would Britain’s prime minister be considering a crackdown on telephone use (as author Douglas Adams once mentioned), or the publication of images on blogs or websites? Would the British police be questioning or arresting people for discussing the unrest and violence in bars or the public square? That seems unlikely (although not impossible). But the British government’s apparent willingness to consider shutting down or blocking access to Twitter and BlackBerry’s BBM falls into the same category.

Fundamentally, these tools are used for what amounts to public speech. That speech can be about violence and where a mob should go next to burn something, or it can be about how to overthrow a dictator. It can be about images of disorder and calls for looting, or it can be about how to organize a cleanup crew. It may be tempting to smother that kind of speech when a government feels it is under siege, as Britain seems to feel that it is. But doing this represents nothing less than an attack on the entire concept of freedom of speech, and that has some frightening consequences for any democracy.

Post and thumbnail photos courtesy of Flickr users Jennifer Moo and Petteri Sulonen

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• Survey: How apps can solve photo management

In the wake of a controversial police shooting, Britain’s capital city has been rocked by two straight days of widespread rioting and looting. As with previous riots — such as those in Vancouver, British Columbia following the Stanley Cup final — everyone seems to be looking for a culprit, with some blaming Twitter and Facebook, and others pinning the violence on BlackBerry and its instant messaging abilities. But that’s a little like blaming individual trees for the forest fire. As we’ve pointed out before with respect to the uprisings in Tunisia and Egypt, these are just aspects of our increasingly real-time, mobile and connected lives, and that can be an incredibly powerful force for both good and bad.

Although they are completely different in important ways, there are also some interesting similarities between the riots in London this weekend and the uprisings in Egypt’s Tahrir Square. Both were triggered by the death of a man whom some believed was unfairly targeted by the authorities. In Britain, it was Mark Duggan — a 29-year-old father of four shot dead after being stopped by the police — and in Egypt, it was Khaled Said, a 28-year-old businessman who was pulled from an Idknternet cafe and beaten to death by security forces. Both deaths also led to the creation of Facebook pages that became the focus of a social-media effort that ultimately fueled the protests.

Different causes, same network effects

That said, the two demonstrations obviously had completely different causes and outcomes. In Egypt, the protests were the result of decades of corrupt and authoritarian rule by a dictator, as well as food shortages, unemployment and so on — and they led to the toppling of the government, followed by the military taking control of the country. In Britain (as in Vancouver), the events that allegedly triggered the riots seemed to be mostly an excuse for young thugs to loot stores and burn things.

As more than one person has pointed out (including notorious social-media skeptic Malcolm Gladwell) these kinds of riots and uprisings have been happening for centuries, without the need for Twitter or Facebook or BlackBerry’s BBM. But while they may not cause revolutions, there’s no question that these kinds of mobile, real-time networks and technologies can help to fuel them when they occur. As Jared Cohen of Google Ideas described in the aftermath of the revolutions in Tunisia and Egypt, social media tools may not be a trigger for such events, but they can clearly act as “an accelerant.”

In some cases, this is because mobile and social tools like Twitter and Facebook and SMS messaging can be used to coordinate specific acts or gatherings, as some observers have said they were by thugs in London. But there’s another aspect to social-media use that can fuel these events, and that is the impact of seeing others posting about their behavior.

Social media provides validation

In other words, being able to see that demonstrators were revolting in Tunisia seemed to help trigger the same kind of response in Egypt, because it helped protesters in Tahrir Square in Egypt see themselves as part of a larger movement, or at least not alone in their desire to revolt. That’s a positive use of these tools (unless you’re a member of the totalitarian government in either country, of course), but the same phenomenon also theoretically makes it easier for people to justify their behavior in a riot in London, because others are doing the same thing.

Is this specific to social media like Twitter or Facebook? Hardly. As some noted about the almost hysterical coverage of these tools by mainstream media, television news reports and tabloid newspapers arguably do as much to publicize and legitimize that kind of behavior as any social network does.

The difference with Twitter and Facebook is that they are always on, and real-time in a way that even television often isn’t. But the real power comes from the connections that such tools allow between individuals: people who may not even know each other, but become part of a much larger phenomenon via their social connections and their ability to communicate quickly and easily. That can help citizens rise up against their dictatorial governments, but it can also help thugs and thieves take advantage of a cause to create panic and disorder. Unfortunately, you can’t have one without the other.

Post and thumbnail photos courtesy of Flickr user Petteri Sulonen

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Connected consumer first-quarter 2013: Analysis and outlook
• Best practices in optimizing content for social engagement

However, the company has a rich past that stretches back decades. Heck, its previous CEO was a former lineman who actually spliced cables for telephone wires. The striking workers, who are responsible for maintaining Verizon’s wireline telephone network as well as its fiber-to-the-home FiOS network, are being asked for the usual share of rising healthcare costs, but also that they become more flexible. As Verizon’s wireline DSL business shrinks, the company is still supporting an expensive workforce that’s responsible for a business that may soon represent a much smaller portion of Verizon’s profits.

Verizon is competing now against companies such as Amazon and Rackspace, which don’t have unionized workers, and against pay TV providers which sometimes do. As Verizon gets more digital, mobile and cloud-focused, its culture has to shift from one where a person has a single job for 40 hours a week with set time off to one where people work all out to deliver services and innovation for decent paychecks or the chance to take home a piece of the pie. There’s plenty of room for debate over whether this is a good thing, but it’s certainly the thing that Verizon has to adjust to. Will its workers let it?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• 12 tech leaders’ resolutions for 2012
• The Future of Pay TV Services

Multiple videos of the unarmed Grant being shot spread virally last January, resulting in a local surge of outrage that lead to rioting on the streets. This civil unrest was also well-documented on YouTube and other sites, and lead to the trial receiving a change of venue to downtown Los Angeles.

Of the options available to the jury trying Mehserle, involuntary manslaughter was the least severe sentence possible without actually acquitting him. This means that today’s verdict has many, to paraphrase the Kaiser Chiefs, predicting a riot. Via the Sacramento Bee comes the below livestream of Sacramento station Fox40′s coverage of reaction to the decision.

The Grant incident was an early and local example of the power video can have in a politically unstable situation, later followed by Iranian martyr figure Neda Agha Soltan, whose on-camera death had a global and profound impact on the 2009 Iranian uprising. So later tonight, appropriately enough given how this story began, video evidence of how Oakland chooses to deal with the verdict will undoubtedly be uploaded to YouTube.

Related GigaOm Pro Content (subscription required): Forget Twitter, the Real Firehose is Government