The study, which MIT’s Technology Review highlighted in a recent post by the Physics arXiv blog, was done by a pair of researchers in Virginia, Hany SalahEldeen and Michael Nelson. They took a number of recent major news events over the past three years — including the Egyptian revolution, Michael Jackson’s death, the elections and related protests in Iran and the outbreak of the H1N1 virus — and tracked the links that were shared on Twitter about each. Following the links to their ultimate source showed that an alarming number of them had simply vanished.

After two and a half years, 30 percent had disappeared

In fact, the researchers said that within a year of these events, an average of 11 percent of the material that was linked to had disappeared completely (and another 20 percent had been archived), and after two-and-a-half years, close to 30 percent had been lost altogether and 41 percent had been archived. Based on this rate of information decay, the authors predicted that more than 10 percent of the information about a major news event will likely be gone within a year, and the remainder will continue to vanish at the rate of .02 percent per day.

It’s not clear from the research why the missing information disappeared, but it’s likely that in many cases blogs have simply shut down or moved, or news stories have been archived by providers who charge for access (something that many newspapers and other media outlets do to generate revenue). But as the Technology Review post points out, this kind of information can be extremely valuable in tracking how historical events developed, such as the Arab Spring revolutions — which the researchers note was the original impetus for their study, since they were trying to collect as much data as possible for the one-year anniversary of the uprisings.

Other scientists, and particularly librarians, have also raised red flags in the past about the rate at which digital data is disappearing. The National Library of Scotland, for example, recently warned that key elements of Scottish digital life were vanishing into a “black hole,” and asked the government to fast-track legislation that would allow libraries to store copies of websites. Web pioneer Brewster Kahle is probably the best known digital archivist as a result of his Internet Archive project, which keeps copies of websites dating back to the early days of the web (Kahle also has a related project called the Open Library).

Getting access to social data is not easy

Although the Virginia researchers didn’t deal with it as part of their study, a related problem is that much of the content that gets distributed through Twitter — not just websites that are linked to in Twitter posts, but the content of the posts themselves — is difficult and/or expensive to get to. Twitter’s search is notoriously unreliable for anything older than about a week, and access to the complete archive of your tweets is only provided to those who can make a special case for needing it, such as Andy Carvin of National Public Radio (who is writing a book about the way he chronicled the Arab Spring revolutions).

As my colleague Eliza Kern noted in a recent post, an external service called Gnip now has access to the full archive of Twitter content, which it will provide to companies for a fee. And Twitter-based search and discovery engine Topsy also has an archive of most of the full “firehose” of tweets — although it focuses primarily on content that is retweeted a lot — and provides that to companies for analytical purposes. But neither can be linked to easily for research or historical archiving purposes. The Library of Congress also has an archive of Twitter’s content, but it isn’t easily accessible and it’s not clear whether new content is being added or not.

Twitter has talked about providing a service that would let users download their tweets at some point, but it hasn’t said when such a thing would be available — and even if users did create their own archive in this way (or by using tools like Thinkup from former Lifehacker editor Gina Trapani) it would be difficult to link those in a way that would provide the kind of connected historical information the Virginia study is describing. And it’s not just Twitter: there is no easy way to get access to an archive of Facebook posts either, although users in Europe can request access to their own archive as a result of a legal ruling there.

For better or worse, much of the content flowing around us seems to be just as insubstantial as the clouds that it is hosted in, and the existing tools we have for trying to capture and make sense of it simply aren’t up to the task. The long-term social effects of this digital amnesia remain to be seen.

Post and thumbnail images courtesy of Shutterstock user Ribah and Flickr user See-ming Lee

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s IPO filing: ideas and implications
• Defining work in the digital age: an analysis by GigaOM Pro
• Connected world: the consumer technology revolution

That’s why Storyful, a service that partners with media companies to aggregate and verify news from social networks, says it has decided to open up its formerly private Twitter account to help crowdsource the distribution and verification of breaking news reports.

Before he started the company in 2010, Storyful’s founder Mark Little was a foreign correspondent for a number of outlets such as Ireland’s Raidió Teilifís Éireann — much like Burt Herman, a former Associated Press reporter who started a company with a somewhat similar name: Storify. But while Storify is designed as a tool that anyone can use to pull together or “curate” a social-media stream from sources like Twitter and Flickr, the idea behind Storyful was to build a professional service staffed by journalists who could track breaking news reports through social networks and help media companies verify them. The company has a staff of 33 editors working in dozens of countries, and works with a number of outlets such as the New York Times and Reuters.

Collaboration is becoming a key journalistic skill

As part of its service, Storyful had a private Twitter account called StoryfulPro, which collected and distributed breaking news reports from both its own team and the various sources they monitored within their countries or their fields of expertise — including both professional journalists and citizen reporters, or what the company likes to call “networked journalists.” The primary audience for the account was over 1,000 professional journalists that Storyful had worked with before. On Tuesday, Little announced that the service had decided to make the Pro account public, allowing anyone to use or contribute to the process.

In a blog post, the Storyful founder said he decided to do this because he believes crowdsourcing is the best way to determine the truth of a breaking news report as quickly as possible. As he puts it:

“Storyful believes the key skill for journalists in a social age is collaboration. There really is no alternative to working with others in the Golden Hour. If a newsroom decides to go it alone, the chance you will be consistently first is nonexistent. The chance that you will often be wrong is 100 percent.”

As we’ve seen in a number of recent cases — including the mass shooting in Aurora, Colo. and the death by suicide of director Tony Scott — the pressure on media outlets of all kinds to break news first can result in a profusion of incorrect reports, which then get redistributed through Twitter and other social networks faster than any correction or clarification can match. Little’s phrase “the golden hour” refers to the first hour after a news event occurs, which Storyful believes is the most crucial period for fact-checking, and he says one of the most important contributions that can be made is when someone — either a professional journalist or reliable source — kills a false report before it can spread. Says Little:

“Breaking news now emerges in a ‘Golden Hour’, when skilled intervention is most valuable, when a celebrity death starts to trend on Twitter or an explosive video goes viral on YouTube. In this Golden Hour, the best journalists are often the ones who STOP a story, not start it.”

Crowdsourced news verification is almost always better

Storyful isn’t the only company or media-related startup that is trying to bring some kind of professional rigor to the process of real-time news verification: the NBC project Breaking News, which started as a Twitter account, also has a growing team that curates and distributes real-time news it has verified, and Sulia develops Twitter lists of credible sources (both professional and amateur) around various topics and breaking news events. Some media outlets also have their own internal teams that do this, such as the BBC’s “user-generated content desk,” which verifies reports from social media for use by BBC reporters.

I’ve argued before that one of the most compelling examples of crowdsourced news verification is the way that Andy Carvin of National Public Radio used his Twitter account as a real-time newswire — or what he prefers to call a public newsroom — to filter and verify reports coming out of Egypt, Libya and elsewhere, something other media outlets should emulate. And in a recent post, I also tried to make the case that this kind of verification or fact-checking is almost always better when it is done in public (although many readers seem to disagree with me on that).

One of the reasons for that is the amount of knowledge that can exist in what journalism professor Jay Rosen has called “the people formerly known as the audience.” Little says in his post that the company’s golden rule is that there is always someone closer to the story — and in many cases that person is not a traditional journalist or mainstream news source:

“Often, the closest person is still the wire reporter or networked journalist. But rarely do we rank the key source on the basis of authority and power. Authority has been replaced by authenticity as the currency of social journalism.”

Little says the closed nature of the Storyful Pro account always troubled him, because of his belief that crowdsourcing is almost always a better route to take for fact-checking the news (something he has written about in the past for the Nieman Foundation) and that’s why the decision was made to open it up. I’m glad the company decided to do so as well, because the more services and networks and media outlets there are trying to do this — whether it’s Storyful or Sulia or Breaking News or even Wikipedia — the better off we will all be as news consumers.

Post and thumbnail images courtesy of Flickr users Rosaura Ochoa and Petteri Sulonen

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Content Farms: The Players, The Benefits, The Risks
• Facebook’s IPO filing: ideas and implications
• NewNet Q1: Content Farms and Niche Networks on the Rise

By now, many people are familiar with the story of how NPR editor Andy Carvin used Twitter to create a kind of crowdsourced newswire during the Arab Spring revolutions in the Middle East last year, inventing a brand-new kind of journalism on the fly and in full public view. In a discussion with me on Thursday in Toronto about the lessons that can be learned from his experience, Carvin made some interesting points about the value of such an approach — including the importance of being transparent about the process, and the virtues of being human.

The discussion at the Mesh 2012 conference (full disclosure: I am a co-founder) touched on a number of different elements of what Carvin did during the uprisings in Tunisia and Egypt, including two important factors that allowed him to take on the role that he did. The first was the nature of his job at NPR, which — as a senior digital strategist — allowed him to experiment with new tools and take risks. The second was the fact that he already had a number of contacts in the Middle East through his work with Global Voices and other social advocacy groups (Harvard researcher and author David Weinberger, whom I also interviewed at the conference, live-blogged the session with Carvin).

Both of these meant that Carvin was perfectly positioned to do what he did when dissidents started revolting in Tunisia, and then following that in Egypt and Libya. He also noted with a laugh that “it helps when you have ADHD” (which he does), because for several months during the height of those revolutions, he was spending almost every waking minute reading or posting on Twitter, managing several lists of dissidents and thousands of responses from followers. His peak output reached 1,400 tweets a day at one point, whereupon Twitter blocked his account as spam.

Not a newswire, but a crowdsourced newsroom of public editors

But Carvin also talked about how he approached the reporting of real-time events on Twitter, and how he doesn’t really like having what he did called a “newswire.” Instead, he says he prefers to think of it as a crowdsourced newsroom — with him as the reporter, or the anchor (or “news DJ,” another term he likes to use) pulling in reports from different places, and then relying on his followers to act as editors and sources, fact-checking and verifying and also distributing the news that he was curating. As he put it:

I get uncomfortable when people prefer my twitter feed as a newswire. It’s not a newswire. It’s a newsroom. It’s where I’m trying to separate fact from fiction, interacting with people. That’s a newsroom.

In many cases, Carvin says, this process worked remarkably well — and quickly. In one photo of Egypt, for example, someone he asked for an opinion said that the corner of a building in the background was clearly a prominent local landmark, and then sent a link to a Google Earth view of the building, allowing Carvin to confirm within minutes that it was the same location. He also gave his followers what he called “fire drills,” in which he would ask them to fact-check photos that he knew were fake and then he would look at how many errors they found.

And what happened when he made a mistake and posted something that wasn’t accurate? In one case, he distributed a photo he thought was of a woman who had been shot in battle and was being attended to by nurses — but it turned out she was actually dead, his followers told him, and her body was being prepared for burial. Carvin says he admitted his mistake multiple times, and then retweeted both the criticisms and the corrections as broadly as possible:

You have to be prepared to be accountable in real time. When I screw up, my followers tell me.

News as a process, and the virtues of being human

The NPR editor, who is now working on a book about his experiences, says he believes in the “news as a process” approach, as author Jeff Jarvis and others have described it — in which not only is the reporting of an event crowdsourced in real time, but new information is added and mistakes are also corrected by readers, who journalism professor Jay Rosen has called “the people formerly known as the audience” (recent events have also shown how social networks like Twitter and Reddit can act as fact-checking engines).

As I tried to argue in a Twitter debate on Friday with a number of people (which Craig Silverman of the Poynter Institute curated with Storify) I think there is a lot of public value in doing what Carvin did, by assembling and fact-checking and correcting information in real time. That’s not to say editors don’t have value, or that reporters shouldn’t try to report things as accurately as possible. But when errors are made, I think admitting them publicly and being seen to correct them (not something traditional media is very good at) actually builds trust.

For me — and I think for Carvin — doing this is connected to a larger principle, and that is the value of being human, and of expressing that humanity, even if it means acknowledging a mistake. The NPR editor also admitted that in some cases he was so disturbed by the videos and images he was seeing from Egypt and elsewhere that he responded on Twitter in a way that he says might not have been professional — but he still felt was justified. As Weinberger noted in his live-blog: “Andy perfectly modeled a committed journalist who remains personal, situated, transparent, and himself.”

This is the kind of thing that mainstream media outlets discourage, just as many try to avoid admitting that they have made mistakes. Restrictive social-media policies put in place by many of these outlets seem designed to remove as many of the elements of being human as possible from the practice of being a journalist — which I think is the exact opposite of what needs to happen if traditional journalism is to survive. And I think Andy Carvin is a pretty good example of what one possible future of real-time, crowdsourced journalism actually looks like.

Post and thumbnail images courtesy of Flickr users Luc Legay and personaldemocracy

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Examining the rise of crowd labor platforms in 2012
• The quantified self: hacking the body for better health
• Facebook’s IPO filing: ideas and implications

As various levels of government both in the U.S. and around the world have stepped up their attempts to track down dissidents through social networks, the pressure has intensified on companies like Twitter and Facebook to comply with these demands — even at the expense of their users’ privacy. Despite that pressure, Twitter at least seems determined to fight these incursions wherever possible. As a case in point, the company has filed a motion in New York state court to quash a court order compelling it to hand over information about a user involved in the Occupy Wall Street protests, arguing that the order violates that individual’s rights.

The case in question involves a protester by the name of Malcolm Harris, whose Twitter handle was @destructuremal, and who was involved in a protest against Wall Street financial mismanagement in October of 2011 that saw more than 700 people arrested for a variety of charges, including destruction of public property and resisting arrest. Earlier this year, the New York district attorney’s office sent Twitter a subpoena for information relating to Harris’ use of the network during the protest — including personal details about him, and also specific messages that he sent.

Judge ruled that users do not own their tweets

As my PaidContent colleague Jeff John Roberts reported last month, Harris’ attempt to have this court order struck down failed for a somewhat unusual reason: namely, the judge hearing the case decided that Harris did not have any legal interest in the tweets he sent, because such rights only apply to things a user actually owns — and users do not own their tweets for the purposes of the U.S. Constitution. According to the judge:

While the Fourth Amendment provides protection for our physical homes, we do not have a physical “home” on the Internet… some of our most private information is sent to third parties and held far away on remote network servers.

Now Twitter has stepped in to try and force the court to quash the order, as reported first by the American Civil Liberties Union blog. According to Twitter’s motion (embedded below) the judge’s decision that Harris doesn’t own his tweets contradicts both Twitter’s terms of use — which specifically state that a users “retain [their] rights to any Content [they] submit, post or display on or through” the service — as well as the federal Stored Communications Act, which the Twitter motion says “expressly permits users to challenge demands for their account records.”

Twitter seems to be stepping up its opposition to these kinds of cases: in an earlier case related to the Boston version of the Occupy protests, Twitter handed over a user’s data to Boston police after they submitted a court order alleging that the user in question — who went by the names @@pOisAnON and Guido Fawkes — was involved in a hacking attempt on the police department. But the company said that it only handed over the minimum amount of information required by the police investigation.

Twitter says it remains committed to users’ rights

And while the company ultimately gave this user data to the Boston police, it defied a request not to make the police department’s court order public, something it has also done in the past in more serious cases, such as the Justice Department’s attempts to get details about the Twitter activity of WikiLeaks supporters such as Jacob Appelbaum and Icelandic MP Birgitta Jonsdottir. In that case, Twitter fought for the right to tell users that authorities were seeking the information — unlike a number of other companies who likely also got similar requests — although the company was eventually ordered to provide the data.

As it has evolved from being just a social toy into a real-time information network used by “Arab Spring” dissidents in countries such as Egypt and Tunisia, Twitter has repeatedly affirmed its commitment to free speech by its users — both in blog posts about how the “tweets must flow” despite attempts by governments to stop them, and in comments by CEO Dick Costolo and general counsel Alex Macgillivray that the company is the “free-speech wing of the free-speech party.”

That commitment was questioned by some when the company announced late last year that it had developed the capacity to selectively censor content from the network within specific countries, but Twitter stressed that it would only do this if required by law, and that it would publicize these requests at the Chilling Effects website. The latest moves in the Harris case suggest that Twitter plans to continue fighting for the rights of its users, and also that it intends to make these battles as public as possible. And Harris, who now tweets under the name @BigMeanInternet, seems pretty thankful.

Post and thumbnail images courtesy of Flickr users Hank Ashby and Petteri Sulonen

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s IPO filing: ideas and implications
• The 2013 task management tools market
• How consumer media will change in 2013

Cont3nt founder and CEO Anton Gelman calls the marketplace “an e-commerce platform for journalism,” enabling videographers and photographers to license their content to news agencies by giving them full control of the transaction. Journalists can specify 30-, 60- or 90-day exclusive or non-exclusive licenses to their content, with Cont3nt.com handling all billing and order fulfillment.

One big advantage of the Cont3nt platform is that it gives more money back to the content provider. While other marketplaces command as much as 50 or 60 percent of the license fee for content, Cont3nt takes a maximum of 5 percent per transaction. That’ll be key as it hopes to recruit more professional and semi-pro video journalists.

The goal is to bring to light video that might not have been seen, and to help monetize content that would have otherwise ended up on YouTube. While some news agencies have used YouTube footage in their reports, it’s often difficult to verify the accuracy of the reporting and the source of the content.

To that end, Cont3nt currently has about 170 publishers tapped into its platform, and about 15,000 journalists signed up from around the world. It’s focusing specifically on international markets like Sudan, Syria, Libya and Russia. As a result, much of its work has been around ensuring that journalists on the platform provide high-quality content.

“It’s very difficult to verify a piece of content,” Gelman told me in a phone interview. So Cont3nt works to verify and create a trusting relationship with the person delivering the content. To do so, the startup has been searching for good content online and trying to contact journalists directly, and has been recruiting media organizers who work in the regions it’s targeting. Once signed up, they can create media profiles and establish credibility as their content is licensed by major publishers.

All that said, there’s really only one major requirement for producers who wish to license their video on Cont3nt, according to Gelman: “It has to be true, it has to be good and it has to be yours.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• When video gets democratized, who wins and who loses?
• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1

Citizens in Homs have been reporting on the attacks through live streaming for days. Their footage has been picked up by numerous major media organizations, including the BBC and CNN, which are barred from reporting in the country. A Bambuser spokesperson estimated that about 90 percent of the footage coming out of Syria on recent days was originally broadcasted via Bambuser.

This isn’t the first time of live streaming and video hosting sites being blocked during the Arab Spring. Egypt and Lybia cracked down on Internet use during the uprisings in both countries, and Lybians resorted to sneakernet methods to smuggle footage out of their countries, uploading clips from Internet cafes in Egypt to circumvent their government’s censorship.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Controversy, courtrooms and the cloud in Q1
• How consumer media will change in 2013
• Facebook’s IPO filing: ideas and implications

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• Connected world: the consumer technology revolution
• Controversy, courtrooms and the cloud in Q1

The deal, which was announced late yesterday by Whisper Systems itself, was also confirmed by Twitter to the Wall Street Journal. Financial terms of the deal have not been disclosed.

Whisper Systems has been a fairly low-profile company that develops a number of mobile security products that protect voice calls, messaging and the sending of users’ private data to apps and the mobile web.

Perhaps one of the most-used has been RedPhone, which was taken up by people in Arab countries during the uprisings earlier this year to safeguard their voice communications on mobile devices. In its post announcing the deal, Whisper Systems’ development team noted that RedPhone would be “interrupted” immediately, while another product, FlashBack, is giving users one month to back up their data, before that too goes offline. They do promise, however, that both products will “live on.”

How they will live on, for now, is not entirely clear — although it makes perfect sense if Twitter is entertaining ideas of how it could offer different classes of service. For example, enterprise users could willing to pay Twitter for a more private and encrypted version of the platform. Or some of the services from Whisper Systems’ developers could be integrated into a larger analytics platform that would supplement the advertising and marketing products that Twitter is developing around its main site.

On a more general level, Twitter and Whisper Systems also appear to share a common goal of focusing more on user privacy. Whisper Systems was founded in 2010 by Moxie Marlinspike and Stuart Anderson, two IT security specialists who will now join Twitter. In announcing the acquisition, they wrote on their site that they formed the company to focus on mobile “not only because we saw it as an opportunity to reinvent the security solutions that never really worked in the PC environment to begin with, but also because the stakes are much higher – due to the nature of mobile devices themselves – and we didn’t like the way that things were looking.”

That is an area that Twitter, too, is having to scrutinize more carefully as well: Twitter has been investigated by the Federal Trade Commission over violations concerning how it safeguards users’ privacy, and as part of its settlement it has agreed to outside monitoring of its information security systems for the next decade. Buying Whisper Systems to beef that up can only be a help in that mandate.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Hyperlocal: opportunities for publishers and developers
• Facebook’s IPO filing: ideas and implications
• Connected world: the consumer technology revolution

There’s been a lot of talk recently about Twitter trending topics, and how they fail to reflect evolving events such as the Occupy Wall Street movement (although some argue that this is the fault mainly of our inflated expectations, rather than Twitter’s algorithms). But despite those kinds of setbacks, there is an emerging industry aimed at using the tweetstreams of millions of people to help predict the future in some way: disease outbreaks, financial markets, elections and even revolutions. According to new research released today by Topsy Labs — which runs one of the only real-time search engines that has access to Twitter historical data — watching those streams can provide a window into breaking news events. But can it predict what will happen?

The theory behind all of this Twitter-mining is that the network has become such a large-scale, real-time information delivery system (handling more than a quarter of a billion messages every day, according to CEO Dick Costolo at the recent Web 2.0 conference) that it should be possible to analyze those tweets and find patterns that produce some kind of collective intelligence about a topic. It’s the same idea that drives companies to do “data mining” on their customers’ behavior, or compels Google and Facebook to track your browsing activity in the hope that they can generate some aggregate information that will be of value, and predict what you might be interested in.

Predicting markets and the spread of disease

One of the first attempts at doing this with Twitter appeared last year, when a team of researchers published a report that looked at the predictive value of sentiment analysis extracted from Twitter (PDF link) compared to the movements of the Dow Jones Industrial Average. The study said that its system could predict the market index with 87-percent accuracy, and within months a hedge fund called Derwent Capital Markets launched a fund that it said would make stock and fund trades based on a similar kind of analysis of Twitter (so far it seems to be doing pretty well).

Medical researchers have also been trying to use Twitter trends and analysis to predict the outbreak or spread of disease, in much the same way that Google came up with Google Flu Trends, which tracks searches for terms associated with the flu — data that seems to correlate fairly well with actual outbreaks of the flu. Two researchers from Johns Hopkins University recently released a study that looked at more than two billion tweets and analyzed them for medical information, and said that this could be a useful tool for researchers and medical staff.

Could Twitter have predicted revolution in Egypt>

In one of the research reports the company released today, Topsy Labs looked at tweets related to the recent Arab Spring revolutions in Tunisia, Egypt and elsewhere in the Middle East, and tried to correlate the rising and falling trends in hashtags such as #iran, #egypt and #yemen with actual events such as the suicide of Mohammed Bouazizi in Egypt — the 26-year-old food vendor whose death crystallized for many dissidents the problems in their country and the need to take action. Twitter was a key tool for raising awareness of this revolution, and Topsy’s data shows that there was a high correlation between actual events and Twitter-related activity around those topics.

Topsy also looked at what it called the “share of voice” or influence and reach that one specific Twitter user gained over a short period of time: Sohaib Athar, the Pakistani programmer who live-tweeted the U.S. military raid on Osama bin Laden’s compound without even realizing it. According to Topsy’s data, when he first began posting, Athar had very little exposure — he wasn’t being followed or retweeted by many people, and those he was being followed by didn’t have much reach (meaning they weren’t followed by or retweeted by many people either). But that all changed over the next 24 hours:

[A]s his tweets were retweeted and mentioned more than 30,000 times, his exposure grew to a whopping 82.68 million unique tweets within 21 hours. As his tweets became more interestig to the Twittersphere, his exposure and influence grew dramatically. He went from 0 to 20 million in under 10 hours and over 82 million in just under 30 hours.

Topsy’s research certainly shows how quickly a single individual can become hugely influential in a very short space of time, and the correlation of Twitter data with events in Egypt and Tunisia is also interesting. But could someone have predicted that Egypt was going to break into open revolution based on the activity Topsy recorded? Perhaps — which is why the U.S. government’s Intelligence Advanced Research Projects Activity unit or IARPA is looking at using data from social media like Twitter and Facebook as part of its intelligence gathering.

The research that Topsy did is far from conclusive, however. In particular, the company didn’t apply any filters based on language or a Twitter user’s location to its analysis — which means that many of the tweets could have come from outside Egypt and Tunisia — and it didn’t try to use any influence-ranking to determine connections between those who were tweeting about the topic (as researcher Kovas Boguta did to produce this fascinating visualization). But it shows what could be done with that kind of data, and it is likely just the start of an ongoing attempt to understand the giant collective consciousness that is Twitter.

Post and thumbnail photos courtesy of Flickr users totalAldo and timetrax

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Readers weigh in: future prospects for Twitter
• How social discovery is transforming entertainment

It took months for the world to pick up on the plight of Amina Arraf, a lesbian blogger from Damascus who challenged the authorities — and her readers — to understand the troubles of Syria’s population. Her end, however, was decidedly swift.

After a series questions and investigations into her identity, it rapidly emerged that the whole thing was a hoax. In a posting published a short while ago, “Amina” was revealed to be a fictional character — the work of a Scottish educator an American grad student based in Edinburgh, Scotland, Tom MacMaster, who claimed that he had created the character in order to publicize the situation in Syria:

I never expected this level of attention. While the narrative voıce may have been fictional, the facts on thıs blog are true and not mısleading as to the situation on the ground. I do not believe that I have harmed anyone — I feel that I have created an important voice for issues that I feel strongly about.

I only hope that people pay as much attention to the people of the Middle East and their struggles in thıs year of revolutions. The events there are beıng shaped by the people living them on a daily basis. I have only tried to illuminate them for a western audience.

This experience has sadly only confirmed my feelings regarding the often superficial coverage of the Middle East and the pervasiveness of new forms of liberal Orientalism.

The truth is, however, that his confession came only after the net had already closed in, thanks to a series of links back to MacMaster and his wife, American campaigner Britta Froelicher. There had already been a number of skeptics by the time the first shoe really dropped, when the blog carried news of Arraf’s disappearance and arrest. The sudden flurry of media activity that followed led to the revelation that the photo of “Amina” was actually Jelena Lecic, a Croatian woman living in London. Alarm bells started ringing, though obviously there were plenty of reasons why Arraf might use a fake photograph (as well as, possibly, an obscured identity, though she claimed at times it was her real name).

But after a collective effort from a number of individuals, notably NPR’s Andy Carvin, San Francisco developer Liz Henry and Electronic Intifada’s Ali Abunimah, the truth came out. It also emerged that MacMaster and his wife were a pair of seasoned pro-Palestinian campaigners based largely in Edinburgh, Scotland.

The reaction so far has been a mixture of relief — that Amina was not really imprisoned after all — and anger; anger not only at being duped, but also because the hoax potentially endangers the lives and stories of real Syrians trying to fight the regime of Bashar al-Assad. The Electronic Freedom Foundation’s Jillian York, for example, who has written eloquently about her feelings on the situation, says she is ”furious”.

So what does this hoax tell us? What will it mean?

There will likely be some social and political repercussions. Actual members of Syria’s gay communities were concerned that they would be targeted by the authorities since the blog (and therefore their repressive tactics) had become so widely discussed. That doesn’t sit well with MacMaster’s statement that he does not believe he’s harmed anyone. At the same time, the political stance of both MacMaster and Froelicher will inevitably allow some political sites to paint the whole enterprise (and by association, concern for Syrians) as some anti-Israeli conspiracy.

And then, of course, there will be the inevitable hand-wringing. How did readers get fooled so easily? Why did nobody find this out before? How did news organizations get sucked in?

It’s likely that some will blame this on the mainstream media’s failure to out Arraf. Numerous outlets covered her story: CNN quoted her in an article and the Guardian also ran an interview with her. How could they not know? Was there a failure of process?

There have been plenty of high-profile failures in the news business over the years, from the Hitler Diaries to the Balloon Boy hoax. News organizations are looking for credible, compelling stories and Arraf’s was too interesting to ignore. The blog had been online for some years, telling a consistent story. Of course, confirming her identity was hard, but she conducted long conversations with people over IM and Skype and had credible reasons for staying a little under the radar (NPR’s Carvin asked the Guardian about their interview and it turned out it did not happen in person). This was a failure, obviously, since the hoax wasn’t spotted — but right now, it’s not clear how serious that failure was.

At the same time, some will likely suggest that it’s the online world’s fault for allowing her story to spread so far and so fast. Without the instant pass-it-along-and-don’t-check-the-facts nature of a service like Twitter, without the anyone-can-do-it nature of blogging, would we even be in this situation? Despite the fact that Twitter, blogging and the rest helped solve this puzzle, there’s not much to crow about. The online world has more than its share of hoaxes, and the fake blogger routine has been around for a long time.

I remember writing a story more than 10 years ago about a blogger called Kaycee Nicole, who purported to be an American teenager documenting her fight against leukaemia. She gained a significant following in the run-up to her death in 2001 — at which point, after some skeptics decided to investigate, it turned out to be a hoax. The culprit, a Debbie Swenson, had gone to extraordinary lengths to keep the fantasy going: online chats, photos, phone calls and so on. She received lots of good will and plenty of gifts from well-wishers — so much so, that the FBI ended up looking into the case over fraud allegations. Online hoaxes of this sort have been going on for years and will continue as long as people are trying to deceive the audience.

The truth is, whether it’s a fictional character like Amina Arraf or faked material like Hitler’s diaries or simply a sad fantasy like Kaycee Nicole, moments like this will always exist as long as there are rules for somebody to work around.

Attention is an incredible, addictive thing, and verification can sometimes be difficult. As long as some people want to deliberately deceive you, for whatever reason, and there are people who want to believe, then hoaxes will be hard to eradicate.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s IPO filing: ideas and implications
• Social 2013: The enterprise strikes back
• Examining the rise of crowd labor platforms in 2012