Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Even as some news outlets like Associated Press hire social-media editors to try and figure out how to make use of tools like Twitter for journalistic purposes, others seem to be intent on locking these tools down and removing as much of the social aspects from them as possible. According to a report in The Guardian, broadcaster Sky News has come out with a new policy that bars reporters from posting anything other than work-related content on Twitter, prevents them from breaking news through the service — and even forbids them from retweeting anything that doesn’t come from a Sky News account. As with so many other similar social-media policies, this completely misses the point of what makes Twitter so powerful.

Although it doesn’t link to an actual document, the Guardian story quotes from the Sky News guidelines, which tell reporters not to tweet about stories if they are not “a story to which you have been assigned or a beat which you work,” and says that anything approaching breaking news must be sent to a Sky editor first before being posted. The policy says that retweeting other Sky journalists is fine — provided they are posting updates about a story to which they have been assigned — but it says Sky staff are forbidden from retweeting anything that hasn’t been posted by a Sky News account:

Do not retweet information posted by other journalists or people on Twitter. Such information could be wrong and has not been through the Sky News editorial process.

Twitter is the newswire now, for better or worse

This is even more draconian than the most recent example of a news outlet trying to lock down Twitter use — namely, the Associated Press newswire, which came out with standards for retweeting that not only mis-stated how the process works on Twitter, but also forbade journalists working for the newswire from retweeting anything without adding a comment to make it clear that they were not agreeing with the person being retweeted. The AP rules also strictly forbid breaking news on Twitter, which ignores the fact (as I pointed out at the time) that for many people the real-time information network has become the newswire.

Since then, AP has hired Eric Carvin to be the service’s social-media editor (Carvin is the brother of National Public Radio’s Twitter phenom Andy Carvin, who turned his Twitter account into a one-man newswire during the Arab Spring revolutions). At a recent social-media event in New York, Eric told me that he was trying hard to convince the wire service that the benefits of social tools like Twitter outweigh the disadvantages. But as with so many traditional media outlets, both AP and Sky chose to focus their policies on what their staff shouldn’t do, instead of concentrating on what they should do.

As we’ve pointed out before, these kinds of rules seem to be aimed at trying to remove the human being from the process, something that may work in traditional forms of media, but fails miserably when using social tools like Twitter. The whole point of using them is to be social, and that means expressing human emotions and possibly even opinions in some cases. The best social-media policies — like the exceptionally minimalist version that Media News CEO John Paton came up with — simply ask reporters and editors to be themselves, but to think about what they post before doing so, and to use common sense and “don’t be stupid.”

Why remove the social from social media?

Sky News says in the email it sent to employees that the guidelines were necessary to ensure that “there is sufficient editorial control of stories reported by Sky News journalists and that the news desks remain the central hub for information.” And obviously, a news service doesn’t want dozens of reporters tweeting rumors and innuendo about major breaking stories, or tipping competitors off to a scoop. But banning staff from retweeting anyone outside the Sky News operation makes no sense whatsoever, as Charlie Beckett of the London School of Economics notes — Sky reporters should be seen as the key sources for information, regardless of where it comes from.

During the raid on Osama bin Laden’s compound, New York Times reporter Brian Stelter was the first to broach the rumor — on Twitter — that the terrorist leader had been killed, when he retweeted a post from the former chief of staff for Defence Minister Donald Rumsfeld. Some wondered whether Stelter would get in trouble from the Times for retweeting something that hadn’t been confirmed, and for posting it before his own newspaper. But as far as I know, there were no repercussions — and Stelter’s tweet in turn was retweeted thousands of times, and likely broke the news to many.

That’s what Twitter can accomplish if you use it properly, instead of seeing nothing but threats and potential negative repercussions. Like other media outlets that have tried the same approach, Sky News risks removing all the benefits of a powerful media tool by treating its staff as though they were disobedient children. Elana Zak of 10,000 Words has a Storify roundup of some other responses to the Sky News policy.

Post and thumbnail photos courtesy of Flickr users and Rosaura Ochoa

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Frenemy mine: The pros and cons of social partnerships for online media companies
• Content monetization: News licensing and syndication still need marketplaces and infrastructure

If Google and Twitter were to describe their relationship in one word, it would probably be “complicated.” For the past week or so, the two have been sniping at each other about Google’s new social-search features, and how Twitter doesn’t show up as high as it should in those results — thanks to what it sees as favoritism of Google’s own Google+ network. But this particular brouhaha is only the latest manifestation of a much deeper problem between the two, like a fight over the toothpaste, or who did the laundry last. Both sides need each other more than they would probably like to admit.

When Google launched its new “Search plus Your World,” which the search giant claimed would give users a view of what their social networks were recommending and sharing, Twitter was among the first to point out that all Google was really doing was promoting its own social network in search. Twitter said it was “disappointed” in the move, and suggested Google wasn’t fulfilling its chosen role as an impartial search provider, and Twitter’s general counsel (and former Googler) Alex Macgillivray went even further and said the search company’s move was “a bad day for the Internet.”

Not one to take criticism lying down, Google responded with a somewhat passive-aggressive statement about how it would love to show more Twitter results, but was obeying the “rel=nofollow” rules laid down by Twitter (which are designed to prevent Google from assigning page-rank value to certain links as part of its indexing process). The search company also pointed out that Twitter was the one that broke off the previous deal between the two which gave Google access to the full “firehose” of Twitter data, which formed the basis of Google’s short-lived real-time search offering.

Google: “It’s your fault.” Twitter: “No it’s your fault”

This week, Twitter came back with its own argument, and stuck a thumb in Google’s eye to boot: Developers with the company collaborated with Facebook Director of Product Blake Ross on a browser plugin called “Don’t be evil,” which is designed to show what Google’s search results would look like if the search giant gave content from Twitter and Facebook the prominence it deserves, instead of favoring Google+ results.

In a lot of ways, listening to Google and Twitter feels like watching a divorced couple fighting in court over who gets custody of the kids. And while neither side wants to go into detail about what’s keeping them apart, or what the root of their problems are, there are clues there to be found: For example, Google is clearly miffed it spent so much time developing its real-time search based on Twitter’s firehose feed, only to have Twitter pull out of the deal and leave it hanging. According to several sources, the breaking point in that discussion was that Twitter wanted more money for access to its data.

Twitter, meanwhile, keeps pointing out that Google can and does index its content without any kind of special access. Twitter’s communications team noted Google hits its servers more than 120 million times a day, and Search Engine Land’s Danny Sullivan has described how there is plenty of content from Twitter in Google’s results. But the real issue is more complicated: According to some observers, including Rakesh Agrawal, Google can’t index all the content that streams through Twitter in real time without special access, because with 250 million tweets a day or so, there’s just too much of it.

Google could crawl Twitter more aggressively and more often, these observers say, but that would cost more time, money and bandwidth — and on Twitter’s side of the coin, if Google were to crawl more aggressively, it could impact the network by slowing it down or even causing it to crash, which Twitter definitely doesn’t want. Having raised almost a half billion dollars in financing last year at a valuation of $8 billion, the last thing the company wants is to have the “fail whale” start popping up because Google is hammering away at its servers trying to catch up with all the new content.

Google and Twitter both need each other

As with most troubled relationships, the saddest part of this whole situation is that Google and Twitter really need each other, and in many ways they should be the perfect couple: Twitter has a huge and rapidly-growing information network, but it has no real search function to speak of — or at least not one that works very well. Indexing and searching 250 million tweets a day is not a small problem. Google, of course, is an expert at making sense of huge quantities of data, and it also needs more social signals in order to improve its search. That’s why it started Google+ in the first place.

Theoretically the two have plenty to offer each other, and plenty to gain from a better relationship — which is why Google has reportedly tried to acquire the company in the past. But Twitter seems determined to build a standalone entity, and appears to be heading towards an IPO rather than an acquisition — and a market valuation of $8 billion or so makes it a rather large mouthful, even for Google. And so we have a classic standoff, in which neither side wants to admit that it needs or wants what the other one has to offer.

Is there some kind of relationship counsellor who could fix this broken couple? No one seems to be stepping up to offer their services.So users wind up with no functional Twitter search, and Google results that are one-sided to the point of being distorted, which as I’ve pointed out before is a breach of the search company’s promise to users when it went public in 2004, not to mention a red flag for antitrust regulators. In other words — as with so many dysfunctional relationships — no one wins.

Post and thumbnail photos courtesy of Flickr users fPat Murray and Stefan

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• Social media in Q1: commerce and discovery dominated
• NewNet 2012: companies and technologies set to disrupt

“I meant to send a direct message, but I ended up sending a public @ message to someone,” he laughs. “Suddenly there were people who were following us both who had signed up and were saying it looked interesting.”

That early moment of calamity turned out to be a blessing, however. The initial interest was strong enough to encourage the team to keep developing and now, just a couple of years later, the business is gaining plaudits — and customers.

So what is Pusher? Co-founder and CEO Max Williams describes the service as a set of tools that make it simple to add real-time functions to other websites or services. And at heart, that’s it: an API that lets people offload some potentially tricky work.

“We allow developer to make applications real-time, so that users don’t have to refresh the page, and they automatically get information streamed to them where they are,” he explains. “But it’s a flexible enough service that it can be used for all sorts of things.”

Indeed it is, pulling in clients as diverse as MailChimp, Slideshare and The U.S. Open tennis. Gaming companies are latching on to Pusher to provide multiplayer experiences without having to build out lots of infrastructure or admin, and the business is constantly expanding the ways for people to hook into its REST API.

Building this sort of thing could be achieved in-house, of course — and many web apps currently do it for themselves — but Pusher hopes that the idea of handing off the server and administration load to somebody else will appeal because it allows developers to make and deploy real-time elements quickly and at scale and focus on their product.

“If you have 10,000 people looking at a page, waiting for some sports scores or tweets, you send one message to us and we relay it to those 10,000 people,” says Tanner. “Previously you’d have had an Ajax thing requesting new information from the server every second or every five seconds — and if you have 10,000 people doing that at once it’s actually a scaling challenge. We make it so much easier by having a websocket connection with the browser, which is just a pipe we can just push data down.”

Like so many projects, Pusher was initially developed to scratch their own itch. When the pair’s last business, web consultancy New Bamboo, started building its own products, the team discovered it was coming across one particular issue time and again.

“We ran into the same problem — synchronization across browsers,” says Williams. “Someone’s messing around changing things, but those changes aren’t reflected on somebody’s else’s browsers… so then they start to get out of sync.”

“We built what would become Pusher to solve that problem — but when we came around to implementing it, it only took a few hours. We suddenly thought that to have this problem solved in a few hours is actually a much more interesting thing than the original applications we built. So we put them to one side and started working on getting Pusher out.”

And it was good timing. Growing the real-time web is hot right now: not just through the explosion activity around Twitter over the last few years, but also through the growth in mobile apps and the spread of into businesses. And while some may feel ambivalent about the data deluge it creates, there’s no doubt that many more companies are looking at ways to speed up what they do. And if that’s the case, then providing real-time as a service could be lucrative — at least that’s the feeling of Pusher’s investors, including London-based Passion Capital and the founders of cloud app platform Heroku, who gave the company $1 million in funding last summer.

40 billion messages and counting

Six months on, Williams and Tanner are cagey about Pusher’s numbers, but say that the service now has 10,000 registered users, hundreds of active ones and some high-profile partnerships — and is running close to break-even. And with 40 billion messages delivered, they are hoping that it can become a fundamental foundation on which hundreds or thousands of new real-time businesses can be built.

“A lot of people have their existing infrastructure or their existing applications, and they’re not going to rebuild the whole thing today,” says Tanner. “So a lot of people are adding Pusher on to the side or as an element to add some real-time parts.

“But there are people exploring building fully real-time web apps, where all the communication works over WebSockets. As we move into the future, we see that being the primary method of communication between a service and the browser. You’re still going always going to send audio and video through HTML or streaming methods, but for interacting with web applications, WebSockets is the ultimate technology to use.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• What Enterprise Software Vendors Could Learn from the Consumer Space
• Sector RoadMap: Social customer service in 2013

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Social customer service in 2013
• NewNet Q4: Platform mania and social commerce shakeout
• Can Mining and Filtering Monetize NewNet?

The new year brought a treat for those who like to follow aging media moguls, with the launch of official Twitter accounts belonging to both News Corp. chairman Rupert Murdoch and his wife Wendi Deng, including some awkward banter around a tweet that Murdoch later reportedly deleted (although as a commenter notes below, the original tweet remains). The only problem with the voyeuristic appeal of this exchange is that Deng wasn’t the real thing — although the account was marked as “verified,” with Twitter’s blue check mark, it was revealed to be a fake on Tuesday. A simple slip-up? Perhaps, but one that reinforces how little we know about Twitter’s verification process, something that is becoming more and more important as the service grows.

When Murdoch showed up on Twitter on December 31, there was widespread skepticism about whether it was the real News Corp. billionaire or not, despite the fact that the account was marked as verified. But a tweet from Twitter co-founder and chief product officer Jack Dorsey confirmed that it was the real Murdoch — and the “verified” check-mark, combined with the apparent back-and-forth between the Wendi Deng account and Murdoch’s, convinced many that it was also real (although some, including publishing industry veteran Michael Wolff, continued to doubt this).

How was the account verified? We don’t know

On Tuesday, however, it emerged that the Wendi Deng account had been set up as a prank by a British man, who said he “set up the account for a laugh” during the holidays, when he saw how much attention the Murdoch account was getting. The account’s creator said that he was as surprised as anyone when his account showed up with a blue check-mark, and that he hadn’t been contacted by anyone at Twitter about who he was or whether the account was for real, telling the Guardian:

I just couldn’t believe they would have verified such a high profile account without checking it out, but I absolutely received no communication from Twitter to the email address I used to register.

Twitter has refused to speak publicly about what happened with the Deng account, or to explain why it was verified and then suddenly un-verified — and the company has also repeatedly refused to talk on the record about how the verification process as a whole works, and why some accounts are chosen for verification and others aren’t. Even if the Deng verification was a simple screw-up due to reduced staffing levels over the holidays, Twitter’s radio silence on the issue makes it even harder to trust the entire process, and that could have ramifications that go beyond just the Murdoch case.

The “verified” program started with the blue check mark as a beta in 2009, primarily because a number of celebrities had complained about fake accounts pretending to be them, and the company said it wanted to help users figure out which were real. For a time, anyone could apply to have their account verified by using a form on the Twitter website, but this was later phased out and verification is now done on what the company calls a “case by case” basis, including advertisers and partners.

Twitter needs to be more transparent about the process

Given the rapid growth in Twitter’s user base, it’s not surprising that Twitter would have problems scaling a widespread verification program — and in some ways, doing this runs against the grain for the network, which has made a point of not requiring real names from users the way that Facebook and Google+ have. But even worse than having an arbitrary verification process is having one that doesn’t work properly, and one that the company is so opaque about. It’s not clear why Twitter doesn’t talk about it, but this vacuum of information is hardly conducive to gaining the trust of users.

And trust is something that Twitter needs in spades, especially as it grows and becomes a crucial part of the way that news and other information spreads in a social-media age. The network is already in a delicate situation when it comes to issues like free speech, with the State Department pressuring it to shut down accounts that belong (or appear to belong) to terrorist organizations, and other lobby groups launching legal claims against the company because it allegedly supports entities like Hezbollah by giving them a platform.

The company’s refusal to provide more details about how the verification process functions may stem in part from its desire to protect the users it is verifying, or to prevent the system from being gamed somehow. But if it is going to continue to ask for the trust of its users, it is going to have to be more transparent about how it manages the network, or risk losing the faith that it has spent so much time building up.

Post and thumbnail photos courtesy of Flickr users Hans Gerwitz and See-ming Lee

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• Why the next front in big data might be psychological
• NewNet 2012: companies and technologies set to disrupt

We’ve written many times about how journalism is changing in the age of social media, thanks to what Om has called the “democracy of distribution” provided by tools like Twitter — and how everyone now has the opportunity to function as a journalist, even for a short time, during news events like the attack on Osama bin Laden’s compound. A new study of the way information flowed during the Arab Spring uprisings in Tunisia and Egypt earlier this year paints a fascinating picture of how what some call “news as a process” works, and the roles bloggers, mainstream media and other actors play during a breaking news event. More than anything, it’s a portrait of what the news looks like now.

The study, entitled “The Revolutions Were Tweeted: Information Flows During the 2011 Tunisian and Egyptian Revolutions,” was published in the International Journal of Communications, and involved several researchers from the Web Ecology Project, Gilad Lotan from the social-media service Social Flow, and Microsoft researcher and sociologist Danah Boyd. (A PDF version of the study is available here.) The researchers looked at two datasets: one composed of 168,000 tweets from January 12 to 19 that contained hashtags such as #sidibouzid and #tunisia, and one composed of 230,000 tweets from January 24 to 29, containing hashtags such as #egypt or #jan25 (the date of a mass demonstration that played a key role in the subsequent Egyptian revolution).

The research broke those who tweeted about both events down into a number of groups of “key actors” — including activists, mainstream media outlets, individual journalists, bloggers, digerati and celebrities — and then tracked how information about various events during both periods flowed from one source to another. One interesting aspect of the study is that some key players in both events were almost impossible to classify as belonging to a single group. Jillian York, for example, is a researcher who works for the Electronic Frontier Foundation but is also a prominent blogger for Global Voices and is passionate about issues in the Arab world.

Twitter becomes a crowdsourced newswire

As the study describes, Twitter has come to play a crucial role in the way that news functions during events like the Egyptian revolution — like a crowdsourced newswire filled with everything from breaking news to rumor and everything in between, and one that both uses and is used by mainstream media:

The shift from an era of broadcast mass media to one of networked digital media has altered both information flows and the nature of news work… during unplanned or critical world events such as the Tunisian and Egyptian uprisings, MSM turn to Twitter, both to learn from on-the-ground sources, and to rapidly distribute updates.

The evolution of what media theorist Jeff Jarvis and others have called “networked journalism” has made the business of news much more chaotic, since it now consists of thousands of voices instead of just a few prominent ones who happen to have the tools to make themselves heard. If there is a growth area in media, it is in the field of “curated news,” where real-time filters like NPR’s Andy Carvin or the BBC’s user-generated-content desk verify and re-distribute the news that comes in from tens of thousands of sources, and use tools like Storify to present a coherent picture of what is happening on the ground.

The study makes the point that mainstream media outlets play a key role in the dissemination of news during such events (and also notes that journalists tend to retweet other journalists more often than they do non-mainstream sources), but it also makes it obvious that prominent bloggers and activists are crucial information conduits as well. In graphic representations created by Global Voices using the study’s data, for example, blogger Nasser Wedaddy is a key hub that distributes information to bloggers, activists and mainstream media. (Here’s another fascinating visualization of networked data flows in Egypt during the revolution in February.)

It’s called social media for a reason

As noted by Nancy Messieh at The Next Web, one of the additional points the study makes is that the personal Twitter accounts belonging to journalists were far more likely to be retweeted or engaged with by others than official accounts for the media outlets they worked for. The point here is one we have tried to make repeatedly: Social media is called social for a reason. It’s about human beings connecting with other human beings around an event, and the more that media outlets try to stifle the human aspect of these tools — through repressive social-media policies, for example — the less likely they will be to benefit from using them.

Another benefit of a distributed or networked version of journalism is one sociologist Zeynep Tufekci has made in the course of her research into how Twitter and other social tools affected the events in Tunisia, Egypt and elsewhere. As she wrote in a recent blog post, one of the realities of mainstream media is what is often called “pack journalism”: the kind that sees hundreds of journalists show up for official briefings by government or military sources, but few pursue their own stories outside the official sphere. Social media and “citizen journalism,” Tufekci says, can be a powerful antidote to this kind of process, and that’s fundamentally a positive force for journalism.

As we look at the way news and information flows in this new world of social networks, and what Andy Carvin has called “random acts of journalism” by those who may not even see themselves as journalists, it’s easy to get distracted by how chaotic the process seems, and how difficult it is to separate the signal from the noise. But more information is better — even if it requires new skills on the part of journalists when it comes to filtering that information — and journalism, as Jay Rosen has pointed out, tends to get better when more people do it.

Post and thumbnail photos courtesy of Flickr user Petteri Sulonen

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• Content monetization: News licensing and syndication still need marketplaces and infrastructure
• Sector RoadMap: Social customer service in 2013

But a New York company is trying to breathe new life into e-mail marketing by making e-mails real-time and context aware. Movable Ink offers advertisers and publishers a way to push out e-mails that remain current whenever a user opens it instead of existing e-mails that become increasingly irrelevant and stale the longer time goes on.

Movable Ink does this by allowing clients to pull real-time content from their websites and easily embed that information into their e-mails. So whenever someone opens a message, it contains the most up-to-date information as well as contextual data based on a person’s location, time of day or the device they’re viewing the e-mail on. And it can reflect changes that have happened throughout the day.

“E-mail isn’t going away but it should be as real-time as the web,” said Vivek Sharma, co-founder and CEO of Movable Ink. “We’re bring e-mail up to par with all the interesting stuff on the web.”

This can be helpful for daily deal sites, marketers, publishers or anyone with an e-mail relationship with a consumer. For example, a daily deal site can provide a countdown clock within an e-mail and direct people to alternative discounts when one deal sells out. Publishers can include the latest news in a message. Event sites or ticketing services can show a live calendar, how many seats are currently available or who’s RSVPed.

GroupMe, for instance, uses Movable Ink to show the latest conversations in a group chat when new members are joining a group. Daily Candy includes live tweets in its e-mails. Double Cross Vodka includes a map showing the closest place to buy the alcohol. Sharma said daily deal sites are seeing a 33 percent lift on click-through rates and some clients are seeing up to 120 percent increases in click throughs using Movable Ink.

Users not only get the most up-to-date information, but publishers can also engage in A/B testing to see what messages or campaigns are working and update their campaigns on the fly. Sharma said customers can implement Movable Ink within a few minutes and get access to a dashboard tool to build and control their campaigns. Or for more serious developers, they can tap Movable Ink’s API for deeper integration. Sharma said his company has done a lot of back-end work building essentially a custom app server that can deliver real-time messages to any e-mail client.

Movable Ink CEO and co-founder Vivek Sharma

Sharma and co-founder Michael Nutt came up with the idea last year after trying to start a syndicated e-commerce company. Sharma said he realized the bigger opportunity was in fixing e-mail, which is an increasingly important channel for companies to stay in touch with customers. He said while some start-ups have improved the data and delivery of e-mail, there was still an opportunity in attacking the design side.

“It was just an a-ha moment because no one’s really innovating in the e-mail space,” he said. “It turns out real-time content delivery is a real problem and we’re talking to some large e-mail providers to make this is a part of their products.”

It seems like Movable Ink would be a ripe acquisition target. Or perhaps some e-mail companies might look to replicate the technology. But Sharma believes Movable Ink can become a standalone business and is more than just one feature. He’s hoping to host a gallery of real-time content widgets that clients can drop into their e-mails. Movable Ink offers a free version and a pro version that costs $49 a month. The company raised a seed round of $260,000 in March.

I think this makes sense to make e-mails more interactive. Google has done some work to make YouTube videos embeddable in e-mails, but there’s more that can be done to make e-mails much more relevant and current. This could be significant news for services like Groupon, which is trying to get people to interact directly with its website and apps but still relies a lot on e-mails to alert people about deals. Many people sign-up for e-mail alerts but fatigue can set in over time. But if the e-mails are more valuable and current, even if a user doesn’t open it right away, it can still be an important way for a marketer to keep that relationship going with a consumer.

It reminds me of PlayHaven, a mobile marketing provider that I wrote about last week, which makes mobile ads real-time and interactive. They’re showing that there’s still opportunity to bring the dynamism of the web to more products.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• Call it Real-Time, Squared, or NewNet, The Web Is Changing
• Survey: How apps can solve photo management

1. Cloud MapReduce is focused on real-time analysis of streaming data
2. Appistry customers now have access to an entirely distributed Hadoop alternative. Earlier this year, the company released its Hadoop Distributed File System (HDFS) alternative called CloudIQ Storage Hadoop Edition.

As I explained in a post on that product, Appistry’s primary goal in developing these products is to improve performance and reliability by eliminating single points of failure. In HDFS, that’s the NameNode; in the Hadoop MapReduce engine, that’s the JobTracker. Running atop Appistry’s CloudIQ platform, Cloud MapReduce takes advantage of a peer-to-peer architecture in which these concerns are largely ameliorated.

Appistry’s Sam Charrington said those were the same issues that led Accenture to develop Cloud MapReduce in the first place, as its customers wanted higher reliability and performance for mission-critical jobs. As it turns out, however, certain users in intelligence and defense weren’t too keen on the cloud-based model. Jointly developed and distributed by both companies, on-premise Cloud MapReduce keeps the same focus on bleeding-edge customers in intelligence, defense and financial services.

Then there are the real-time capabilities. Cloud MapReduce utilizes a streaming API that frees it from the batch-processing boundaries typically associated with MapReduce. As Charrington explained, Hadoop’s popularity has shaped many connotations of MapReduce, but “the algorithm can be applied much more broadly.”Cloud MapReduce also leverages existing CloudIQ capabilities, such as Fabric Accessible Memory, a form of in-memory caching to speed data processing. “It’s not a competitor to Hadoop,” he added, “so much as an alternative to other approaches for processing data streams [such as IBM InfoSphere Streams].” In fact, Appistry retains partnerships within the Hadoop ecosystem so that customers have a choice of options depending on their applications.

In terms of scope, Cloud MapReduce appears to be in the same vein as the S4 project that Yahoo open-sourced last week. Once described as “real-time MapReduce,” the project website now describes S4 as a “distributed stream computing platform” that “fills the gap between complex proprietary systems and batch-oriented open source computing platforms.” According to a research paper (PDF), S4 was inspired by MapReduce but more closely resembles the Actors model. Like Cloud MapReduce, S4 is wholly decentralized to improve reliability and performance.

Both Cloud MapReduce and S4 should catch on (S4 likely sooner because it’s an open source project, not a paid product), but it might take time. In the case of Cloud MapReduce, many organizations with Big Data problems are still experimenting with Hadoop for batch-processing, and might not be ready to take on writing parallel-processing applications for real-time data. Even Charrington acknowledges that Appistry’s two products might be unnecessary for Hadoop experimentation or R&D projects, but are designed for mission-critical production applications that require real-time analysis. And there aren’t too many of those around right now.

Aside from relatively low-hanging fruit like fraud detection and instant search, it will be fascinating to see the applications for these types of technologies once organizations are able to wrap their minds around the full scope of their data situations. You can bet social-media analysis will be an early priority, but that’s just the tip of the iceberg. Just ask IBM, which is beating the real-time drum with its Smarter Planet initiative.

Image courtesy of Flickr user jpctalbot.

Related content from GigaOM Pro (sub req’d):

• What IBM Does With Big Data
• The Incredible, Growing, Commercial Hadoop Market
• Will the Real-Time Web Bring High Performance to a System Near You?

Inside Yahoo Labs, S4 is being used for “[a]pplications such as personalization, user feedback, malicious traffic detection, and real-time search.” The project website gives a high-level description of how S4 works:

In S4, we abstract the input data as streams of key-value pairs that arrive asynchronously and are dispatched intelligently to processing nodes that produce data sets of output key-value pairs. In search, for example, the output data sets are made available to the serving system before a user executes her next search query. We use this rapid feedback to adapt the search models based on user intent.

The S4 wiki provides more detailed information on the project, and code is available at github.

S4 should become a hot commodity among the community of MapReduce – particularly Hadoop – developers. Just as it has with certain tools developed for Yahoo’s Hadoop distribution, it seems likely Cloudera would incorporate S4 into its Hadoop distribution, which has established itself as solid choice among enterprise users. Perhaps Karmasphere, which sells a platform for developing Hadoop applications, will take up the S4 cause. Either way, S4 represents a free- to low-cost alternative presently available proprietary real-time processing options like multiple IBM InfoSphere products and SAP’s new in-memory HANA appliance.

The recent analytics landgrab illustrates just how hungry customers are to derive insights from their personal data deluges. Churning through streaming data is probably still a ways out for many organizations, but having the tools to actually do it should help catalyze a few efforts. Workloads like those suggested by Yahoo for S4 bring enough value to make it at least worth a try.

To learn more about deploying the right cloud strategy for your needs, attend the free GigaOM Pro webinar, The Scalable Cloud. The webinar takes place at 10:00 a.m. PST on Nov. 4.

Image courtesy of Flickr user bdu.

Related content from GigaOM Pro (sub req’d):

• With Caffeine, Google Reveals Challenges of Real Time
• What IBM Does with Big Data
• Will the Real-Time Web Bring High Performance to a System Near You?