GigaOM » datasift

DataSift open-sources its social media analysis tool

Jordan Novet — Thu, 14 Feb 2013 13:00:51 +0000

DataSift is releasing an open-source version of its Query Builder service to work alongside enterprises’ existing business-intelligence software, allowing more employees to gain more insight from social media mentions.

The open-source presentation of Query Builder, which permits existing DataSift customers’ developers to simplify the tool’s appearance and functionality, might seem like a matter of crossing big data with even more data. But it’s an important step in trying to prompt business decisions based on what companies can learn about users of Twitter, Facebook and other outlets, not just see what people are saying. Social media analysis becomes more actionable and worthwhile with this sort of functionality.

Plenty of companies ask Twitter to filter out certain parts of its enormous data set. But DataSift is one of just two companies licensed to syndicate the firehose of all Twitter feeds. (The other is Gnip.) Its internet-based Query Builder service also allows customers to run natural-language processing off the entire Twitter firehose and adjust it on the fly in several ways. The processing requires a massive amount of storage, to the tune of 1.3 petabytes, said Nick Halstead, DataSift’s founder and chief technology officer. With the open-source versions, developers can add the Query Builder’s streams and processing to business-intelligence platforms, and users won’t even be able to tell it’s running in the background, Halstead said.

With Query Builder, which was announced in August, users can also pull in Amazon forum messages, YouTube comments, bitly links, Topix posts, Facebook status updates and other social statements, in addition to tweets. The data streams and insight on them all cost subscribing customers $3,000 or more per month. Those users will be able to use open-source versions and get more employees on board.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

As the firehose matures, Twitter tightens grip on valuable asset

Eliza Kern — Fri, 30 Nov 2012 00:39:01 +0000

This summer, Twitter showed that it had no problems playing favorites when it came to developers using its API to build a business. So when it comes to the company’s firehose – the unfiltered deluge of 1 billion tweets every two and half days — it’s becoming apparent the company will continue to play favorites there as well, limiting who has access to one of the company’s most valuable assets.

Twitter is currently being sued by PeopleBrowsr, a company that has had full access to Twitter’s firehose, after Twitter said it would restrict PeopleBrowsr’s access to the firehose following the expiration of a previous contract. From Twitter’s perspective, it makes sense to consolidate who is accessing and re-distributing that data due to the scope of the data and work involved in transmitting those tweets. But it hasn’t always been this way, and PeopleBrowsr is challenging Twitter’s earlier commitment to openness when it comes to that firehose of data.

In 2010, Twitter licensed its full firehose to both big guys like Bing, Microsoft, and Google, as well as several startups focused on realtime search, with then-Twitter spokesman Sean Garrett explaining that they wanted to build partnerships that were “sustainable and scalable.”

With access to the full Firehose of data, it is possible to move far beyond the Twitter experiences we know today. In fact, we’re pretty sure that some amazing innovation is possible. Today, we’re happily turning the Firehose on for some new partners focused mainly on exploring the incredibly rich field of real-time search and discovery. We are thrilled to announce that Ellerdale, Collecta, Kosmix, Scoopler, twazzup, CrowdEye, and Chainn Search join us as partners. These companies range from funded startups to part-time, one-person operations so we came up with a fair way to license access that scales with their business. If you think there may be a potential partnership involving access to the Firehose, let’s start a conversation.

Clearly, that conversation about access to the firehose has changed since 2010. A highly-public partnership with Google broke down, killing Google’s realtime search product that relied on the firehose of data. Twitter and Bing re-negotiated their terms in 2010, with AllThingsD reporting at the time that Twitter was asking about $30 million for the access. And most of the startups who recieved access in 2010 are no longer powering search engines: Ellerdale was acquired by Flipboard in July 2010, Collecta has removed their “customer-facing site,” Kosmix was acquired by WalMart in April 2011, Scoopler shut down after the team was acquired by Google to work on Google+ in 2011, and CrowdEye shut down after the founders said they were unable to build a profitable business.

So who’s still has access and distribution rights to the firehose? The full list of companies with commercial partnerships isn’t publicly available, but there are a few like Bing and Salesforce that have publicized this relationship, and several companies listed as part of Twitter’s certified program do have firehose access and can provide enterprise access to that data for other companies who are interested. Duncan Greatwood, CEO of Topsy, which is one of the companies licensed to re-distribute the data, said that including Topsy’s archives, there are now more than 250 billion tweets. That makes it unrealistic, in his opinion, that many companies would be prepared to access the full stream — most companies only access smaller percentages.

But PeopleBrowsr CEO Jodee Rich said his company relies on full access to the firehose to serve its customers, and an agreement with one of the other providers wouldn’t make sense.

“The nature of those agreements are so short-term that no one could possibly build a viable business on those agreements,” he said in an interview. In Rich’s legal statement, which is available online here, he notes that PeopleBrowsr was paying Twitter more than $1 million per year for firehose access.

Topsy, Gnip and DataSift are three providers who license the full firehose of data from Twitter and then re-sell portions of it to either developers building products with the data (showing your influence on Twitter, for instance), or to marketers and brands using sections of tweets to track conversations and trends on the network. PeopleBrowsr is arguing in the court documents that being forced to get data from one of these companies would compromise its ability to serve its clients, since it previously had full access. Twitter counters that it’s totally within its rights to change the terms given that its initial contract with PeopleBrowsr has run its course:

But as Twitter has grown, its contracting practices have matured. Where it once contracted directly with just a handful of data customers like PeopleBrowsr, it now has hundreds of data customers. In order to handle that broad commercial demand in a consistent and transparent manner Twitter has created channel resyndication partnerships with Gnip, DataSift, and Topsy. PeopleBrowsr is free to contract with any of them, just as its competitors do. What it is not free to do is insist that Twitter preserve forever its earlier business model, or continue to be bound by a contract that expired more than a year ago.

A Twitter spokesperson replied that, “We believe the case is without merit and will vigorously defend against it.”

PeopleBrowsr has won a restraining order that forces Twitter to continue firehose access for the time being. As we’ve learned this year, building your company on top of another company’s data is so inherently risky. But the lawsuit does demonstrate the value Twitter’s data has grown to hold, and the lengths companies will go to secure it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

DataSift teams with Bitly to show what social content really gets clicks

Ryan Kim — Thu, 15 Nov 2012 13:00:32 +0000

Social data platform DataSift, one of two companies with access to Twitter’s firehose, is teaming with link shortener Bitly to help its customers understand what content is actually clicked on and read, not just shared and retweeted.

Marketers spend a good chunk of their money on content marketing but when it comes to measuring how well that content is doing, it often comes down to counting tweets, retweets, posts, comments and likes. By tabulating friends and followers, marketers can estimate how much reach a message had. But it’s still harder to know what content is really getting clicked on and read the most.

By partnering with Bitly, which is used to share 80 million new links a day, DataSift can help its clients look at social conversations and click data together. That allows them to see what content actually drives the most interaction. This gives brands a way to create content that’s really engaging for users and also have a better way to measure their “return on influence.”

“Now, with Bitly, you don’t just look at links but you look at content. Now you can see who is engaging with stories and you can see which bits of news are resonating. Just because something was retweeted doesn’t mean someone clicked on it,” DataSift’s founder and CTO Nick Halstead told me in an interview. “This will change the perception of success.”

DataSift customers can now pull up live data on what stories are being shared and how many clicks they’re actually getting. They can filter by geography and see where the referrals are coming from and what keywords and titles are working. A company can now apply some SEO techniques to come up with new content that matches what people are clicking on or rework existing content and headlines to maximize engagement.

Halstead says the partnership makes sense for both companies. DataSift has a critical way to provide link analytics on social content while Bitly has a way to get more use out of its data and it can sell DataSift customers on its data stream. DataSift now has 300 big enterprise clients and just raised $15 million earlier this week led by Scale Ventures.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How to handle a firehose: An interview with DataSift’s CTO

Derrick Harris — Tue, 13 Nov 2012 17:40:26 +0000

DataSift, best known as one of the two companies with full access to Twitter’s firehose of streaming data, has raised $15 million in Series B. The new money, which comes from Scale Ventures — along with GRP Partners, IA Ventures Northgate Capital and Daher Capital — adds to the $14.7 million DataSift has previously raised, for a total of $29.7 million.

DataSift is more than just a collector of Twitter data, however. It also takes in data streams from dozens of other web and social media sources, and can handle corporate data, as well. And the company’s real value comes in the analytics it applies to all that data, letting users filter and correlate across myriad different factors.

Here’s an interview I did with DataSift Founder and CTO Nick Halstead during our Structure: Europe conference in Amsterdam last month, in which he describes how DataSift built an infrastructure capable of handling so much real-time data and how companies can use such data effectively.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Even tiny tweets can be big data

Stacey Higginbotham — Wed, 17 Oct 2012 12:00:54 +0000

Datasift, the UK company that has access to the Twitter firehose analyzes a petabyte of tweets and ships terabytes of insights around the world. And the infrastructure needs to keep up. The company has replaced its older networking gear with Arista switches to support its analytics operation, Datasift CEO and founder Nick Halstead said in a conversation with Derrick Harris at our Structure:Europe 2012.

Check out the rest of our Structure Europe 2012 live coverage here, and a video recording of the session follows below.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

7 reasons why Europe really matters to cloud computing

Derrick Harris — Fri, 05 Oct 2012 20:30:46 +0000

Cloud computing tends to be a very North America-centric topic, if only because so many of the biggest providers of cloud resources and services are based in the United States. That’s fair enough — the business side of things is very important — but other continents, particularly Europe, have a lot more to bring to the table than just seemingly restrictive data privacy laws.

We’ll discuss many of the finer points of European cloud computing and web infrastructure at our Structure: Europe event Oct. 16 and 17 in Amsterdam, both business and technological. To whet your appetite, though, here are seven reasons why Europe is a lot more important than many people might think.

1. Clean, innovative energy

Northern Europe is turning out to be a testing ground for some of the most-innovative data center designs around thanks to its cool temperatures and abundant supply of renewable energy. Most notably, there’s Google’s seawater-cooled facility in Finland, and Facebook’s hydropowered plant in Sweden. Iceland is popular too, hosting Verne Global’s 100-percent renewable converted NATO hangar, and home to GreenQloud — a cloud provider running wholly within Verne Global’s facility and another all-clean data center by Thor Datacenter.

2. Eastern Europe’s talent pool

Eastern Europe has become an important geography for the IT world in general, boasting a glut of programmers, software engineers and other workers with skills that today’s companies really need. Not only is the region a hotbed for outsourcing, but it’s also the home to a growing number entrepreneurs doing innovative cloud and big data startups. Among them is Nginx, the popular web server created and since commercialized by Moscow resident Igor Sysoev .

3. London’s financial center

There are probably many reasons Dublin, Ireland, is emerging as a data center hotspot — including acting as the European homebase for Amazon Web Services’ and Microsoft’s cloud businesses — and one of them is almost certainly to be as close as possible to London without having to deal with that city’s severe power shortage. But London is an important market, especially when it comes to the financial world, an industry that spends billions on IT each year and that cloud computing providers want desperately to tap into.

4. CERN

One really needn’t say more than “Tim Berners-Lee” when describing CERN’s influence on cloud computing, because the cloud as we know it wouldn’t be possible without the web. But CERN also created a little thing called the Large Hadron Collider, which aside from discovering the Higgs boson, is the impetus of possibly the world’s most-impressive research network — spanning dozens of countries and supercomputers worldwide and generating around a petabyte of data per second. CERN is a big supporter of open source cloud efforts, including OpenStack, OpenNebula and the Helix Nebula research cloud in Europe.

5. OpenNebula

Led by Spanish computer scientist Ignacio Llorente, OpenNebula is a fairly popular open source cloud platform that rivals the work being done, largely in the United States, by the OpenStack and CloudStack projects. The project has been around since 2005, and claims a handful of large companies and European research institutions as users. Although it doesn’t have the big-name backers of the other two, it should remain viable for a long time because of its rather large user base.

6. One-third of Twitter’s firehose

DataSift, which is headquartered in Reading, England, is one of three companies (along with Gnip and Topsy) certified to resell the all the billions of data points streaming from Twitter every day. Social media, and Twitter especially, are a huge focus of corporate analytics efforts, and anyone that can capture and analyze all the world’s tweets is a kind of a big deal.

7. Individual rights

American cloud providers might not always agree with the European Union and its individual countries when it comes to data governance laws, but every country’s citizens should pay European governments some respect. Their regulations on data privacy — such as the prohibition on Facebook’s facial-recognition feature in Germany — help keep privacy in the forefront of companies’ minds as we release more and more personal info to web-based services.

Privacy image courtesy of Flickr user striatic.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Data markets: in search of new business models

Paul Miller — Wed, 22 Aug 2012 06:55:34 +0000

From information on U.S. census returns to the location of every Starbucks in Canada, the demand for data to support decision making is increasing. Fittingly, a number of new data markets have emerged in the past few years that provide access to this data. A wide range of companies exists in this space, and often there are more differences than similarities in the various products on offer, not to mention the many different financial models. This report describes the basics of a data market, explores the ways in which various companies are beginning to position their offerings, and looks for evidence that there is sufficient demand for this market segment to prove sustainable.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Marketing is dead, long live marketing

Barb Darrow — Mon, 13 Aug 2012 19:15:04 +0000

Depending on what research you believe, chief marketing officers are either more powerful than they’ve ever been — or they’re on their way out.

Early this year,a Gartner analyst predicted that CMOs will have bigger IT budget power than CIOs by 2017. Naturally, that analysis was quickly bandied about by CMOs.

But last week, in a blog post unambiguously titled “Marketing is Dead,” Bill Lee wrote that CMOs, as a species, are under fire.

Lee, president of the Lee Consulting Group which focuses on “customer engagement,” cited data from a 2011 Fournaise Marketing Group study suggesting that CEOs don’t see ROI on marketing efforts and are sick of being asked for marketing money with no discernible payoff. On top of that, Lee posits that shoppers don’t pay attention to traditional marketing anymore. Ouch.

Cloud and big data reshape the marketing role

Underlying this seeming contradiction is that marketing is being redefined in the era of cloud-delivered, self-service applications and services and web-connected consumers. Several CIOs and CTOs have told me that they agree that CMOs gaining clout in their businesses — but the most successful CMOs are those who “get” that effective marketing is both broader and more focused than it’s been in the past.

“Broader” here means that the channels are no longer limited to radio, TV, print and online publications but social networks as well. CMOs who understand that the data flowing in via Twitter and Facebook is an important source of market intelligence — a big data feed that must be monitored and tapped.

Filtering the social networks

The explosion of social networking use means that “multi-channel” marketing is more multi-channel than ever. You don’t have to just track newspaper, TV and radio “thought leaders,” you need to watch for your company’s own thought leaders — your best customers and what they’re saying. That means narrower, less scattershot messaging — why hit up people who are not even remotely interested in your product or service? The idea is that your thought leaders will convince others that your offering is worth a look.

At the very least, you’d better know Gnip and DataSift.

Lee wrote:

Many firms spend lots of resources pursuing outside influencers who’ve gained following on the Web and through social media. A better approach is to find and cultivate customer influencers and give them something great to talk about.

There’s more insight here in a blog post by Evangelos Simoudis, senior managing director of Trident Capital. Simoudis wrote:

In order to become customer-centric and deliver a consistent message to each individual regardless of the communication channel, companies must first integrate all their customer- and prospect-related data. Up to now, organizations would silo the various types of customer-related data.

So whether a given CMO has clout or is about to get pink-slipped depends a lot on her ability to understand the importance of this data trove and capture and make use of that big data resource.

Feature image courtesy of Shutterstock user lznogood

Thumbnail photo courtesy of Shutterstock user Aaron Amat

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Social net sifter DataSift adds $7.2M to its war chest

Barb Darrow — Wed, 02 May 2012 11:00:50 +0000

The desire of companies to better understand the public perception of their products and reputation via social media shows no sign of abating. That’s why DataSift, the company that sorts through tons of Twitter, Facebook and other social network data, has garnered an additional $7.2 million in venture funding from existing backers GRP Partners and IA Ventures.

That money comes as a follow-on to the $6 million already raised in a Series A round last summer, and $1.5 million in seed funding. The new cash will be used for product development and to hire more engineers, customer service and sales reps, the company said. DataSift, which launched in the U.K. in 2010, opened a San Francisco office early this year.
DataSift and Gnip are the only companies authorized by Twitter to funnel and parse the Twitter firehose of social media data — information that many companies see as critical for planning service, promotion and marketing campaigns and then gauging their success (or lack thereof).

Companies from consumer electronics manufacturers to fast food chains all watch the Twitter feed — as well as Facebook, blogs, YouTube and other social network sources — collecting data that they then analyze along with their own internal metrics to see how online (or in-store) promotions are working. In early April, DataSift said it was also integrating with Newscred so that customers could study how news and perceptions spread in the media.

In a statement, DataSift CEO Robert Bailey said the company had not been actively seeking new financing but that GRP and IA Ventures came forward with this offer. The fact that they stepped up helps eliminate distraction that comes with raising new funding, he added.

Expect DataSift to use its a some of this money to add new data sources and services, as it and Gnip try to one-up each other. In February, DataSift announced its ability to sort through two years into Twitter data — Gnip can only go back 30 days. In mid April, Gnip announced an exclusive partnership with Tumblr, the fast-growing blogging site. It seems that arms race is bound to continue.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How Notion will spend $100m on Euro cloud startups

Bobbie Johnson — Tue, 17 Apr 2012 10:30:35 +0000

Good news for European cloud startups, as U.K.-based Notion Capital announced a $100 million fund for investing in software as a service companies across the continent.

Started by Brits Stephen Chandler and Jos White, who sold their messaging security company Messagelabs to Symantec for around $700 million in 2008, Notion says it has already deployed a fund of $50 million — and is now expanding with the backing of Britain’s Enterprise Capital Fund and the European Investment Fund.

And flying high on the IPO of social games company Zattikka, which Notion invested in two years ago, the venture firm says it’s ready to start putting that money to work.

I caught up with partners Chandler and White to find out more. Here’s what they told me:

The fund has a deliberate, narrow focus

“We focus on the B2B space — that’s where our pedigree is,” said Chandler. “We’re really excited about the opportunities. The consumer space can be exciting, but business are prepared to pay for technology, whereas consumers expect to get most things for free these days. In B2B it may take longer, but you can build a real business with revenues and customers.

White says that ideally they are looking for companies that can disrupt or digitize an existing area of business.

“We’re very interested in business process — that sounds boring, but take financial services like invoicing or currency exchange: these are services that have been traditionally delivered by banks and there is room to disrupt them. Marketing automation is also something we’re really interested in — something delivered via the internet rather than through an agency.”

And there are specific requirements

Many of the investments are likely to be in the series A range, around $2 million, but Notion is prepared to back companies with up to $15 million in series B and C rounds.

“We break it up between SaaS and infrastructure, and we’re more focused on SaaS because it’s our background and it’s less capital intensive,” said White. “We like to see recurring revenues, products that are available on a subscription basis, ideally with some customers who are quite sticky.”

But there will still be seed money available

… even though it won’t be the focus of the team’s investments.

“Our model is not to do hundreds of investments, it’s to work very closely with founders,” says Chandler. “And that strategy will continue. We do some seed investing, but it’s not film script stuff — we like to see working prototypes or very early products. Our focus is really on series A investing, but we still have an allocation for seed. “
The group

It’s not just public sector money

The announcement made play of the fact that there was money from the British and European governments going into the fund — but in fact there is actually much more private cash going in. That includes around $10 million from the Messagelabs founders themselves (their previous $50 million fund was entirely self-financed) and from a number of private individuals. However, said Chandler, the public money was very important.

“At the moment institutional investing is an important part of the ecosystem that’s pretty quiet,” he said. “It’s a challenging fundraising environment, and the public sector has done a good job of stepping into the gap.”

Big data, big exits

White gives a hint at the sort of investments Notion plans on making when he tells me he’d have liked to back DataSift, the U.K.-based service for interrogating Twitter.

“It’s a really interesting business, and big data is really interesting to us: being able to understand and interpret large amounts of data is really valuable,” says White. “We were disappointed that they felt they had to go to the U.S. for money — but we hope to do some deals like that ourselves.”

Part of the problem, they suggest, is that Europe isn’t necessarily hunting down big exits for companies — the sort of deal that they secured for Messagelabs, or valuable public flotations.

“The tendency in Europe is to be pre-occupied with protecting the downside — people are almost embarrassed about chasing big exits,” says White.

“And not cashing out too early,” adds Chandler. “European investors often take their money off the table too early on.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.