GigaOM » data mining

With Lucky Sort creators on board, Twitter is officially a data company

Derrick Harris — Mon, 13 May 2013 23:09:57 +0000

We all kind of knew that Twitter’s path to making money was paved with data, and the announcement on Monday that it’s buying analytics startup Lucky Sort makes it official. Unless I’m totally misreading the writing on the wall, this move is all about giving advertisers — and anyone, in theory — the tools to learn about what people are talking about.

Word that Lucky Sort is shutting down and that several of its team are joining Twitter’s revenue engineering department suggests this is exactly what the acquisition aims to accomplish.

As it stands, companies use Twitter as a way to track how people are talking about them and maybe, if they’re really advanced, do some sentiment analysis. If they’re willing to pay a third party, Datasift and Gnip are more than happy to broaden marketers’ views to encompass the entirety of Twitter’s data, both real-time and historical. What companies really can’t do, though, is run their own advanced analytics about topics straight from the Twitter platform.

One view of the Lucky Sort dashboard

The value proposition from such a product should be obvious at this point. Facebook, Google and Yahoo all collect a lot of data about how people are using their platforms and what topics are trending, and they all offer it up via a variety of products targeting marketing types and the public at large. If Twitter wants to be taken seriously as a venue for advertising budgets and a platform for measuring the pulse of the nation, people need to be able to ask questions of its data without relying on an intermediary or the occasional Twitter blog post.

As a journalist, I’d love to have access to this type of tool to track trending topics in real time and spot possible stories as they’re happening. The appeal to marketers should be obvious. As IBM’s Erick Brethenoux told me recently, “[Marketers] talk a good game about social data. Very few actually leverage it effectively today.”

At Twitter, though, data is a slightly different beast than at other web companies. Twitter’s value lies largely in real-time data — topics can be peak, crest and all but vanish within a 48-hour window. This situation has hampered some of Twitter’s efforts to surface optimal search results, and it has spurred the decision to buy companies such as Backtype (for its streaming-processing Storm technology) and parallel-processing startup Ubalo.

The latter move, which happened last week, should help Twitter’s development team create new features without worrying about the intricacies of making them run — and run fast — across a cluster of machines. (You can learn a lot more about how companies such as Google, Facebook and Box are rethinking infrastructure to handle their unique data needs at our Structure conference next month in San Francisco.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Researchers have created a 21st century global mood ring with data mining

Amanda Alvarez — Tue, 30 Apr 2013 16:00:26 +0000

After your morning stock market and weather updates, maybe add a check of the hedonometer to your list. The new site draws on tweets — and soon, the New York Times, Google Trends, and other sources of textual sentiment — to gauge population-level happiness. This big data approach is taking the collective mood temperature across space and time, but it’s unlikely to reveal the secret to achieving happiness.

Launching today and updated every 24 hours with faster refresh rates to come, Hedonometer.org uses English-language tweets to create a happiness index. The system is based on a 10,000-word strong “emotional temperature” database, where words are ranked on a scale of 1-9 by volunteers using Amazon’s Mechanical Turk. Words like “laughter,” “happiness,” and “love” top the list, while “loneliness,” “bad,” “inflation,” and “surgery,” along with assorted expletives, round out the bottom, with rankings close to 1. The emoticon “:(“ has a rating of 2.36.

Users can zoom in on any day all the way back to September 10, 2008, check out the balance of positive and negative words, and see how these compare to the week before and after. Saturdays, for example, tend to be happier than Tuesdays. Christmas Day stands out as being the happiest day of the year, every year. The hedonometer developers, mathematicians from the University of Vermont along with scientists from the MITRE Corporation, found that April 15, the day of the Boston bombings, was the unhappiest day on record, with an average happiness index of 5.88. Other recent sad days include December 14 last year (Newtown school shooting) and June 25, 2009 (death of Michael Jackson).

Indeed, eyeballing the global happiness index suggests a slight downward slope since 2008. Whether or not this effect is real depends on establishing a normal background happiness level, and comparison with geographic, socioeconomic, and political metrics. What’s interesting is that the hedonometer is turning more than 50 million daily micro-statements into a “quantitative macro-story,” as UV’s Chris Danforth put it. Individually insignificant words and tweets swell into a collective emotional response, the blips and dips of which stand out and correlate with major events.

The research from Danforth and his colleagues got some press earlier this month, when they reported that happiness went up the further Twitter users were from home. Other insights from the same team included the fact that obesity and happiness were inversely correlated, and that cities’ happiness scores were related to swear words, suggesting that “geoprofanity” could be a good marker for regional happiness differences.

The hedonometer is set to draw on more data streams soon, including blogs, news transcripts, and Bit.ly shortened links, and will be data mining in a dozen languages. Nonetheless, the happiness index will remain an aggregate measure, like a nation’s GDP, and may not have much impact in and of itself. The underlying methodology, however, is the real driver, with broad applicability to big data, whether social media-generated or not.

Image via Chris Danforth, University of Vermont

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

OK, now I’m convinced Facebook is trying to be creepy

Derrick Harris — Thu, 11 Apr 2013 18:48:10 +0000

What if Facebook is just trolling us, carrying out the biggest social experiment in history by testing the limits of how much privacy-invading creepiness we’ll take before we actually quit using it?

I ask because Facebook’s newest attempt to boost ad revenues — it’s “partner categories” program — seems too dunderheaded a move to be real. A company that’s regularly lambasted and has even been sanctioned by the government for privacy indiscretions is now going to let advertisers place ads based on what users have purchased offline, or at least off of Facebook? And, worse yet, based on public records such as the type of car someone drives?

This type of advertising happens all the time, of course, but I just can’t believe it’s for real from Facebook. Is it oblivious to public opinion about its privacy record (a recent study, for example, ranked it 42nd in overall reputation, far behind Google, which is 4th, and even behind Verizon and AT&T)? Or how ineffective this type af advertising might be, especially on a platform where people are trying to interact rather than look at ads for things they already buy or have bought?

But here’s the catch: If Facebook wants more people to click on ads and still doesn’t want to be called a creep, it probably isn’t doing enough data mining. I think some of the best ideas we’ve covered have to do with intent — that is, serving up ads that are in line with what users are actually expecting, or at least receptive to, from the data they’re already giving Facebook. Surely, Facebook’s highly talented data scientists are aware of these methods.

Intent-based targeting might be a bit creepy, but it’s not blunt-force creepy like a peeping tom staring in your bedroom window. It’s more like a cute co-worker whose pickup lines are so on-target you know he’s been researching you, but you’re in the mood to go on a date and he’s there and speaking your language, so …

Solariat Founder and CEO Jeffrey Davitz explained the concept to me last year as mining “big, sucky data” in order to put ads in front of users at the right times on the right topics. It’s a 180-degree difference from showing someone a sponsored story every time a friend “likes” something, placing sidebar ads based on someone’s stated — and static — interests, or even the new partner categories method of advertising for things people might have already purchased and might not want Facebook to know about.

Here’s how I described it then:

Davitz thinks there’s a way for social platforms to overcome this problem by using techniques such as natural-language processing and machine learning to identify those instances where users really are expressing “query-like intent.” It will never be as clear as entering “best hiking shoes” into a search engine, but, for example, someone certainly might note in a wall post or a tweet that he’s going hiking and needs new shoes. He might specifically ask friends which shoes they prefer. If you sell hiking shoes, there’s your signal. Rather than simply peppering someone’s page with ads about hiking because he listed it as an interest, now he’s actually in the market for gear and might pay attention.

This approach could help get Facebook ads heard above the noise that’s surrounding users and coming from their intended purpose for visiting Facebook — social interaction — as well as external sources like Twitter, email and text messages. And if timed right, Davitz noted, users might notice the utility rather than the creepiness. An ad for hiking boots that comes hours later, for example, might be more like a guy gracefully playing the rebound rather than asking a woman out the second he heard she broke up with her boyfriend.

It’s some extra work, yes, but a little nuance might make Facebook seem like it’s not just trying to push our privacy buttons as much as it can before we crack.

NYU Stern School of Business professor Arun Sundararajan nicely summed up the risks of ignoring user intent leading up to Facebook’s IPO last year. He compared annoying — and possibly offending — users with ads at every possible turn with playing the stock market and only thinking about making money. “[I]f you’re investing in the stock market and you’re only thinking about returns and not risk,” he said, “at some point you’re going to lose your shirt.”

Feature image courtesy of Shutterstock user Steven Frame.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

At Eventbrite, the challenge is with realtime data

Eliza Kern — Thu, 21 Mar 2013 20:01:43 +0000

Eventbrite has become one of the leading ticket sales platforms, facilitating events across the world from concerts to festivals to meet-ups. But beneath the exterior site, there’s a lot of data flowing through the company’s platform, and it’s not easy keeping up with the realtime data challenges that come with the company’s growth.

Vipul Sharma, director of data engineering for Eventbrite, spoke at GigaOM’s Structure:Data conference in New York on Thursday about the data challenges facing his company, which fields customers and data from across the globe. By June 2012 the company had crossed $1 billion in ticket sales, and Sharma said Eventbrite has hosted more than a million events so far.

Sharma,who has worked as an engineer since 2010 after leaving Digg, where he worked on data mining and machine learning, said one of the biggest data challenges Eventbrite deals with is the speed of the flow of information:

“So basically what I’m trying to tell you is that the challenge in solving realtime is two-fold. One is the data flow, or how do I transfer the data I have in realtime to my downstream applications,” he said. “And the second part is processing, or how do I process this significant amount of data in realtime. To process in realtime you really need a distributed processing system.”

Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Researchers mine 2.5M news articles to prove what we already know

Derrick Harris — Tue, 27 Nov 2012 02:54:38 +0000

A group of British researchers has published the results of a data mining experiment that analyzed nearly 2.5 million articles from 498 newspapers on criteria such as topic selection, writing style and sensationalism, and found — no surprise — that tabloids are the easiest to read and reporters don’t often cover women’s sports. If these findings sound predictable, that was exactly what the researchers were aiming for.

The experiment’s techniques actually point to a future where researchers are spared the grunt work of poring through thousands of pages of news or watching hundreds of hours of programming, and can actually focus their energy of explaining. As the researchers note in their paper, the real ramifications of this research lie more in what it accomplished than in what it found.

Namely, they demonstrated that with new big data techniques such as machine learning and natural-language processing, it’s possible to accurately analyze millions of pieces of content spanning almost a year without requiring humans to read and score it all. Choosing hypotheses with predictable results meant it was easier to verify their accuracy.

Here’s how how they explain the promise of their work and some potential use cases, the latter of which they go into far more detail about in the paper:

“[I]t allows researchers to focus their attention on a scale far beyond the sample sizes of traditional forms of content analysis. Rather than spending precious labour on the coding phase of raw data, analysts could focus on designing experiments and comparisons to test their hypotheses, leaving to computers the task of finding all articles of a given topic, measuring various features of their content such as their readability, use of certain forms of language, sources etc. (just a few of the tasks that can now be automated).

… Our approach — apart from freeing scholars from more mundane tasks — allows researchers to turn their attention to higher level properties of global news content, and to begin to explore the features of what has become a vast, multi-dimensional communications system.”

Put more simply: This research underscores the common big data maxim that knowing the right questions to ask is now the biggest challenge in gleaning insights from data. It’s increasingly easy to get data, analyze it and visualize it, so humans really just need to hypothesize and be able to explain the results. (This also seems like a good place to plug ScraperWiki as a great source for gathering potential research data from websites.)

Creating the workflows for gathering and analyzing the data as the authors suggest still isn’t child’s play (it might take some assistance from the computer science department), but it’s a lot better than the alternative.

Feature image courtesy of Shutterstock user Ruggiero Scardigno.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

If data is the new oil, don’t end up being BP

Mon, 19 Nov 2012 12:00:43 +0000

In the run-up to the US election, stats nerds and nervous Democrats sought and found reassurance in Nate Silver’s aggregate polling model, spiking traffic at the New York Times but it was Harper Reed, CTO of the Obama 2012 re-election campaign, who really taught us a lesson about the value of personal data in politics.

In the most sophisticated (and expensive) campaign season in history, Harper and his team provided the highly targeted information that allowed the Democratic machine to reach potential donors and optimize the get-out-the-vote efforts across just about every communication channel possible. It’s probably not an exaggeration to say that their analytics work changed the course of the election, giving Obama the edge over Mitt Romney and his SuperPAC donors …and four more years in the White House.

No matter what your politics, the power of personal data in the 2012 election season was undeniable. And all over the world, c-level executives are salivating as they think about they will use the same quantitative analysis and personal data mining techniques to supercharge growth and revenue.

Just about every would-be pundit has been telling us this year that “Data is the new Oil”. You can practically see them scratching off “Social Media Guru” from their business cards and replacing it with “Data Scientist”.

But without considering the ethical and best practices of personal data collection, you are more likely to end up as the Exxon Valdez or with a BP-sized sized disaster on your hands. If you’re serious about using customer personal data effectively in your business, build your data policies on solid ground from the beginning and focus on three key concepts: permission, control, and transparency.

Permission

Always get explicit, opt-in permission to use personal data. Opt-out models are deceptive and slimy. Only ask for the data you need to make your service work better for your customers. Storing more personal data than you actually use just opens you up to risk from malicious hackers or unscrupulous employees.

Think twice about using personalized data when aggregate, non-identifiable data could just as effective — especially for advertising purposes. And while the temptation might come up to sell customer data to third parties when revenues are tight, it is never worth it. Selling your customers to the highest bidder destroys the trust you established with every great interaction with your product.

Control

Start off with a simple rule of thumb: You don’t own your customer’s personal data: they do. Increasingly, consumers know that the data they give you is valuable currency and when they deposit it with you, they expect a return on their investment. Make it as easy to view and change personal data or even remove it entirely as you did to collect it. Be clear on how each data field is used in your service and let customers decide what should be private and what should be public. Don’t make it hard for them to walk away from your service whenever they want.

Transparency

Be honest about how you use personal data in your services and explain this as you collect it. Educating people about what you do with their information and the benefits they get from depositing it with you creates better informed and more loyal customers. Don’t write privacy policies in nine-point type or obscure them by burying them deep in your services dark corners.

And don’t listen to anyone who tells you that we’re in a post-privacy society. It’s as important as ever to people and when the inevitable privacy screw-up happens, how transparent you are about what happened and how you are fixing the problem will be more important than the incident itself.

By focusing your policies and day-to-day communications on these concepts, you’ll build a more valuable asset with your customers: Trust. It’s more valuable than oil and much harder to refine. It’s what your reputation is based on and if you lose it, your customers go with it.

Matthew Hawn is product development and strategy consultant based in London, and previously VP of product at Last.fm

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Alliance Health Networks tackles the low-hanging fruit of medicine and data

Stacey Higginbotham — Fri, 05 Oct 2012 21:22:33 +0000

The combination of information technology and medicine has led to hundreds of startups and another round of excitement for modernizing the way we diagnose patients, track illness and even administer care. There are efforts to build a tricorder, plans to track millions of people’s heartbeats to create a data set for cardiac care and even remote intensive-care units that help keep people safer in the emergency room.

But outside of some whiz-bang technical stuff, there are also hundreds of million of people with access to the internet and a desire to get more information about something as esoteric as PHACE syndrome or as common as depression. Add thousands of medical research papers and descriptions about diseases and treatments available via the National Library of Medicine for free, and you have a business opportunity.

Making meaning from medical research

When Jay Bartot and Derek Streat decided started Medify in 2010 the goal was to use those free research papers to train a machine learning algorithms how to deliver intelligible information to health queries from the masses. Bartot, who had co-founded Farecast, a startup that built a predictive algorithm to tell users the best time to buy airplane tickets, decided to take his knowledge of prediction to the health world ~~after his own family’s brush with a medical problem~~.

In May, Medify was purchased by Alliance Health Networks a Salt Lake City, Utah-based startup that has built out a community of 1.5 million people who gather to discuss diseases and medical conditions. Now, with Medify on board and a community of people whose discussions about health are also a great source of data, Alliance Health is seeing how mining unstructured data from professionals and patients alike can help improve heath.

Medify had used the National Library of Medicine to build out ontologies that it uses to “teach” the algorithms to understand medical terminology and treatment plans, and has then built up a user interface around those algorithms. Like IBM has found with Watson, its supercomputer that has found a role helping doctors diagnose illnesses based on symptoms, medicine is a good place for this type of data mining.

The goal is to take the ontologies learned from Medify and combine its algorithms with what people discuss in Alliance Health’s communities. Then, Alliance can apply new algorithms to see who in the community is offering the best advice, understand how patients influence and inspire each other, and then help pharmaceutical companies and even doctors understand and influence how patients make medical decisions.

Getting value from big data doesn’t have to be a big undertaking.

It’s actually a great example of how thinking about big data doesn’t have to be as complicated as using a supercomputer and expensive clinical research filter through algorithms to help doctors diagnose illnesses. Streat says the company’s data is only in the low-terabyte range and they process it using Amazon Web Services, including EC2 and Elastic MapReduce. They add new data every day and refresh their machine learning algorithms weekly, if not every few days.

There’s a place for these simpler solutions, and by bringing together a community and providing it with information, Alliance Health might become a company much like Spiceworks is in the IT space — able to both monetize and help a community of niche users in a way that benefits everyone. For example, a company that makes a new diabetes test might pay to sponsor the diabetes channel on Alliance or may even pay to find out who the big influencers are in the forums associated with that channel. If done correctly, users might even welcome sponsored how-tos or better information delivered about a new drug or device.

Streat explained that as far as medicine and data-combining go, there are many efforts around devices and even fancier data sets. However, he’s confident that even with something like expensive clinical data that’s locked behind paywalls, just being able to direct people to better answers and give them a sense of community is a good place to start.

Investors seem to think it’s a decent bet as well. Alliance Health has raised $20 million since it’s founding in 2006 from investors such as New World Ventures, Physic Ventures, Epic Ventures and Highway 12 Ventures.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

RelateIQ: A hush, hush data-savvy startup

Stacey Higginbotham — Thu, 30 Aug 2012 20:15:14 +0000

There are a thousand, maybe more hot startups in Silicon Valley at any one moment, but one name that’s popped up a few times this summer has been RelateIQ, a startup supposedly doing next-generation customer relationship management with some very data-savvy founders and investors.

The company’s web site is fairly bare, but the pitch to folks has been that RelateIQ will track your digital life and then suggest actions to take and people to call. I’ve been told by investors that I should think of it as the next generation of personal assistant technology based on machine learning that I’ve been waiting for. Those investors, including Accel, Morgenthaler and SV Angel, have put at least $1.25 million in funding into the company.

Because it probably wants to sell its service, RelateIQ plans to target enterprises as a next-generation CRM play as opposed to helping normal people remember to pick up the milk on the way home after seeing an email exchange between spouses. DJ Patil (former LinkedIn data scientist) is involved and CTO and co-founder Adam Evans is the former CTO of Palantir Health. Palantir is the secretive Silicon Valley data mining startup that’s growing like crazy.

And a quick scan of LinkedIn profiles associated with the startup indicate that data mining and machine learning are two big areas where it has made recent hires. That fits if RelateIQ really hopes to build software that’s intelligent enough to scan your inboxes and/or social contacts to understand when and how often you should reach out to people. Top sales people already track this sort of thing, but having a computer analyze when that might be most effective may make a good salesman into a great one.

The company recently had its trademark approved and says in the application that it’s building:

Software as a service (SAAS) services featuring software in the field of customer, business and personal correspondence management for sharing, organizing, recording, reporting, charting and analyzing all of a user’s digital signals, namely, digital and electronic correspondence, emails, instant messages, calendar items, social media postings, phone calls by landline, cell and VoIP, voice messages, text messages and communications through a program’s Application Programming Interface.

I can’t wait to learn more when the company launches.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Mendeley injects some pace into academia with fast, big data

Bobbie Johnson — Mon, 06 Aug 2012 10:30:39 +0000

Just look around at the likes of Evernote, Springpad, Lawloop and DocTrackr and it should be obvious: managing documents online is a pretty busy space right now.

But there’s one service that has quietly been making waves — and now it’s about to stage a real revolution.

London-based Mendeley doesn’t get a lot of press because it focuses on a very specific part of the market: the academic world. But it’s a big, valuable market worth billions of dollars each year, and one that’s ripe for disruption. And the site has become a big hit with academics and researchers, signing up nearly 2 million members from universities and institutions all over the world, because it allows them to keep tabs on all the research papers, documents and files.

Now, thanks to that popularity, the startup is rolling out its first big data product — and it’s a doozy.

Mendeley Institutional Edition, announced on Monday, is a new data dashboard that takes all of the activity on the site and presents it to universities as a way of gaining deeper insights into what their researchers are doing.

“The biggest problem in academia is the long waiting time: it can take three to five years from the time you have done research to get it published — all the decisions you make in an academic career are based around that time lag,” Victor Henning, Mendeley’s co-founder and CEO told me.

“We’ve developed a product that’s packaged into a data dashboard and allows universities to see what’s happening right now: what are the journals they’re reading? What are they not reading?”

It’s not just about optimizing efficiency by dropping unread journal subscriptions, or watching which areas are growing fast. The service can also let universities see the other side: which members of their faculty are publishing most? Who’s being cited? What areas are they active in? Those are things that institutions care deeply about — but struggle to find right now because most data out there is old data.

The system is built on the back of data that’s constantly streaming in from the service’s 1.8 million users. Between them, the site’s members have uploaded some 260 million documents, representing around 65 million unique research papers and studies — around 50 percent larger than any of the existing commercial databases. By mining those documents and watching activity around them, Mendeley’s able to help institutions understand the trends as they emerge… not years afterwards.

In fact, some universities see this effort as so vital that it’s already signed up its first seven partners, including the Harvard-Smithsonian Center for Astrophysics and the Korea Advanced Institute of Science and Technology. And plans on rolling out the product to more places soon, through a distribution partnership with Dutch library giant Swets.

“Our data is now helping some of the world’s best universities work more efficiently and get to life-changing discoveries faster,” said Henning. “My inner nerd is going: Wow, this is freaking amazing.”

There are a number of companies trying to work in this area, including social networks like ResearchGate and document management services like Digital Science. But the biggest players are academic publisher Reed Elsevier (see disclosure) which has been subject to a boycott over its profiteering tactics recently), and the information giant Thomson Reuters.

Thomson, specifically, publishes its Impact Factor — a rating of how well-received or well-cited a paper is. But that also suffers from a lag. In the end, Henning says, he’d like to replace that rating with one based on immediate, real-time data uncovered through Mendeley — something he thinks would be hard to compete with.

“The real incumbents who have the best chance of dominating this landscape are Thomson Reuters and Reed Elsevier,” he said. “But killing us is difficult because they’d have to replicate what we do. We’ve built up a big user base over the past three years and I think you really have to understand how to do that if you want to compete with us.”

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in GigaOmniMedia, the company that publishes GigaOM

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Social-TV apps and consumer behavior

Nicole — Tue, 17 Jul 2012 06:55:18 +0000

A new category of digital media has emerged in the living room: social TV. This relatively new concept can be defined as any application, website or software that allows viewers to interact with television programming and share that interaction with others. Currently social TV occurs in three ways: organically, as pure play or through TV- or set-top-enabled communication. This report will focus on the pure-play aspect of social TV as it relates to content providers, television networks and advertisers. It answers key questions relating to the segment’s growth potential, important companies, their competitors and likely business models for the future.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.