That’s why a group of researchers from Columbia University and Stanford University developed an algorithm that can do just that, turning those cells into something that resembles your social graph. This lets researchers see how the various cells are related to each other so they know , for example, where to focus cancer treatment and what to track as that treatment progresses.

The idea of representing large or complex data as a graph is nothing new, but it has taken on more prominence thanks to the rise of social media and those ubiquitous social graphs that map out who’s connected to whom. As we highlighted recently, however, graph analysis is becoming more popular outside the realm of social networks, and is being applied to problems that are more complex than just figuring out simple relationships within a network. In cases such as medical research, especially, graphs can provide a very effective way of seeing how potentially hundreds of thousands of data points spanning perhaps hundreds of variables are similar to each other.

That’s exactly what the team at Columbia and Stanford has done with a new algorithm that they’ve demonstrated within the realm of mass cytometry. According to a press release announcing the research (which is available via paid download at Nature Biotechnology):

“The method, called viSNE (visual interactive Stochastic Neighbor Embedding), is based on a sophisticated algorithm that translates high-dimensional data (e.g., a dataset that includes many different simultaneous measurements from single cells) into visual representations similar to two-dimensional ‘scatter plots’ ….

“The viSNE software can analyze measurements of dozens of molecular markers. In the two-dimensional maps that result, the distance between points represents the degree of similarity between single cells. The maps can reveal clearly defined groups of cells with distinct behaviors (e.g., drug resistance) even if they are only a tiny fraction of the total population. This should enable the design of ways to physically isolate and study these cell subpopulations in the laboratory.”

I assume they say similar to scatter plots because the algorithm is analyzing data across more than two dimensions, although the resulting chart is essentially the same (i.e., data points with similar characteristics will form clusters).

The results of viSNE, showing cell densities in diagnosis and relapse samples.

Whether or not they’re technically similar, this research seems similar to what Ayasdi is doing with its new data-analysis software based on a technique called topological data analysis. In both cases, though, the algorithms aren’t necessarily concerned with how data points interact with one another (like in network graphs), but rather what similar characteristics the points share. Ayasdi’s software has been used in cancer research, too, including on datasets spanning hundreds of patients and tens of thousands of variables.

In theory — although not likely in practice considering the complexity of the datasets medical researchers are dealing with — these approaches are similar to clustering approaches that are also popular among data scientists working with web companies. In areas such as e-commerce or email management, for example, where there isn’t a strong social element, companies can broadly break customers into distinct groups based on their behavior or interests.

A sample cluster of MailChimp subscribers.

Of course, curing cancer is a slightly more compelling — and difficult — goal than targeted advertising. The algorithms have to be precise so as not to miss similarities hidden within the mass of data. In the case of viSNE, the researchers say they’ve been able to spot small groups of cells (like 20 out of tens of thousands) that might be able to survive chemotherapy and increase the likelihood of a recurring tumor.

But we probably shouldn’t bee too quick to discount the work that web companies do as somehow less valuable than that of cancers researchers, for example. The big data era arguably started with the web, and web companies have generated some of the most important data-analysis techniques and technologies around today (see, for example, Google’s Jeff Dean, with whom I’ll be speaking at our Structure conference next month). As medical researchers start generating more and more data via cytometry, genome sequencing and even electronic medical records, it will be critical for individuals in all fields to keep track of what data scientists in other fields are doing and figure out how that might apply to their own work.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout

President and COO George Mathew says the company’s mission is to be a one-stop shop for statistical analysis. It wants to be the one place where analysts and data scientists can blend their data, model it on it and then visualize it. Often, he noted, that same process might require two or three separate products.

Another feature that Alteryx hopes will set it apart is its collection of prebuilt models in what the company calls an analytics gallery. Users can share their own work or find models others have built for tackling similar issues. Alteryx also offers up its own pre-formatted datasets for analysis, often public data such as the U.S. census.

“The canvas for creating an analytics application should never be blank for the analyst when they’re getting started,” Mathew explained. They often need to understand external data as well as their internal data, so Alteryx’s software gives them easy access to it.

Because it’s based on the R statistical-programming language, heavy R user Walmart has been able to transition some workloads to Alteryx when employees need an easier user experience. McDonald’s uses it to analyze data about franchisees and about its growth strategy in China, and Bloomin’ Brands (parent of company of Outback Steakhouse and other restaurants) is using it to help build menus that take into account what diners in various parts of the country prefer to eat. Nine of the 10 leading top wireless providers providers are also users, Mathew said, trying to blend actual call data with traditional sources such as customer service data.

Mathew compares Alteryx’s current growth as analogous to software-as-a-service applications like Salesforce.com in the CRM space, or even Tableau in the traditional business-intelligence space. In a business world increasingly driven by at least the idea of big data, one might expect any vendor pushing a more consumer-like purchase and consumption experience to get interest from companies tired of dealing with legacy software or never wanting to experience it in the first place.

“The disruption that’s happening is creating a new space for ourselves,” Mathew said, “without having to go head to head, frankly, with the a status quo out there.”

Feature image courtesy of Shutterstock user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Internet of things: creating tomorrow’s health care
• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution

“We’re thrilled,” Tableau co-founder and CEO Christian Chabot told me during a call after the market closed. One should hope so.

Chabot and his fellow co-founders stand to make a lot of money if today’s closing price holds up, as does its sole investor NEA. The firm put $15 million into Tableau since it launched in 2003, and has rode that sum to profitability and more than $127 million in annual revenue.

Here’s a quick chart (made using Tableau Public) showing who owns how many share and what they’re potentially worth.

The company didn’t really need more capital to operate, Chabot said, but one of the primary drivers was to raise awareness of the company. It has about 12,000 customers, he said, but there are millions more possible users. As part of attracting them, the company is going to expand globally and is working to improve its reach across mobile devices, the cloud and the Mac operating system.

“I don’t believe in the this whole ‘or’ philosophy with computers,” Chabot said. “It’s ‘and’” — meaning people will use desktops and tablets and smartphones.

More prominence and more users singing its praises might also dispel the notion that Tableau is just about visualization. It has some fairly advanced features under the covers (as a commenter to my earlier post about the company’s influence pointed out), even if they’re hidden by the relatively simple user experience.

“Tableau is not a visualization company, per se, it’s really an analytics company,” Chabot said.

However, if the company really wants to expand its reach to everyone one who wants to gain knowledge from data — something Chabot calls a “timeless human need” — it might actually need to get simpler. More marketing can let potential business users know about new features like forecasting and data-extraction, but it won’t make a dentist is Des Moines better at formatting his data.

After raising $254 million in its IPO, though, Tableau is in a good place to do whatever it has to.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• The importance of putting the U and I in visualization

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

That’s an apt ticker symbol for a company that is in some ways a bellwether for the current fascination with all things data. Tableau isn’t a big data company, per se, but its visualization software breathes life into many big data calculations. Its focus on making software that’s easy to use and that creates visually captivating charts has turned people from numerous professions into amateur data analysts. (I’ve even used it in the past, including for the first time in 2011.)

Christian Chabot

As Chabot told me during a conversation in 2011, “In any field of human endeavor … there are a hundred to a thousand more people who understand the data of that field more than they understand reporting and analytics.”

Anytime you read about a hot new visualization or analytics startup promising the moon, you’re also seeing the results of what Tableau has sown in terms of the user experience. Many of those same companies will be quick to tell you how limited Tableau’s capabilities are. It’s memory-bound, it doesn’t have a database, it’s not available in the cloud (or on the Mac operating system), it can’t do predictive analytics. All true.

Of course, if it raises the kind of capital it expects to by going public, it can build and buy a lot of those capabilities. If pricing stays flat all day Friday, Tableau stands to make $155 million from its 5 million shares. Previous estimates had Tableau’s market cap at around $1.7 billion at a price of $29 per share (the company’s S-1 filing is available here).

If investors have really bought into the company and the concept of a data-driven world, then who knows. Machine-data expert Splunk wnet public in 2012, flying the big data banner, and saw shares peak at 91 percent above its original asking price of $17.

I’m not suggesting Tableau is the biggest name in data, or even that it will some day become it. This next-generation analytics field is very young, with startups and larger vendors alike sometimes competing against themselves to win wholly new accounts than trying to displace legacy vendors within large enterprises. And every month, it seems, I come across some new startup that was built with the same principles in mind as Tableau, but with the advantage of having today’s best practices baked into its software.

But Tableau definitely commands a lot of the mindshare. How it fares as a public company could be a strong indicator of just how powerful the data movement is, and how well it capitalizes on a new influx of cash will determine how long it stays on the top of customers’ minds.

This post was updated at 7:01 p.m. to include previous estimates of the company’s market capitalization and a link to its S-1 filing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 4 iPad apps to help wrangle data
• Health care and big data in 2012

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

Word that Lucky Sort is shutting down and that several of its team are joining Twitter’s revenue engineering department suggests this is exactly what the acquisition aims to accomplish.

As it stands, companies use Twitter as a way to track how people are talking about them and maybe, if they’re really advanced, do some sentiment analysis. If they’re willing to pay a third party, Datasift and Gnip are more than happy to broaden marketers’ views to encompass the entirety of Twitter’s data, both real-time and historical. What companies really can’t do, though, is run their own advanced analytics about topics straight from the Twitter platform.

One view of the Lucky Sort dashboard

The value proposition from such a product should be obvious at this point. Facebook, Google and Yahoo all collect a lot of data about how people are using their platforms and what topics are trending, and they all offer it up via a variety of products targeting marketing types and the public at large. If Twitter wants to be taken seriously as a venue for advertising budgets and a platform for measuring the pulse of the nation, people need to be able to ask questions of its data without relying on an intermediary or the occasional Twitter blog post.

As a journalist, I’d love to have access to this type of tool to track trending topics in real time and spot possible stories as they’re happening. The appeal to marketers should be obvious. As IBM’s Erick Brethenoux told me recently, “[Marketers] talk a good game about social data. Very few actually leverage it effectively today.”

At Twitter, though, data is a slightly different beast than at other web companies. Twitter’s value lies largely in real-time data — topics can be peak, crest and all but vanish within a 48-hour window. This situation has hampered some of Twitter’s efforts to surface optimal search results, and it has spurred the decision to buy companies such as Backtype (for its streaming-processing Storm technology) and parallel-processing startup Ubalo.

The latter move, which happened last week, should help Twitter’s development team create new features without worrying about the intricacies of making them run — and run fast — across a cluster of machines. (You can learn a lot more about how companies such as Google, Facebook and Box are rethinking infrastructure to handle their unique data needs at our Structure conference next month in San Francisco.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Finding the Value in Social Media Data
• Listening platforms: finding the value in social media data

Storage, databases, query processing and algorithms are all extremely important — heck, visualization is next to nothing without them — but in a data-driven world where is obsessed with insights, they’re just the foundational layers. They are to big data what server and network configurations are to mobile-app development on platforms like Parse. If you’re going to find out new things from massive and highly complex data sets, or going to give new types of people the ability to analyze even simple data, the presentation of that data and the ability to create consumable presentations are critical.

With that in mind, here are six startups I’ve seen trying to fundamentally change the way that data is visualized. Some are highly complex under the covers, some are not and none are perfect, but they’re all doing their part to make us rethink what it means to look at data and make spreadsheets and static charts look like relics. (And this list is by no means exhaustive, so feel free to add your favorite visualization tools in the comments.) We’ll be highlighting data visualization at our design-focused RoadMap conference in San Francisco in November (sign up here to get first access to tickets this Summer).

Ayasdi

The idea of network graphs isn’t new, but Ayasdi’s approach to it is. Under the covers, there’s an HBase data store, a technique called topographical topological data analysis and hundreds of machine learning algorithms to churn through complex data sets and determine the similarity among the data points. To the end user, though, there’s a map of the data set that looks a lot like a network graph (only it’s probably not network data) highlighting clusters of related data points that analysts might want to investigate further.

BeyondCORE

BeyondCore actually operates under the same basic premise as Ayasdi — show users the significant correlations so they don’t have to think of the queries that will uncover them — but it uses some different techniques to get there. It uses a different visualization method, too: BeyondCore sticks to standard charts, but actually offers the option of having an avatar talk users through the correlations the software has discovered.

ClearStory

ClearStory has a pretty unique product in the works — even if it’s keeping many details and all of its screenshots under lock and key until its formally launches. Essentially, though, it’s trying to tell stories via visualizations that display mashups of numerous data sources, update automatically when the source data changes, and invoke collaboration and social concepts. Here’s Co-founder and CEO Sharmila Mulligan explaining the idea behind ClearStory at Structure: Data in March.

Datahero

Unlike so many data startups, Datahero isn’t trying to woo people fed up with business-intelligence software or the difficulties of getting insights from Hadoop data. Rather, it’s trying to let people with simple business or personal data make simple charts without ever having to enter an Excel function or worry too much about how their spreadsheets are formatted. Early on, Datahero’s visualizations are still pretty commonplace (bars, pies, plots, etc.), but it’s the ease of creating them that’s so unique.

Platfora

Platfora has undertaken the ambitious task of trying to make analyzing mountains of data stored in Hadoop clusters as easy as analyzing their own Stripe data might be for developers using Datahero. It’s based on a foundation of Hadoop and massively parallel query processing, but is presented like an HTML5 version of current visualization golden boy Tableau that’s all about dragging, dropping, and visually slicing and dicing through data. The latter capability is actually critical in a big data world where there are likely more data points than you can ever digest at once.

Zoomdata

Zoomdata is far from the only analytics company to support mobile devices, but it’s one of the few I know of (Roambi also comes to mind) designed primarily for them. Zoomdata connects to standard business data sources, but takes advantage of touch screens and the D3.js visualization project to offer up some visually interesting charts that are designed to be manipulated like an artist’s palette.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Cloud computing and trickle-down analytics
• A near-term outlook for big data

The short answer is no. The Sunday Times (paywall alert) may have billed its story as being about the potential sale of 27 million people’s details to the cops, but the reality is somewhat less alarming. As Ipsos Mori has been forced to explain in response to the exposé:

“In conducting this research we only receive anonymized data without any personally identifiable information… We do not have access to any names, personal address information, nor postcodes or phone numbers. We can see the volume of people who have visited a website domain, but we cannot see the detail of individual visits, nor what information is entered on that domain. We only ever report on aggregated groups of 50 or more customers. We will never release any data that in any way allows an individual to be identified.”

So what does this data tell us? According to the original article, it provides insights based on “gender, age, postcode, websites visited, time of day text is sent [and] location of customer when call is made”.

Reverse engineering

Now, as we discussed recently, it is easier than you might think to de-anonymize data due to the uniqueness of our personal movement patterns — as long as you have the will, the datasets and the pieces of identifying information that can be correlated with the anonymized individuals effectively described in those datasets. So those horrified reactions to the weekend’s revelations are not entirely groundless. They are over-the-top, though.

There is a significant difference between a register of communications (who contacted whom and when) and a pool of anonymized data where the most fine-grained nugget of information that might be reverse-engineered would tell you that Person X visited the Gmail domain while within a 100 meter radius of the corner of Oxford Street and Tottenham Court Road. To assume equivalence between the two ideas is to ignore the elements of intent, will, data-crunching capacity and, frankly, competence. In short, there are far easier ways for the police to track individuals through their handsets, such as just going to the carrier and demanding to do so.

(The Sunday Times said sources claimed “officers had been enthusiastic about the potential for tracking users of pay-as-you-go phones,” but – quality of sources notwithstanding — I suspect those officers may have been slightly overestimating their own data-crunching powers. They may have also overlooked the fact that the operators would have no idea of their pay-as-you-go users’ age or gender, making it near-impossible to tease out an individual from the anonymized mass. Either way, they backed off once the story broke.)

Not damning

And then there’s the matter of this data’s innocent utility. Of all the sources of “big data” that is both largely untapped and genuinely useful, mobile operators must be among the most potentially fruitful. In societies where everyone is carrying a phone, there can be no better way to establish the density and fluidity of traffic flows and footfall. This data is gold dust, not just for retailers, but also for town planners and councils. It shows us how our cities and roads really work, and it can help us make them more efficient and pleasant to live in or use.

I feel a bit sorry for EE in this particular case. After all, its rivals Telefonica (trading as O2) and Vodafone are also offering up their customer data for analytics purposes – Telefonica’s “Dynamic Insights” program is being carried out in partnership with market research firm GfK, while Voda launched its mobile analytics play just last Friday.

“Everyone is doing it” would be a lousy apology in itself, but I don’t think any of these carriers or their partners are doing anything wrong, as long as their datasets are suitably anonymized. If people could feasibly be personally identified from this data, the carriers and their market research partners would instantly find themselves on the wrong side of existing data protection legislation — the fines in the UK for this stuff are pretty paltry, but they would also quickly lose the trust of their customers, so there’s little motivation for the telcos and their partners to cross the line.

It’s great that people are concerned and watchful about their privacy, and long may they continue to be. However, this is a case where the potential benefits of the data are both great and realistically attainable, and where the downsides are so unfeasible as to be worth discounting, at least at this stage. It’s now up to the carriers to explain this to their customers in understandable and honest terms.

There will be great battles worth fighting in the war over our personal data and its exploitation. This ain’t one of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• The future of mobile: a segment analysis by GigaOM Pro
• 4G: State of the Union