The reason for creating Pattern is pretty simple, according to Concurrent founder and CTO Chris Wensel: “Hadoop is never used alone.” It’s always part of a data environment that also includes databases, visualization tools, analytics software and/or statistical analysis tools that arguably do the really valuable work. Hadoop’s real value is an integration platform that can feed data into these other systems and, ideally, put their outputs to work across much larger datasets.

Developers can use the Pattern Java API to create machine learning jobs, but they can also simply export a Predictive Model Markup Language (PMML) file from software like R, SAS and MicroStrategy that Pattern will read and run them as a Cascading workflow. Models are useless unless you can run them in production, Wensel said, and Pattern lets them run across more data, stored in Hadoop, than you can use to build them with those other tools.

However, Wensel noted, “The real takeaway isn’t Pattern itself.”

From his perspective, the real story is Pattern plus Cascading plus Lingual, the open source SQL-to-Hadoop tool that Concurrent recently developed and released. Lingual is the tie that binds everything together, creating a sort of assembly line for data as it works its way from generation to delivering some value. For example, someone might create a Cascading job that adds structure to incoming data, and then pull some of the data into R using Lingual. Once a model is created in R and exported to the Hadoop cluster using Pattern, Lingual can feed the MapReduce output file back to R so a data scientist can test the model’s accuracy.

And actually, Wensel said, Lingual could have a positive effect on companies’ bottom lines. Airbnb recently replaced a departed engineer with Lingual for monthly migrations of data from Hadoop and into SQL environments. Climate Corporation, a massive Hadoop and Cascading user, could use Lingual to let its crop-and-weather insurance customers access their data from the company’s Hadoop store.

Lingual and Pattern should help Concurrent finally make some money, too. Both of them, as well as the Cascading framework that underpins them, will always be open source, Wensel said, but it plans to create “a suite of products that will make your life much better if … you standardize on Cascading.”

For example, the company has the ability to monitor jobs at the application level rather than the cluster level, meaning it can tell you the details of that job that’s locking up all the resources and whether you really want to kill it (it might be an important report for the CFO …). “We can do some really interesting things,” Wensel said.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

This post was updated at 2:48pm PT to correct Chris Wensel’s title. He is CTO.

Feature image courtesy of Shutterstock user PENGYOU91.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

That’s why a group of researchers from Columbia University and Stanford University developed an algorithm that can do just that, turning those cells into something that resembles your social graph. This lets researchers see how the various cells are related to each other so they know , for example, where to focus cancer treatment and what to track as that treatment progresses.

The idea of representing large or complex data as a graph is nothing new, but it has taken on more prominence thanks to the rise of social media and those ubiquitous social graphs that map out who’s connected to whom. As we highlighted recently, however, graph analysis is becoming more popular outside the realm of social networks, and is being applied to problems that are more complex than just figuring out simple relationships within a network. In cases such as medical research, especially, graphs can provide a very effective way of seeing how potentially hundreds of thousands of data points spanning perhaps hundreds of variables are similar to each other.

That’s exactly what the team at Columbia and Stanford has done with a new algorithm that they’ve demonstrated within the realm of mass cytometry. According to a press release announcing the research (which is available via paid download at Nature Biotechnology):

“The method, called viSNE (visual interactive Stochastic Neighbor Embedding), is based on a sophisticated algorithm that translates high-dimensional data (e.g., a dataset that includes many different simultaneous measurements from single cells) into visual representations similar to two-dimensional ‘scatter plots’ ….

“The viSNE software can analyze measurements of dozens of molecular markers. In the two-dimensional maps that result, the distance between points represents the degree of similarity between single cells. The maps can reveal clearly defined groups of cells with distinct behaviors (e.g., drug resistance) even if they are only a tiny fraction of the total population. This should enable the design of ways to physically isolate and study these cell subpopulations in the laboratory.”

I assume they say similar to scatter plots because the algorithm is analyzing data across more than two dimensions, although the resulting chart is essentially the same (i.e., data points with similar characteristics will form clusters).

The results of viSNE, showing cell densities in diagnosis and relapse samples.

Whether or not they’re technically similar, this research seems similar to what Ayasdi is doing with its new data-analysis software based on a technique called topological data analysis. In both cases, though, the algorithms aren’t necessarily concerned with how data points interact with one another (like in network graphs), but rather what similar characteristics the points share. Ayasdi’s software has been used in cancer research, too, including on datasets spanning hundreds of patients and tens of thousands of variables.

In theory — although not likely in practice considering the complexity of the datasets medical researchers are dealing with — these approaches are similar to clustering approaches that are also popular among data scientists working with web companies. In areas such as e-commerce or email management, for example, where there isn’t a strong social element, companies can broadly break customers into distinct groups based on their behavior or interests.

A sample cluster of MailChimp subscribers.

Of course, curing cancer is a slightly more compelling — and difficult — goal than targeted advertising. The algorithms have to be precise so as not to miss similarities hidden within the mass of data. In the case of viSNE, the researchers say they’ve been able to spot small groups of cells (like 20 out of tens of thousands) that might be able to survive chemotherapy and increase the likelihood of a recurring tumor.

But we probably shouldn’t bee too quick to discount the work that web companies do as somehow less valuable than that of cancers researchers, for example. The big data era arguably started with the web, and web companies have generated some of the most important data-analysis techniques and technologies around today (see, for example, Google’s Jeff Dean, with whom I’ll be speaking at our Structure conference next month). As medical researchers start generating more and more data via cytometry, genome sequencing and even electronic medical records, it will be critical for individuals in all fields to keep track of what data scientists in other fields are doing and figure out how that might apply to their own work.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout

President and COO George Mathew says the company’s mission is to be a one-stop shop for statistical analysis. It wants to be the one place where analysts and data scientists can blend their data, model it on it and then visualize it. Often, he noted, that same process might require two or three separate products.

Another feature that Alteryx hopes will set it apart is its collection of prebuilt models in what the company calls an analytics gallery. Users can share their own work or find models others have built for tackling similar issues. Alteryx also offers up its own pre-formatted datasets for analysis, often public data such as the U.S. census.

“The canvas for creating an analytics application should never be blank for the analyst when they’re getting started,” Mathew explained. They often need to understand external data as well as their internal data, so Alteryx’s software gives them easy access to it.

Because it’s based on the R statistical-programming language, heavy R user Walmart has been able to transition some workloads to Alteryx when employees need an easier user experience. McDonald’s uses it to analyze data about franchisees and about its growth strategy in China, and Bloomin’ Brands (parent of company of Outback Steakhouse and other restaurants) is using it to help build menus that take into account what diners in various parts of the country prefer to eat. Nine of the 10 leading top wireless providers providers are also users, Mathew said, trying to blend actual call data with traditional sources such as customer service data.

Mathew compares Alteryx’s current growth as analogous to software-as-a-service applications like Salesforce.com in the CRM space, or even Tableau in the traditional business-intelligence space. In a business world increasingly driven by at least the idea of big data, one might expect any vendor pushing a more consumer-like purchase and consumption experience to get interest from companies tired of dealing with legacy software or never wanting to experience it in the first place.

“The disruption that’s happening is creating a new space for ourselves,” Mathew said, “without having to go head to head, frankly, with the a status quo out there.”

Feature image courtesy of Shutterstock user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Internet of things: creating tomorrow’s health care
• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• Infrastructure Q1: Cloud and big data woo enterprises
• Health care and big data in 2012

“We’re thrilled,” Tableau co-founder and CEO Christian Chabot told me during a call after the market closed. One should hope so.

Chabot and his fellow co-founders stand to make a lot of money if today’s closing price holds up, as does its sole investor NEA. The firm put $15 million into Tableau since it launched in 2003, and has rode that sum to profitability and more than $127 million in annual revenue.

Here’s a quick chart (made using Tableau Public) showing who owns how many share and what they’re potentially worth.

The company didn’t really need more capital to operate, Chabot said, but one of the primary drivers was to raise awareness of the company. It has about 12,000 customers, he said, but there are millions more possible users. As part of attracting them, the company is going to expand globally and is working to improve its reach across mobile devices, the cloud and the Mac operating system.

“I don’t believe in the this whole ‘or’ philosophy with computers,” Chabot said. “It’s ‘and’” — meaning people will use desktops and tablets and smartphones.

More prominence and more users singing its praises might also dispel the notion that Tableau is just about visualization. It has some fairly advanced features under the covers (as a commenter to my earlier post about the company’s influence pointed out), even if they’re hidden by the relatively simple user experience.

“Tableau is not a visualization company, per se, it’s really an analytics company,” Chabot said.

However, if the company really wants to expand its reach to everyone one who wants to gain knowledge from data — something Chabot calls a “timeless human need” — it might actually need to get simpler. More marketing can let potential business users know about new features like forecasting and data-extraction, but it won’t make a dentist is Des Moines better at formatting his data.

After raising $254 million in its IPO, though, Tableau is in a good place to do whatever it has to.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• The importance of putting the U and I in visualization

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

That’s an apt ticker symbol for a company that is in some ways a bellwether for the current fascination with all things data. Tableau isn’t a big data company, per se, but its visualization software breathes life into many big data calculations. Its focus on making software that’s easy to use and that creates visually captivating charts has turned people from numerous professions into amateur data analysts. (I’ve even used it in the past, including for the first time in 2011.)

Christian Chabot

As Chabot told me during a conversation in 2011, “In any field of human endeavor … there are a hundred to a thousand more people who understand the data of that field more than they understand reporting and analytics.”

Anytime you read about a hot new visualization or analytics startup promising the moon, you’re also seeing the results of what Tableau has sown in terms of the user experience. Many of those same companies will be quick to tell you how limited Tableau’s capabilities are. It’s memory-bound, it doesn’t have a database, it’s not available in the cloud (or on the Mac operating system), it can’t do predictive analytics. All true.

Of course, if it raises the kind of capital it expects to by going public, it can build and buy a lot of those capabilities. If pricing stays flat all day Friday, Tableau stands to make $155 million from its 5 million shares. Previous estimates had Tableau’s market cap at around $1.7 billion at a price of $29 per share (the company’s S-1 filing is available here).

If investors have really bought into the company and the concept of a data-driven world, then who knows. Machine-data expert Splunk wnet public in 2012, flying the big data banner, and saw shares peak at 91 percent above its original asking price of $17.

I’m not suggesting Tableau is the biggest name in data, or even that it will some day become it. This next-generation analytics field is very young, with startups and larger vendors alike sometimes competing against themselves to win wholly new accounts than trying to displace legacy vendors within large enterprises. And every month, it seems, I come across some new startup that was built with the same principles in mind as Tableau, but with the advantage of having today’s best practices baked into its software.

But Tableau definitely commands a lot of the mindshare. How it fares as a public company could be a strong indicator of just how powerful the data movement is, and how well it capitalizes on a new influx of cash will determine how long it stays on the top of customers’ minds.

This post was updated at 7:01 p.m. to include previous estimates of the company’s market capitalization and a link to its S-1 filing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 4 iPad apps to help wrangle data
• Health care and big data in 2012

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

Word that Lucky Sort is shutting down and that several of its team are joining Twitter’s revenue engineering department suggests this is exactly what the acquisition aims to accomplish.

As it stands, companies use Twitter as a way to track how people are talking about them and maybe, if they’re really advanced, do some sentiment analysis. If they’re willing to pay a third party, Datasift and Gnip are more than happy to broaden marketers’ views to encompass the entirety of Twitter’s data, both real-time and historical. What companies really can’t do, though, is run their own advanced analytics about topics straight from the Twitter platform.

One view of the Lucky Sort dashboard

The value proposition from such a product should be obvious at this point. Facebook, Google and Yahoo all collect a lot of data about how people are using their platforms and what topics are trending, and they all offer it up via a variety of products targeting marketing types and the public at large. If Twitter wants to be taken seriously as a venue for advertising budgets and a platform for measuring the pulse of the nation, people need to be able to ask questions of its data without relying on an intermediary or the occasional Twitter blog post.

As a journalist, I’d love to have access to this type of tool to track trending topics in real time and spot possible stories as they’re happening. The appeal to marketers should be obvious. As IBM’s Erick Brethenoux told me recently, “[Marketers] talk a good game about social data. Very few actually leverage it effectively today.”

At Twitter, though, data is a slightly different beast than at other web companies. Twitter’s value lies largely in real-time data — topics can be peak, crest and all but vanish within a 48-hour window. This situation has hampered some of Twitter’s efforts to surface optimal search results, and it has spurred the decision to buy companies such as Backtype (for its streaming-processing Storm technology) and parallel-processing startup Ubalo.

The latter move, which happened last week, should help Twitter’s development team create new features without worrying about the intricacies of making them run — and run fast — across a cluster of machines. (You can learn a lot more about how companies such as Google, Facebook and Box are rethinking infrastructure to handle their unique data needs at our Structure conference next month in San Francisco.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Finding the Value in Social Media Data
• Listening platforms: finding the value in social media data