Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• Infrastructure Q1: Cloud and big data woo enterprises
• Health care and big data in 2012

“We’re thrilled,” Tableau co-founder and CEO Christian Chabot told me during a call after the market closed. One should hope so.

Chabot and his fellow co-founders stand to make a lot of money if today’s closing price holds up, as does its sole investor NEA. The firm put $15 million into Tableau since it launched in 2003, and has rode that sum to profitability and more than $127 million in annual revenue.

Here’s a quick chart (made using Tableau Public) showing who owns how many share and what they’re potentially worth.

The company didn’t really need more capital to operate, Chabot said, but one of the primary drivers was to raise awareness of the company. It has about 12,000 customers, he said, but there are millions more possible users. As part of attracting them, the company is going to expand globally and is working to improve its reach across mobile devices, the cloud and the Mac operating system.

“I don’t believe in the this whole ‘or’ philosophy with computers,” Chabot said. “It’s ‘and’” — meaning people will use desktops and tablets and smartphones.

More prominence and more users singing its praises might also dispel the notion that Tableau is just about visualization. It has some fairly advanced features under the covers (as a commenter to my earlier post about the company’s influence pointed out), even if they’re hidden by the relatively simple user experience.

“Tableau is not a visualization company, per se, it’s really an analytics company,” Chabot said.

However, if the company really wants to expand its reach to everyone one who wants to gain knowledge from data — something Chabot calls a “timeless human need” — it might actually need to get simpler. More marketing can let potential business users know about new features like forecasting and data-extraction, but it won’t make a dentist is Des Moines better at formatting his data.

After raising $254 million in its IPO, though, Tableau is in a good place to do whatever it has to.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• The importance of putting the U and I in visualization

That’s an apt ticker symbol for a company that is in some ways a bellwether for the current fascination with all things data. Tableau isn’t a big data company, per se, but its visualization software breathes life into many big data calculations. Its focus on making software that’s easy to use and that creates visually captivating charts has turned people from numerous professions into amateur data analysts. (I’ve even used it in the past, including for the first time in 2011.)

Christian Chabot

As Chabot told me during a conversation in 2011, “In any field of human endeavor … there are a hundred to a thousand more people who understand the data of that field more than they understand reporting and analytics.”

Anytime you read about a hot new visualization or analytics startup promising the moon, you’re also seeing the results of what Tableau has sown in terms of the user experience. Many of those same companies will be quick to tell you how limited Tableau’s capabilities are. It’s memory-bound, it doesn’t have a database, it’s not available in the cloud (or on the Mac operating system), it can’t do predictive analytics. All true.

Of course, if it raises the kind of capital it expects to by going public, it can build and buy a lot of those capabilities. If pricing stays flat all day Friday, Tableau stands to make $155 million from its 5 million shares. Previous estimates had Tableau’s market cap at around $1.7 billion at a price of $29 per share (the company’s S-1 filing is available here).

If investors have really bought into the company and the concept of a data-driven world, then who knows. Machine-data expert Splunk wnet public in 2012, flying the big data banner, and saw shares peak at 91 percent above its original asking price of $17.

I’m not suggesting Tableau is the biggest name in data, or even that it will some day become it. This next-generation analytics field is very young, with startups and larger vendors alike sometimes competing against themselves to win wholly new accounts than trying to displace legacy vendors within large enterprises. And every month, it seems, I come across some new startup that was built with the same principles in mind as Tableau, but with the advantage of having today’s best practices baked into its software.

But Tableau definitely commands a lot of the mindshare. How it fares as a public company could be a strong indicator of just how powerful the data movement is, and how well it capitalizes on a new influx of cash will determine how long it stays on the top of customers’ minds.

This post was updated at 7:01 p.m. to include previous estimates of the company’s market capitalization and a link to its S-1 filing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 4 iPad apps to help wrangle data
• Health care and big data in 2012

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Storage, databases, query processing and algorithms are all extremely important — heck, visualization is next to nothing without them — but in a data-driven world where is obsessed with insights, they’re just the foundational layers. They are to big data what server and network configurations are to mobile-app development on platforms like Parse. If you’re going to find out new things from massive and highly complex data sets, or going to give new types of people the ability to analyze even simple data, the presentation of that data and the ability to create consumable presentations are critical.

With that in mind, here are six startups I’ve seen trying to fundamentally change the way that data is visualized. Some are highly complex under the covers, some are not and none are perfect, but they’re all doing their part to make us rethink what it means to look at data and make spreadsheets and static charts look like relics. (And this list is by no means exhaustive, so feel free to add your favorite visualization tools in the comments.) We’ll be highlighting data visualization at our design-focused RoadMap conference in San Francisco in November (sign up here to get first access to tickets this Summer).

Ayasdi

The idea of network graphs isn’t new, but Ayasdi’s approach to it is. Under the covers, there’s an HBase data store, a technique called topographical topological data analysis and hundreds of machine learning algorithms to churn through complex data sets and determine the similarity among the data points. To the end user, though, there’s a map of the data set that looks a lot like a network graph (only it’s probably not network data) highlighting clusters of related data points that analysts might want to investigate further.

BeyondCORE

BeyondCore actually operates under the same basic premise as Ayasdi — show users the significant correlations so they don’t have to think of the queries that will uncover them — but it uses some different techniques to get there. It uses a different visualization method, too: BeyondCore sticks to standard charts, but actually offers the option of having an avatar talk users through the correlations the software has discovered.

ClearStory

ClearStory has a pretty unique product in the works — even if it’s keeping many details and all of its screenshots under lock and key until its formally launches. Essentially, though, it’s trying to tell stories via visualizations that display mashups of numerous data sources, update automatically when the source data changes, and invoke collaboration and social concepts. Here’s Co-founder and CEO Sharmila Mulligan explaining the idea behind ClearStory at Structure: Data in March.

Datahero

Unlike so many data startups, Datahero isn’t trying to woo people fed up with business-intelligence software or the difficulties of getting insights from Hadoop data. Rather, it’s trying to let people with simple business or personal data make simple charts without ever having to enter an Excel function or worry too much about how their spreadsheets are formatted. Early on, Datahero’s visualizations are still pretty commonplace (bars, pies, plots, etc.), but it’s the ease of creating them that’s so unique.

Platfora

Platfora has undertaken the ambitious task of trying to make analyzing mountains of data stored in Hadoop clusters as easy as analyzing their own Stripe data might be for developers using Datahero. It’s based on a foundation of Hadoop and massively parallel query processing, but is presented like an HTML5 version of current visualization golden boy Tableau that’s all about dragging, dropping, and visually slicing and dicing through data. The latter capability is actually critical in a big data world where there are likely more data points than you can ever digest at once.

Zoomdata

Zoomdata is far from the only analytics company to support mobile devices, but it’s one of the few I know of (Roambi also comes to mind) designed primarily for them. Zoomdata connects to standard business data sources, but takes advantage of touch screens and the D3.js visualization project to offer up some visually interesting charts that are designed to be manipulated like an artist’s palette.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Cloud computing and trickle-down analytics
• A near-term outlook for big data

Lumira Cloud supposedly gives SAP an answer to the recent explosion in the cloud-based, self-service data visualization offerings. The HTML5-built BI service comes with a “monthly” subscription fee (albeit one that can only be ordered in annual chunks) and lets its users publish and share data visualizations with one another for viewing or editing on desktop or mobile devices.

SAP Lumira Cloud appears to be more an Dropbox-ish add-on for the desktop version of Lumira than a cloud-based replacement, but it does also allow the creation of datasets from Excel documents. The service, which integrates with on-premise data and naturally supports HANA, can also be used to share SAP BusinessObjects Design Studio files and SAP Crystal Reports documents.

This release appears to be the culmination of what SAP has been previously referring to as “project Photon” – supposedly the company’s “true departmental self-service BI offering.” The issue here, of course, is the monumental and somewhat confusing nature of the company’s portfolio. After all, doesn’t SAP already do this SME-courting, departmental analytics stuff through its BusinessObjects BI OnDemand product?

Try visiting at least one of the BI OnDemand product pages and you’ll be taken through to the Lumira page. Look at the Lumira Cloud FAQs and you’ll be told that BI OnDemand will continue to run “in parallel” to Lumira Cloud, but also that OnDemand customers can contact their account representative “to discuss the best timing and strategy” for migrating to the new service.

Perhaps this less-than-clear situation presages a simplification of SAP’s portfolio – no doubt more will be revealed at the company’s SAPPHIRE NOW conference this week. If it doesn’t, customers in search of next-generation data visualization tools have many far more straightforward options to check out.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Taken alone, ScraperWiki isn’t that big of a deal, but it’s part of a huge revolution that has been called the democratization of data. More data is becoming available all the time — whether from the government, corportations or even our own lives — only it’s not of much use unless you’re able to do something with it. ScraperWiki is now one of a growing list of tools dedicated to helping everyone, not just expert data analysts or coders, analyze — and, in its case, generate — the data that matters to them.

After noticing a particularly large numbers of tweets in my stream about flight delays yesterday, I thought I’d test out ScraperWiki’s new Twitter search function by gathering a bunch of tweets directed to @United. The results — from 1,697 tweets dating back to May 3 — are pretty fun to play with, if not that surprising. (Also, I have no idea how far back the tweet search will go or how long it will take using the free account, which is limited to 30 minutes of compute time a day. I just stopped at some point so I could start digging in.)

First things first, I ran my query. Here’s what the data looks like viewed in a table in the ScraperWiki app.

Next, it’s a matter of analyzing it. ScraperWiki lets you view it in a table (like above), export it to Excel or query it using SQL, and will also summarize it for you. This being Twitter data, the natural thing to do seemed to be analyzing it for sentiment. One simple way to do this right inside the ScraperWiki table is to search for a particular term that might suggest joy or anger. I chose a certain four-letter word that begins with f.

Surprisingly, I only found eight instances. Here’s my favorite: “Your Customer Service is better than a hooker. I paid a bunch of money and you’re still…” (You probably get the idea.)

But if you read my “data for dummies” post from January, you know that we mere mortals have tools at our disposal for dealing with text data in a more refined way. IBM’s Many Eyes service won’t let me score tweets for sentiment, but I can get a pretty good idea overall by looking at how words are used. For this job, though, a simple word cloud won’t work, even after filtering out common words, @united and other obvious terms. Think of how “thanks” can be used sarcastically and you can see why.

Using the customized word tree, you can see that “thanks” sometimes means “thanks.” Other times, not so much. I know it’s easy to dwell on the negative, but consider this: “worst” had 28 hits while “best” had 15. One of those was referring to Tito’s vodka and at least three were referring to skyline views. (Click here to access it and search by whatever word you want.)

Here’s a phrase net filtering the results by phrases where the word “for” connects two words.

Anyhow, this was just a fast, simple and fairly crude example of what ScraperWiki now allows users to do, and how that resulting data can be combined with other tools to analyze and visualize it. Obviously, it’s more powerful if you can code, but new tools are supposedly on the way (remember, this is just a beta version) that should make it easier to scrape data from even more sources.

In the long term, though, services like ScraperWiki should become a lot more valuable as tools for helping us generate and analyze data rather than just believe what we’re told. Want to improve your small business, put your life in context or perhaps just write the best book report your teacher has ever seen? It’s getting easier every day.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• 4 iPad apps to help wrangle data
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

Speed is the driving factory behind the in-memory analytics push that spans everyone from classic Teradata rivals like SAP and Oracle to startups such as MemSQL. Estimates vary as to the exact speed difference between data access in RAM versus hard disk, but Teradata is claiming RAM is 3,000 times faster. The speed difference between RAM and solid-state drives or flash memory is smaller, although still significant.

Of course, cost also comes into play, as the speed and cost tend to go hand in hand when it comes to storage media. That’s one reason Teradata says its new technology, called Intelligent Memory, doesn’t operate fully in-memory like some competitive offerings do. Rather, it places only the “hottest” data in memory for super-fast analysis and spreads the rest between solid-state drives and disk within a Teradata environment.

This concept of intelligent data placement has been around a while in the storage space (it’s part of EMC’s new ViPR software-defined storage platform, too), but the advent of big data and abundant flash has given it some new life. Many companies desire a tiered system in which they can pay more for fast access to their important or hot data, while saving some cash on lower-performance for their older and less-accessed data. Facebook is really pushing the envelope here with its cold storage initiative — something VP of Engineering Jay Parikh will likely discuss at our Structure conference June 19 and 20 in San Francisco.

In analytics, though, RAM, not flash, is the fastest medium out there. Whether someone goes all-RAM or a tiered approach like Teradata pushing probably depends on how much performance they need across how much data, as well as how much they’re willing to pay. But if you’re doing interactive analytics in the next decade, they’re almost certain to be in-memory to some degree.

Feature image courtesy of Shutterstock user Hellen Sergeyeva.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Red-Hot Data Warehouse Market: Who’s Buying Next?
• The importance of putting the U and I in visualization
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

When you’re in Atlanta, as I was during a recent trip, the obvious place to start talking about orange juice and data is with Coca-Cola. Foreman can tell you all about how the beverage giant — whose headquarters tower over the city just a just a mile away from MailChimp’s office — uses advanced algorithms and giant vats of different juices to ensure the proper flavor of its Simply Orange line of orange juice. However, it’s something else Coca-Cola is doing that inspired the way Foreman thinks about data and that’s helping MailChimp re-imagine what it means to engage with fans, readers and customer through their inboxes.

Anyone familiar with how large web companies came to pioneer the practice of what we now call “big data” should appreciate the analogy. Coca-Cola, which also owns Minute Maid, produces a lot of excess pulp when it makes orange juice. For decades, presumably, it had just been throwing that pulp away, but in 2006 it decided to make use of it by launching a new product called Minute Maid Pulpy. Sold primarily in Asian countries, Pulpy has become a billion-dollar business for Coca-Cola.

Once MailChimp is done with its primary business of sending emails, it has a lot of pulp of its own in the form of data. And rather than just ignoring it or writing up some cute blog posts (which he also does), Foreman and his bosses want to turn that data into revenue.

First things first: Making better orange juice

Neil Bainton

Actually, though, MailChimp first brought in Foreman in 2011 to help the company improve its core business of letting users build and send their emails. MailChimp’s culture was built around many things, COO Neil Bainton told me, but data wasn’t one of them. It had “various fits and starts” through the years trying to work data into its business model, and each step just added more complexity.

The challenges were technological as well as cultural, but Foreman had a plan, of which focus was a key aspect. Keeping a tight focus meant Foreman and his lone-developer sidekick could build what they needed to in a short timeframe. It also meant the company didn’t have to worry about some massive overnight transformation into a data-obsessed company like Google.

John Foreman

“[They] don’t need to be afraid the entire culture is gonna fall down if we bring in this weird math guy,” he joked.

Foreman’s first project — deploying artificial intelligence models that would automatically detect spammy email lists from MailChimp’s users – is actually critical to the way MailChimp operates, though. It was up and running in production within a year, after a technologically challenging effort of merging separate database instances for each customer into a single environment that would let MailChimp run complex analyses across its customer base.

It’s such an important project, Foreman explained, because internet service and email providers keep reputation scores on the IP addresses that send email through their systems. Because MailChimp serves as the email engine for its millions of users, sending too many messages that get flagged as spam and lower MailChimp’s reputation will have a negative impact on everyone. The company used to deal with spam manually, and only after recipients began complaining about the messages they received.

“It used to be before we had that AI model in place that everyone had a crappier experience,” Foreman said.

Say goodbye to those ’90s fans, Pearl Jam

Source: MailChimp

Now, however, MailChimp knows some of the telltale signs of spam for which it should be on the lookout. If too high a percentage of email addresses on a given list are also available via publicly available lists or those you can buy on sketchy corners of the internet, it’s probably spam. Too many old and far-more-likely-to-be-dead Earthlink or Compuserve addresses, or letters within one keystroke of each other as if someone just mashed the keyboard? Probably spam.

Thankfully, though, about 98 percent of the spam that MailChimp identifies is what Foreman calls “ignorant” — that is, people or companies that just don’t know the laws or best practices around sending emails. But ignorance doesn’t mean MailChimp relaxes its rules. Recently, it even flagged Pearl Jam for spammy practices because the band was trying to reconnect with old fans whose email addresses read like a who’s who list of 1990s email providers.

Having such a high percentage of ignorant spam actually has a positive effect on the company’s overall goal of monetizing its vast data repositories. Because the AI model automates what used to be a manual process, and because most innocent spammers will fall in line quickly once they’re notified (as opposed to nefarious spammers who constantly try to outsmart the system), MailChimp can pretty much set the model loose, forget about it and get to work on new efforts, Foreman said.

Now, about that pulp

Spam under control, MailChimp can focus its efforts on actually building new products with data, just like Coca-Cola did with that extra pulp. One of its first orders of business is figuring out how to help customers get to know better the people to whom they’re sending their newsletters.

With this in mind, the company built a service called Wavelength that shows customers other newsletters that are similar to theirs. But the system that powers Wavelength also stores pretty much every interaction that every email address in the company’s database has with the newsletters they’re sent. That means what emails they open and when they open them, what links they click and when they click them, and what other newsletters they’re subscribed to. MailChimp also has a feature called Ecommerce360 that lets customers track clicks right through to conversions (marketing speak for someone actually buying something).

The company has been playing around with this data to identify clusters of users based on their behaviors and their interests — some of which Foreman has detailed on the company’s blog — and now it wants to roll it out to customers via a product MailChimp is calling ChimpQuery. Built atop Google’s BigQuery analytics service, ChimpQuery will let customers start doing this type of clustering and segmentation on their own, while saving MailChimp the troubles of hosting that infrastructure itself. (You can play with a monstrous, interactive graph of the entire MailChimp subscriber list here.)

If you sell knitting supplies and you find out there’s a big cluster of people on your mailing list who also are interested in wedding planning and custom jewelry, there might be an opportunity to create your content with these interests in mind or even to partner with companies in those spaces.

A sample cluster of subscribers.

Another topic that has been on Foreman’s mind lately is what he calls “frequency elasticity of engagement.” He’s done research suggesting that blasting the heck out of your email list might actually have detrimental effects in the long term (regardless of how the Obama campaign successfully exploited this strategy) but noted that engagement also has a lot to do with content and a particular company’s given user list. MailChimp’s data could help customers figure out the ideal schedule for emailing their subscribers.

For example, Birchbox has really high engagement because people love the service and have to open their emails to find out what goodies they’re receiving. Emails from a company like Papa John’s, on the other hand, might sit in someone’s inbox essentially as spam until they want to order a pizza and go searching for a coupon. Everyone has to figure out what pace and engagement metrics work for them.

Reining expectations back in

However, now that management is fully sold on the power of data, Foreman sometimes finds himself managing expectations rather than just pitching his ideas. COO Bainton, for example, is adamant that MailChimp start aiding its publishing-industry customers by using techniques such as natural-language processing and semantic analysis to help them personalize emails based on readers stated and unstated interests (that is, what boxes they check when they sign up and what stuff they actually click on).

Foreman, well, he’s pretty sure that’s too big a challenge for MailChimp to tackle considering how many publishing customers it has. MailChimp would have to understand all those customers’ industries to some degree (open source tools tend to highlight technically but not situationally relevant relationships, he said, and don’t always understand things like sarcasm) and probably the different languages they publish in, as well. Rather than understand content, he’d rather focus personalization efforts around how users are connected.

The company also needs to balance its ambitions with what’s legally and socially acceptable. The creep factor might be more important than what’s legal when it comes to email marketing. MailChimp determines the legality of everything it does before rolling it out, Foreman explained, but in era of “post-modern spam” where legitimacy is in the eye of the recipient and where some people use their “spam” button as a proxy for unsubscribing, companies must be careful not to offend.

“The more we can tell you about that list without getting creepy is really useful,” Bainton said. However, he added, ”I think expectation is more important than law.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?
• The Red-Hot Data Warehouse Market: Who’s Buying Next?