There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

As BloomReach CEO Raj De Datta told my colleague Derrick Harris early this year, companies don’t know how to show off their product catalogs in a way that best aligns with how customers search. Less than a quarter of web pages get any traffic from natural or paid search in a given month  – a problem that will only get worse as the amount of online data grows. Their products are needles in an ever-expanding haystack. But if they know how people are searching for things and learn how to display their content better to suit that behavior, they can boost discoverability and thus sales.

“Understanding relevance of content to the way people express themselves turns out to be a difficult problem,” De Datta told Harris.

According to the company’s website, its core Web Relevance Engine, which is used by customers including Neiman Marcus Direct, Guess? and Drugstore.com:

analyzes consumer interactions across the web and semantically interprets content on more than one billion web pages daily. The cloud applications powered by the WRE dynamically adapt websites to capture existing consumer demand across search, social and advertising channels — driving relevance and significant incremental revenues across a large customer base including the retail, travel, education, financial and listings industries.

Retailers like Amazon and Wal-Mart are reportedly working on similar technologies but they are not commercially available.  BloomReach, Mountain View, Calif., said it will double its sales and marketing efforts and triple its R&D funding to push more into mobile, social and video marketing channels. Current investors Lightspeed Venture Partners and Bain Capital also contributed to this round.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Listening platforms: finding the value in social media data
• Cloud computing and trickle-down analytics

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• A near-term outlook for big data
• Why service providers matter for the future of big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Overview, Q2 2010

Today’s analytic techniques and technologies can tell marketers not only what campaigns are working, but also where to spend next and — in some cases — the very language to use on their web sites. Here are five companies you’ll likely be hearing a lot more about if you’re not already a user.

1. Acxiom

Most people in the directed marketing world have heard about Acxiom — it has been around for more than forty years and has lots of identity data about people — but they’re about to think about the company in a whole new light if new CEO Scott Howe has his way.  ”[Acxiom is] like the ’62 Chevy I first drove,” Howe told me recently. It’s comfortable and predictable, “but not super-sexy.”

Phil Mui

Howe’s plan involves, among other things, transitioning the company into a software-as-a-service model where customers can access and analyze their own data as well as Acxiom’s voluminous data sets. To make sure Acxiom does it right, the company hired former Google Analytics head  group product manager Phil Mui as its chief products and engineering officer.

Acxiom has the right data –and knows what other types it needs to get –but Mui wants to incorporate Google’s style of user-friendly services and high-end analytics to help deliver that data in the most-effective way possible. Thanks to social media, web television, mobile devices — pick a medium for capturing data and reaching consumers — “it will [very quickly] change from marketers having not enough data to having too much,” Mui told me. [But] the ability to serve up insights will be perhaps more valuable [than the data itself].”

Oh, but the data does matter. “Google will never be a company that is going to do marketing too well,” Mui said, in part because it doesn’t want to host personally identifiable data. However, marketers are hitting a wall because “because there’s only so much you can do with anonymous data.” Acxiom has valuable identity data and, utilized correctly, it’s a potential gold mine.

2. Applied Predictive Technologies

Applied Predictive Technologies is another older company — it has been around for about 12 years — but one that has its hooks into some of the world’s largest companies. Seriously, its customer roster reads like a who’s who of retail (Walmart), restaurants (McDonald’s), hospitality (Holiday Inn) and banking (Wells Fargo). Not that they’re complaining.

According to APT Founder and CEO Anthony Bruce, its customers love its cloud-based platform because it lets them ask entirely new types of questions of their data in order to better understand how to spend their marketing dollars. APT does this by collecting lots of client data — pretty much everything related to sales transactions, as well as demographic, geographic, competitive and other info — and enabling customers to figure out the business impact of any given decision. Those decisions, Bruce said, can be anything from where to target a specific promotion to advertise online or in print or to repaint the inside or the outside of a restaurant.

And APT users can ask question at any point in the process. For example, they can try to predict outcomes by analyzing similar decisions in branches with similar attributes, or they can analyze the outcome of a particular campaign and find out how, and why, it worked out or didn’t work out. They can even ask counterfactual questions, Bruce said.

Although customers’ own transactional data will always be the most-important aspect of APT’s platform, Bruce said the company has its eye on the deluge of data coming from new sources. There’s definitely value to be added by analyzing customer data against sensor, RFID, social media and other data sources.

3. BloomReach

I’ve covered BloomReach a couple times now, and the company just keeps getting hotter. After building up a roster of big-name clients while still in stealth mode, the software-as-a-service startup finally launched in February and has since announced even more household-name users, including Pottery Barn. As I explained when covering the company’s launch, BloomReach works its magic by automatically creating content on web pages, based on what someone is searching for, that will make visitors more likely to find content, click on links and buy merchandise. In order to do this, it employs a plethora of big data techniques and technologies and analyzes billions of web pages and customer interactions daily.

4. InsightsOne

The brainchild of the team that built Yahoo’s consumer analytics engine, InsightsOne launched in March with $4.3 million in venture funding. The company promises big things because of its big data roots, helping users more accurately place their ads across email, mobile devices and the web, and increasing profits by at least 10 percent — sometimes much more. InsightsOne hasn’t talked publicly about customers yet, but it does share a lot of information about its technology, which achieves micro-segmentation of consumers by applying techniques such as machine learning and graph processing atop a Hadoop platform to make sense of endless streams of data in real time.

5. MarketShare

MarketShare puts a lot of emphasis on big data, and the strategy appears to be paying off as interest in big data picks up. According to Co-Founder and CEO Wes Nichols, the foursix-year-old company, which targets chief marketing officers with cloud-based service that helps them better predict how to spend their budgets, has booked more revenue in the last month than it did during its first two years of businesses. Among those customers are Ticketmaster, which is using MarketShare to inform its dynamic pricing engine, and EA, which uses MarketShare to help it understand to price social market video games.

Essentially, Nichols explained, MarketShare cares about three types of data: where a client is investing its market dollars, what the business outcomes are, and literally hundreds of other variables (e.g., time, weather and price) that could affect those outcomes. And it goes deep in order to determine outcomes. If all you do is track clickthroughs, he said, you might miss that an ad campaign actually resulted in someone opening a piece of mail four months later.

Wes Nichols

MarketShare typically stores terabytes of this data for each customer, Nichols said, and some customer data sets have grown 100-fold in the past year. That’s why it’s a heavy Hadoop user and keep strategic relationships with Amazon and IBM around cloud resources. It’s also why MarketShare hires the best and brightest engineers and data scientists it can find in order to ensure a high-performance, highly scalable and accurate platform. “I think every rocket scientist that used to work in the space program in Los Angeles now works at MarketShare,” Nichols joked.

In the future, though, Nichols says marketing tools need to take the user experience to the next level, beyond the omnipresent dashboards that require users deciphering them in order to gain insight. “All our customers are swimming in dashboards,” he said, which is why MarketShare is working with Adobe on a product that actually “does quite a bit of thinking for the user” in terms of determining areas for improvement. Nichols compares it to software that takes the onus off of airline pilots by automatically reacting to certain conditions, but generating alerts when human action is necessary.

Feature image courtesy of Douglas Cumming.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing and trickle-down analytics
• Cloud computing infrastructure: 2012 and beyond

Big data tools could help cafes determine what specials are best on what days.

The headline might sound like buzzword stew, but it couldn’t be any truer. Big data is red-hot right now for good reason — it really can significantly improve companies’ bottom lines — but doing it right can be hard. You have to hire people who know the right techniques for your business, and then invest in infrastructure and software to actually do the analysis. Unless, of course, someone does the work for you.

I wrote about this recently in a report for GigaOM Pro (sub req’d), in which I considered the future of firms that help companies build big data infrastructure and actually develop custom algorithms specific to their businesses. This type of high-engagement company will always have a place, but the advent of cloud-based data services and startups hiding big data behind SaaS applications means will become little more than a product feature for many companies.

On Wednesday, I realized just how deep this democratization will reach into even niche industries. A vacation-rental platform provider called Bookt announced it is bringing a machine-learning specialist on board to help customers of its InstaManager offering maximize the insights they derive from their user data. According CEO Rob Kall in a press release announcing the hire, “We intend to exploit the insights we are unlocking from our platform data, to empower our InstaManager clients in the Vacation Rental industry, with precise intelligence on individual travel shoppers in real time. “

Added CTO Ben Strum: “[T}he next step is focused on the vast amount of data related to leads, pricing and website traffic. We are currently developing algorithms to optimize and visualize this treasure trove. Our goal is simple: Give our clients the tools they need to succeed in 2015, today in 2012.”

It might not seem too newsworthy until you consider it in the greater context. Now, relatively small companies such as Bookt feature customers Century South Beach Hotel, GoToParkCity.com and SteamboatVacationRentals.net can benefit from complicated analytical techniques like machine learning to maximize their bookings. And they don’t have to lift a finger, presumably — Bookt will take care of applying machine learning to their data as part of the overall service.

Heck, if your local brick-and-mortar deli, pet store or even fishing-bait shop is willing to upload its Quickbooks data or other sales and inventory information to a cloud service, they might eventually be able to reap the rewards of big data, too.

That cloud computing is bringing big data to such narrowly focused industries has been a long time coming, of course. We’ve seen it happening for some time now in the form of applications that target broader user bases, such as ad-targeting and web analytics. BloomReach, which applies everything from MapReduce to Monte Carlo simulations on its backend to optimize customers’ web content via a SaaS model, built a very impressive customer base before launching in February. Then there’s CloudFlare for security, DNAnexus for genomics and the list goes on.

The McKinsey Global Institute has famously predicted a big-data skills shortage of nearly 200,000 workers by 2018, but I’m not so certain that will hold true. For companies willing to make the leap to cloud services, there will be a lot of companies willing to make big data as easy as paying your bill every month.

Image courtesy of Richard Croft.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Big data 2013: key trends and companies to watch

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

Raj De Datta

The problem right now, BloomReach Co-Founder and CEO Raj De Datta told me, is that companies just cannot know how to best present their product catalogs or other content in a way that best aligns with what customers are looking for. In fact, he said, less than 25 percent of web pages see any traffic from natural search or paid search in any given month. Companies are missing out on large swaths of customers because they can’t display their content in a meaningful manner, and the problem only gets worse as content volumes grow.

“Understanding relevance of content to the way people express themselves turns out to be a difficult problem,” De Datta said.

In BloomReach’s view, this ultimately boils down to a data problem: if a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities. BloomReach will analyze all of them and figure out the optimal methods for organizing and describing products on web pages to ensure customers want to click.

According to De Datta, its early customers — which include household names such as Neiman Marcus, Crate & Barrel, Orbitz and Williams-Sonoma — see 75 percent of the pages getting search traffic within a month. He said BloomReach generates 80 percent average incremental traffic for users.

How does BloomReach work its magic? With lots of big data, of course:

• De Datta said the company runs 1,000 Hadoop jobs a day that interpret 1 billion web pages and 1 billion consumer interactions.
• It has developed all sorts of algorithms for tasks such as determining relevance and optimizing the connections between pieces of data.
• It has a library consisting of more than 12 million two-word synonyms, which the company claims is 73 times more words than the Oxford English Dictionary contains.
• BloomReach even runs regular Monte Carlo simulations, a complex but effective method for determining the possible outcome of events involving many loosely coupled variables.
• Two-thirds of BloomReach’s engineering team have Ph.D.s in computer science.

BloomReach calls the entire package of big data techniques its Web Relevance Engine, and it now powers three distinct products. There’s BloomSearch, the company’s flagship product that figures out and creates the most-relevant web pages based on what consumers are searching for. As of Wednesday, there’s also BloomLift, which takes consumers clicking on search ads to the most-relevant pages rather than static landing pages (which have 55 percent bounce rates) and also tells companies what search terms they should bid on, and BloomSocial, which helps create pages that consumers are more willing to share.

As an example of the latter, De Datta suggested that customers searching for “umbrella” might actually be concerned with planning a picnic, and an “experience” page dedicated to everything-you-need-for-a-picnic might make that customer more willing to share it socially.

BloomReach, it appears, has a solid grasp on the challenge of figuring out what consumers actually want, but there is no shortage of use cases for big data, nor of techniques for doing it. We’ll be discussing many of them over two days at our Structure: Data event next month in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• Cloud computing and trickle-down analytics

Big data is hot, but infrastructure-level platforms such as Hadoop, which focus on storage and processing, still need help to take them into the mainstream. They need a killer app or two that will let companies analyze, visualize and act on all that data without hiring a team of Stanford Ph.Ds, or that will let developers write big-data apps without having to reinvent the wheel.

Here are five startups (in alphabetical order) either in stealth mode or just out of it that could help take Hadoop and its ilk to the promised land.

1. BloomReach

The stealth-mode BloomReach is taking a very targeted, very hands-free approach to big data for its customers. It’s offering a SaaS-based product that job listings say is for “helping leading online businesses uncover the highest quality, most relevant content sought by their consumers, when and where they want it.” Founded by a team with roots at Google, Cisco, Facebook and Yahoo, among other companies, BloomReach has, according to one estimate, about 160 customers — all of them among the top 10,000 websites, and most of them in the retail space. Among its core technologies and methods are Hadoop, Lucene, Monte Carlo simulations and large-scale image processing.

2. Continuuity

Continuuity, the just-launched stealth-mode startup by former Yahoo VP and chief cloud architect Todd Papaioannou, wants to make it easier to build applications that can leverage both cloud computing and big data technologies. As Papaioannou told me recently, most developers shouldn’t have to go through what Yahoo, Facebook and others did in order to write large-scale, data-driven applications. He also said “the data fabric is the next middleware” and noted that the company name is a play on “continuum.” You figure out what it’s up to.

3. Odiago

Odiago is the brainchild of Hadoop and analytics experts Christophe Bisciglia and Aaron Kimball, and aims to improve the state of web analytics. Its first product, Wibidata, which is in private beta, lets websites better analyze their user data to build more-targeted features. It’s built atop Hadoop and HBase, but also plugs into companies’ existing data-management and BI tools. Current customers include Wikipedia, RichRelevance, FoneDoktor and Atlassian (with whom it shares office space).

4. Platfora

Platfora, which launched in September with $5.7 million in funding, wants to make big data analytics accessible to the masses. Founder and CEO Ben Werther, formerly of Greenplum and NoSQL startup DataStax, told me when Platfora launched that its intuitive, visually stunning interface will make Hadoop-based analytics so easy even a history major could use it. Platfora’s product isn’t available yet, but the company is currently hiring, with an emphasis on frontend and user-experience skills.

5. SkyTree

Skytree is probably the stealthiest of the group, but it’s also is one of the more ambitious — because it’s trying to bring high-performance machine learning to mainstream companies. Machine learning is an impressive technique in which the system itself gets smarter as it digests more data, but it usually doesn’t find its way out of research environments or cutting-edge analytics teams. Skytree is putting together an impressive team, including co-founder Alexander Gray, who also teaches machine learning at Georgia Tech and spent six years at NASA’s Jet Propulsion Laboratory. The company will officially launch later this quarter.

We’ll be addressing many of the issues these companies are trying to resolve at our Structure: Data event that takes place March 21-22 in New York City. Founders from Continuuity, Odiago and Skytree will be speaking at the event, as will dozens of other data visionaries from companies such as IBM, Google, @WalmartLabs and Hortonworks.

Feature image courtesy of Flickr user jurvetson.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data