Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Q4 Wrap-up: SOPA and the future of digital content
• Social third-quarter 2012: analysis and outlook
• Who and what to watch in the new era of the living room

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q2: Big data and PaaS gain more momentum

C.O.R.E. stands for Content Optimization and Relevance Engine, and it’s the system Yahoo uses to personalize the homepages for Yahoo users. (Psst … it also personalizes advertising. You know, like Google is currently being taken to task for.) Yahoo is touting some interesting data points on C.O.R.E.:

• Yahoo’s homepage clickthrough rate has increased 300 percent since implementing C.O.R.E.
• Every hour, C.O.R.E. processes 1.2 terabytes of user data. According to Yahoo, that’s the equivalent of 644,245,094 printed pages.
• Every day, C.O.R.E. personalizes 2.2 billion pieces of content.
• As a result of all this, more than 300 articles a month on Yahoo’s homepage receive more than 1 million clicks.

But if you’re not into big data porn, perhaps the C.O.R.E. visualization tool will just let you see how you stack up against the rest of your demographic (that happens to sign into their Yahoo accounts). As a male between the ages of 25 and 34, I should be reading a lot more sports stories than I am. Females of the same age range, they’re most concerned right now with celebrities, weddings and beauty treatments. Only 16 percent of those concerned about Peyton Manning’s arm are women, but the exact same percentage of men are reading about Scarlett Johannson’s look-alike twin.

Women, it appears, also spend more time in each story. Looking at men’s top stories, very few readers spend more than 2 minutes in any given piece.

C.O.R.E. relies heavily on the Hadoop framework, which lets companies like Yahoo (which actually played a big role in Hadoop’s development) store and process the terabytes of unstructured data that users generate as they click their ways across the web. According to a Yahoo spokesperson, C.O.R.E. “leverages Hadoop to distill this information into a content personalization model, which is updated every 5 minutes based on the most recent data.” Yahoo researchers often tout that Hadoop is “behind every click at Yahoo.”

To learn more about Hadoop, check out this post explaining it in more detail. To learn more about how some companies are pushing the bounds of analytics on the web and elsewhere, check out this story. Or, just attend our Structure: Data conference in New York next month, where veterans from Yahoo, Google and other data-centric companies will talk about where the big data movement is heading.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout

Details about how, exactly, WibiData goes about letting users do web analytics have been sparse, but co-founder Aaron Kimball, who will present at our Structure: Data conference next month, explained some of it in a blog post on Monday. The post is fairly technical, but the gist is that WibiData leverages Apache Hadoop, HBase and Avro, as well as ample proprietary code, to enable both real-time and batch processing of user data. This lets users model customer profiles based on historical data, but also adjust those models in reaction to real-time activity on the site.

Here’s how Kimball describes the problems WibiData addresses:

Data about users has challenges associated with it that you don’t necessarily see with other large-scale data.

• To analyze users, you need to digest large volumes of log-oriented transactional data as well as more concise profile data
• You need to serve recommendations and other derived data interactively
• There’s a mix of batch (offline) and on-the-fly calculations required to deliver recommendations at web speed

WibiData is designed to store this transactional data side-by-side with profile and other derived data attributes. Keeping data logically and physically close enables high-performance analysis of the entire data picture surrounding a user.

FoneDoktor, without Wibidata

In some cases, as FoneDoktor’s Alex Loddengaard explained in a December blog post, WibiData can obviate the need to maintain a Hadoop cluster and a separate online transaction processing system (OLTP) because WibiData provides both capabilities. It does this by using HBase as the real-time data store for transactions, and by incorporating a programming framework that’s abstracted from MapReduce so users can perform either batch or real-time analyses.

Where Avro comes in is for adding fields to data records, or adjusting schema, without affecting existing processes that have to access that data. As Kimball explains, “Does your web site track a new cookie? This can be added as a new field. But even though you start collecting that new data, your existing analysis pipelines can treat records like they always did; programs that don’t yet know about the new cookie are still compatible with both the old records already collected, and the new records with the additional field.”

Its data-management methods and machine-learning libraries for capabilities such as content recommendation make WibiData ideal for web-user data, but Kimball points out it’s also a good fit for “mobile, online gaming, healthcare, finance, and several other industries.” However, WibiData is just one of many startups looking to parlay its founders’ Hadoop expertise into a higher-level analytics product that does things Hadoop alone without requiring deep Hadoop or analytics expertise on the customer end. Good thing there’s plenty of data to go around.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Scaling Hadoop clusters: the role of cluster management
• The importance of putting the U and I in visualization

Big data is hot, but infrastructure-level platforms such as Hadoop, which focus on storage and processing, still need help to take them into the mainstream. They need a killer app or two that will let companies analyze, visualize and act on all that data without hiring a team of Stanford Ph.Ds, or that will let developers write big-data apps without having to reinvent the wheel.

Here are five startups (in alphabetical order) either in stealth mode or just out of it that could help take Hadoop and its ilk to the promised land.

1. BloomReach

The stealth-mode BloomReach is taking a very targeted, very hands-free approach to big data for its customers. It’s offering a SaaS-based product that job listings say is for “helping leading online businesses uncover the highest quality, most relevant content sought by their consumers, when and where they want it.” Founded by a team with roots at Google, Cisco, Facebook and Yahoo, among other companies, BloomReach has, according to one estimate, about 160 customers — all of them among the top 10,000 websites, and most of them in the retail space. Among its core technologies and methods are Hadoop, Lucene, Monte Carlo simulations and large-scale image processing.

2. Continuuity

Continuuity, the just-launched stealth-mode startup by former Yahoo VP and chief cloud architect Todd Papaioannou, wants to make it easier to build applications that can leverage both cloud computing and big data technologies. As Papaioannou told me recently, most developers shouldn’t have to go through what Yahoo, Facebook and others did in order to write large-scale, data-driven applications. He also said “the data fabric is the next middleware” and noted that the company name is a play on “continuum.” You figure out what it’s up to.

3. Odiago

Odiago is the brainchild of Hadoop and analytics experts Christophe Bisciglia and Aaron Kimball, and aims to improve the state of web analytics. Its first product, Wibidata, which is in private beta, lets websites better analyze their user data to build more-targeted features. It’s built atop Hadoop and HBase, but also plugs into companies’ existing data-management and BI tools. Current customers include Wikipedia, RichRelevance, FoneDoktor and Atlassian (with whom it shares office space).

4. Platfora

Platfora, which launched in September with $5.7 million in funding, wants to make big data analytics accessible to the masses. Founder and CEO Ben Werther, formerly of Greenplum and NoSQL startup DataStax, told me when Platfora launched that its intuitive, visually stunning interface will make Hadoop-based analytics so easy even a history major could use it. Platfora’s product isn’t available yet, but the company is currently hiring, with an emphasis on frontend and user-experience skills.

5. SkyTree

Skytree is probably the stealthiest of the group, but it’s also is one of the more ambitious — because it’s trying to bring high-performance machine learning to mainstream companies. Machine learning is an impressive technique in which the system itself gets smarter as it digests more data, but it usually doesn’t find its way out of research environments or cutting-edge analytics teams. Skytree is putting together an impressive team, including co-founder Alexander Gray, who also teaches machine learning at Georgia Tech and spent six years at NASA’s Jet Propulsion Laboratory. The company will officially launch later this quarter.

We’ll be addressing many of the issues these companies are trying to resolve at our Structure: Data event that takes place March 21-22 in New York City. Founders from Continuuity, Odiago and Skytree will be speaking at the event, as will dozens of other data visionaries from companies such as IBM, Google, @WalmartLabs and Hortonworks.

Feature image courtesy of Flickr user jurvetson.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data

Big data has gotten very big if the talking heads at the World Economic Forum in Davos, Switzerland, are talking about it. And they are talking about it.

The big data phenomenon refers to the explosion of data of all types — location coordinates churned out by cell phones and GPS, machine data from manufacturing gear, consumer data from Twitter and Facebook. Just a fraction of that information resides in traditional databases. It’s not only too much to get one’s head around but also overwhelms traditional database and analytics tools, giving rise to a new generation of computing technologies: the Hadoop data framework, NoSQL databases and big data analytics.

The overriding issue is finding the right data, applying the right analytics to it, and letting it deal with real-world problems. And the World Economic Forum’s elite attendees, including prime ministers, corporate titans and star academics, purport to address the biggest of big problems: climate change, the income gap, food inequality, public health crises, Iran.

One session, titled “Decoding the data deluge,” chaired by Jan Hesthaven, a professor of applied mathematics from Brown University, deals with the upside opportunity these huge data sets can enable.

From the course description:

It’s a funny thing about data: If you look up ‘data’ in the literature, it invariably comes laden with negative baggage – words such as ‘overload’ and ‘deluge’. With terms such as these, no wonder scientists and scholars alike shy away! But instead of resisting large data sets, we should be embracing them. Multinational companies are learning how to work with large data sets, and they’re getting better at it all the time. But mid-size and smaller businesses don’t have that expertise. Yet the largest companies will benefit greatly if smaller businesses, many of them key to their supply, production and workforce, can, too, handle massive data. Such ease of data handling can grease the wheels of global commerce.

That is just one of several sessions touching on the topic. Another, “Personal data: the ‘new oil’ of the 21st century,” dealt with the proliferation of personal data generated by new intelligent devices, networks and software and how to build a personal data ecosystem that will “spur economic and societal value without undermining privacy and civil liberties.” Another session, “Convergence on the go,” attacked the consolidation of data voice, video and cellular communications services and its impact on society.

There is also a report, Big data, big impact: new possibilities for international development,” which outlines the impact the collection and proper application of big data can have on financial services, education, agriculture and health care.

According to the report:

Data collected through mobile devices, whether captured by health workers, submitted by individuals, or analysed in the form of data exhaust, can be a crucial tool in understanding population health trends or stopping outbreaks. When collected in the context of individual electronic health records, this data not only improves continuity of care for the individual, but it can be used to create massive datasets with which treatments and outcomes can be compared in an efficient and cost effective manner.

Big data is good, except when it’s bad

All of this discussion in the Alps is happening at a time when the opportunities of big data also raise privacy concerns that are playing out internationally, as discussed by GigaOM’s Derrick Harris here.

The issue of who owns what data is huge, and there are factions fomenting to wall off data because of political concerns. Telcom powers in France and Germany, for example, want to build homegrown data centers that would be impervious to U.S. law enforcement requests for customer data. Their concern is that the U.S. Patriot Act can force U.S. companies running cloud-computing sites in Europe (or elsewhere) to turn over customer data if U.S. authorities suspect terrorist activity. That means American IT powers like Microsoft, Amazon and Google that run cloud operations out of European data centers could be compelled to disclose this information. That specter led some European entities to eschew cloud computing services from U.S. companies. In December, British defense contractor BAE backed out of a Microsoft Office 365 deal, citing this issue.

On Jan. 18, U.S deputy attorney general Bruce Swartz tried to calm the waters, saying that the Patriot Act does not supercede existing international trade agreements. A few days later, the Electronic Freedom Foundation, an advocacy group, debunked Swartz’s claim:

While the DOJ may spin its position one way to try to appease foreign audiences, its actual position is quite clear where it really matters: in US courts when it is trying to access subscriber information held by US-based cloud computing services. Indeed, the DOJ’s position in its court filings is that very little, if any, privacy protection is available against US government access to the records of users of US-based cloud computing services.

Whatever happens in this data privacy and protectionism battle, it is clear that big data is very big indeed. Maybe too big. Big data has been featured in recent issues of mainstream publications including Esquire. A contrarian might say this mainstream hype could also mean the big data bubble is about to burst.

Photo and artwork courtesy of Flickr user World Economic Forum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms
• Why service providers matter for the future of big data

This deal is just the latest evidence that while collecting big data is important, the ability to put that data into formats that make sense to business users is equally critical.

ParAccel builds a fast columnar database specifically designed to analyze lots and lots of information. In that arena, it competes with Vertica, Greenplum, Netezza and Aster Data, once-independent companies that were all acquired in the past two years by Hewlett-Packard, EMC, IBM, and Teradata respectively.

Birst CEO Brad Peters maintains that the Birst/ParAccel technology duo will compete with high-end solutions like those and SAP’s HANA analytics appliance at a fraction of the cost.

Target uses include customer service call centers or big telco billing applications, he said.  ”This is for any jobs where you have a really high volume of interactions that can generate hundreds of millions or billions of data points,” that need to be sorted out, classified and visualized, he said.

The partnership means Birst will use ParAccel technology as part of its solution, and make Birst available to ParAccel customers. A ParAccel spokeswoman said the two companies have done a lot of integration work, that ParAccel is using Birst internally, she said.

ParAccel also partners with MicroStrategy, but MicroStrategy’s technology suits more complex enterprise rollouts that take time to sell and implement while Birst’s SaaS capability suits it for quicker, easier sales cycles.

Birst, which offers its business analytics both in an on-premises appliance and as software-as-a-service, competes with Pentaho, an open-source analytics provider and Jaspersoft, both of which also work with ParAccel databases.

It makes sense for database companies to partner with analytics players but given the amount of consolidation that’s already gone on in these worlds, it’s probably just a matter of time before some of these smaller analytics companies join the ranks of the acquired.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Red-Hot Data Warehouse Market: Who’s Buying Next?
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

The United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts to analyze big data and make decisions based on their findings. 

Stephen Gold, vice president of worldwide marketing at SPSS, an IBM company, talked with me about how people and educators should be thinking about the opportunities that this shortage presents and how people can become data-savvy managers (and individual contributors). He pointed out that there is an “explosion of information” in almost all organizations and functions:

Think about all the activities that are being tracked today, and not just the amount, but the type [much of it unstructured] and it’s doubling every two years. Originally, people said ‘that’s just the social media,’ but there is unstructured data in surveys, call centers… What if we could put that to work?

There is a void of skills: both deep analytical skills — the heavy lifting of data mining — but also for the data savvy manager. If I’m in finance, I need analytics for risk; if it’s supply chain, then I need optimization; for marketing, customer segmentation. So we’re seeing this demand and this need.

And the need is starting to push out. People in a variety of roles are starting to understand the value of being data-savvy and how the different kinds of data and analyses can be used to influence behavior, improve organizational policies and practices, or pick the best answer:

• Predictive analytics: Using past and current data to answer questions about the future
• Scorecarding: Tracking business metrics against strategic and operational outcomes for better decision making
• Dashboards: Real-time presentations and aggregations of relevant data
• Social analytics: Metrics from social media and networks
• Content analytics: Assessment of available content and how it is used
• Web analytics: Tracking traffic and key words used to find particular sites

According to Gold, universities can provide students with data-savvy backgrounds through four steps:

1. Extend the current course curriculum with insights and exercises. People need to be comfortable with information and analytics. This could be basic applied statistics. Moves toward greater visualization of the data with drag and drop interfaces that put the heavy technical issues into the background are helping.
2. Develop specific courses that focus on the applications for marketing, finance, social networks, and operations. There needs to be a basic understanding of natural language processing to know the kinds of questions that can be asked — but the focus needs to be on the questions and problems to be solved.
3. Create more Centers of Excellence that are cross-college collaborations: Computer Science + Business for example.
4. Establish degree programs: DePaul, Carngie Mellon, Northwestern, and many other schools are formalizing the background for data savvy managers — as are community colleges and continuing eduction providers.

Our discussion stayed focused on questioning skills, not skills required to run specific tools or analyses. Gold described an IBM collaboration with Yale’s Center for Customer Insights to create student learning opportunities with real-world data and enterprise analytic software. Even though the students weren’t especially quantitatively-oriented, they were able to understand and digest the technology, understand the problem the customer was having, and suggest “deep meaningful validation points” to the organization.

(On the technical side, IBM is offering IT professionals free bootcamps to get up to speed.)

Business analytics took a huge jump in 1979 with VisiCalc bringing spreadsheets to personal computers. We seem to be poised for a similar jump with data savvy managers needing to open their minds to modern massive, unstructured, and linkable data. Events such as GigaOM’s Structure:Data conference provide opportunities to see the moves leading companies are making and better position yourself for a role as a data savvy manager.

Image courtesy of Flickr user afagen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Why service providers matter for the future of big data
• A near-term outlook for big data

The result is new IBM-branded “Commerce-as-a-Service” and social media marketing cloud services that meld Sterling’s supply chain know-how, Coremetrics web analytics and Unica’s marketing management technology with IBM’s middleware underpinnings.

Previous to the IBM acquisition, the Coremetrics offerings were all cloud based, while the Unica and Sterling technologies were partially there. IBM took those technologies and best practices, tied them into its backend systems and optimized it all to run especially well on IBM Power7 hardware, said Craig Hayman, GM of IBM’s industry solutions effort.

What is net new to IBM’s cloud ecosystem is supply management, logistics management and payments technologies as well as social media tracking and Sterling’s cross-channel sales and customer integration technologies, Hayman said.

A big part of this push addresses the pressure many businesses feel to use social media not only to promote their products and services but to monitor and track customer reaction to them, according to analyst Laurie McCabe, co-founder of the SMB Group.

IBM built out its software services portfolio for years to attack what it sees as a $20 billion market.

In some cases, it’s playing catch up with innovative first movers like Hubspot, which helps companies build in-bound marketing plans that incorporate relevant Facebook and Twitter chatter as part of an overall marketing strategy.  The Unica acquisition helped fill that gap.

Clearly, IBM would bristle at the notion that it’s playing catchup. “We launched the Smarter Commerce initiative in March and [seven] months later we’ve done a lot. We are on a tear,” Hayman said.

The new Commerce-as-a Service entry offers cloud-based configuration, pricing and quoting capabilities to streamline a business’s “quote-to-cash” process that is now often ad hoc or bolted into existing ERP and supply chain management systems.  IBM is trying to take that sales tracking process into the Software-as-a-Service realm.

IBM and software rivals Oracle and SAP are all navigating the tricky straits between on-premises and cloud-based software world and like most of these competitors, IBM is hedging its bets. The bulk of today’s offerings will also be available for on-premises deployment.

Image courtesy of The Library of Congress.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• What Enterprise Software Vendors Could Learn from the Consumer Space
• The fourth quarter of 2012 in cloud

The Metricly system currently plugs into 15 sources of data: things like Google Analytics, MailChimp and Twitter. In many cases, the Metricly interface makes it easier to parse data than the source itself (though I’m sure that’s not always true). For instance, Metricly can cross-match data for up to 60 sites in a single graph, where Google Analytics maxes out at one.

The main Metricly interface is a stack of line graphs configured for your needs, with recent trends highlighted. Metricly currently doesn’t do a lot of analysis or make recommendations; its main purpose is just getting all that data into one place. The service sends weekly summary emails, and it will also offer notifications for any unusual data trends soon.

Metricly was founded by Devin Poolman, Jason DeFillippo and Paul Cloutier, who were all previously at 8020 Media (creator of JPG Magazine). The company is angel-backed by investors such as Paige Craig and Wasabi Ventures and is currently raising funding. For now, Metricly services are all free, but premium plans for additional services like connecting to your own database for a custom feed will eventually cost $50-100 per user per month. So far, about 1,300 users have signed up (the private beta launched earlier this year), with 40 percent logging in every month. Poolman said his target customers are startups, e-commerce and content companies, and teams in larger organizations. Obviously a big focus for Metricly is integrating more data sources, and to that effect, DeFillippo has created a plug-in API that will be opened to any service that wants to join. It’s a race to make the product indispensable for its users.

A significant new expansion for Metricly is a partnership with Etsy that launched a few days ago. Etsy sellers previously had very little insight into their transactions besides a simple list of sold items. Metricly is now providing Etsy sellers with free dashboards to break down their total sales by time and location, average sales price, repeat customers versus new customers, and other filters.

Related content from GigaOM Pro (sub req’d):

Big Data Marketplaces Put a Price on Finding Patterns