Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• The fourth quarter of 2012 in cloud
• IT spending update, third quarter 2012

Check out the rest of our Structure Europe 2012 live coverage here, and a video recording of the session follows below.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Real-­time query for Hadoop democratizes access to big data analytics

Indeed, our ability to analyze this fast-growing mountain of data will increasingly shape the world that we live in. And not just in smaller ways, like whether we can find that perfect pair of jeans without ever setting foot in a store, or how quickly we discover that great new indie band we had never heard of. It will also enable use to tackle some of the biggest problems of our time, like making our healthcare system more efficient, making cities and governments more effective, or even reducing world hunger and the spread of disease.

We write about the power of big data on GigaOM every day. Just in the last few months, our data maven Derrick Harris has done stories on topics as wide ranging as bullying, intelligent transportation and genomics, showing how in each case big data is helping to power new breakthroughs. Starting today, in recognition of the growing importance of data in spurring new insights and even new technologies, we are rolling out a dedicated data channel. Now you’ll be able to find the best stories about the data revolution — from the most interesting use cases to the most important technologies to the smartest data scientists — all in one place

The launch of this new channel coincides with something of a perfect storm with big data over the last few years. Because so much of what we now do on the web — and even in our physical lives — generates new data, we’re swimming in the stuff. We have far more data at our disposal than we’ve ever had before, and we continue to collect it at a furious rate.

At the same time, it’s never been cheaper or easier to analyze large amounts of data. What used to take hours or days now takes minutes or even seconds – in some cases, the data synthesis is taking place in real time. Meanwhile, someone with absolutely no computer science training can create a chart to visualize a data set in just a few clicks using a variety of free web services.

All of this means that the business opportunity with data is palpable. The biggest internet brands of today — companies like Google, LinkedIn, Twitter, Facebook, Amazon and Netflix — are where they are in large part because of their ability to harness big data. And it’s not just the heavyweights of the web that are taking full advantage of this moment; almost every company of any note is thinking about how to use big data to take their business to a new level.

Data is king, and our new data channel will be the place to watch how it’s changing not only the internet but the world around us, too.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Dissecting the data: 5 issues for our digital future
• Finding the Value in Social Media Data

In a study on personal data released this week with the Boston Consulting Group, the report said that while online retail in the Group of 20 countries could reach $2 trillion by 2016, consumers’ perception of trust plays a significant role in enhancing or eroding that value. With more trust, online retail could grow to $2.5 trillion by 2016; with less, it could reach just $1.5 trillion by the same time, they said.

“Given that this $1 trillion range is from just one small part of the broader personal data ecosystem, it provides an indication of the magnitude of the potential economic impact when other sectors (health, financial services, etc.) are considered – potentially in the tens of trillions of dollars,” the report says. That ecosystem includes individuals, as well as data collectors, marketers, data brokers, publishers and other organizations with an interest in using personal information.

As our volume of digital “exhaust” grows (currently, 10 billion text messages and 1 billion blog and social network posts, according to the report), so does consumer concern around privacy. Just this week, (now public) Facebook was hit with a lawsuit over privacy and, over the past few years, regulatory investigations over privacy violations have climbed.

In this report – and a highly cited report on the topic last year – the World Economic Forum calls personal data “an emerging asset class.” But to really extract its value, the organization argues, public and private institutions need to rethink how they do business so that consumers get more protection, rights and opportunities to hold organizations accountable when it comes to their data.

For a data industry making billions of dollars from personal information ($2 billion from third-party consumer data alone, according to Forrester Research) that’s a tall order. But a handful of companies, from startups like Personal and Azigo to industry giants like Experian, are testing new business models that give consumers more ways to control – and, in some cases, receive compensation for – their data.

As Personal’s Shane Green told my colleague Ryan Kim  earlier this month, “A new model is emerging for personal data. You’ll want a simple, clear answer about what data companies are capturing about you, how they’re using it and what’s in it for me.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How HR can make the case for workforce analytics
• The quantified self: hacking the body for better health
• GigaOM Research highs and lows from CES 2013

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Takeaways from the second quarter in cloud and data
• Amazon’s DynamoDB: rattling the cloud market

Better tools for big data.

So, if it’s safe to assume that big data is real, and that you should be investing, where do you start and what should you expect as you go through the adoption process? Big data today, is what the web was in 1993. We knew the web was something and that it might get big, but few of us really understood what “big” meant. Today, I believe we aren’t even scratching the surface of the big data opportunity.

A good example of potential big data use models can be found here.

Current issues with adoption.

There are a number of issues that will affect your ability to successfully adopt and make best use of a big data solution, but the three I believe are most critical are:

1. Useable enterprise tools — The tools that will allow any business to fully utilize big data aren’t ready.
2. Lack of staff expertise — The availability of data scientists or folks with a similar background is limited at best.
3. Data gravity — As Dave McCory pointed out in his post on data gravity, where data is created/sent is where it ends up being used. The applications and people need to come to the data as I explained.

How will these adoption issues affect big data as a business opportunity?

Useable enterprise tools — The current suite of products include Greenplum, Cloudera’s Hadoop and others, which are making headway in many large enterprises. However, these tools are still new and generally require large technical teams trying to solve issues for companies like eBay and Sears. A smaller company would be less likely to gain the appropriate return on investment, because of the high complexity of implementation combined with low overall volume.

Lack of staff expertise — This area is similar to enterprise tools. Even if you’ve got 10 people working on the refinement of the system, it’s likely going to boil down to that one wizard/expert who can work magic with your data. Putting a large number of people on the problem won’t guarantee success.

Data gravity — Considering the strong possibility that most organizations will struggle to fulfill the promise of their big data strategy with internal resources, we are likely to see a proliferation of services from various cloud providers. My concern here is that the use characteristics of Big-Data-as-a-Service aren’t being thoroughly examined.

The questions and big picture concerns.

Thanks to complex implementations, the data divide could grow.

I see big data quickly becoming a competitive advantage, which is the good news. However, I see significant parallels between the ability to pay for and adopt big data and the first decades of the mainframe. Only a few companies could afford mainframes, and those companies that could afford them were able to develop real advantage. With the introduction of the internet and cloud computing we have moved to a much more democratic model of IT availability, but big data has the potential to re-insert that gap between the haves and the have-nots.

When thinking about democratizing the use of data, the following questions come to mind. They can relate to your implementations, but also are worth thinking about in general. They are:

• Where will your data reside?
• How will you get your data to the service?
• Will tools be delivered across the wide area network (WAN) to be run locally against your in-house data?
• How will you collect and capture your own data?
• If you store your data with a service how often will you use it? Or will you likely be paying to keep it handy for rare uses? (I call this the problem of “Data in Waiting”)
• If you store your data on the service providers storage such as on S3 but you don’t want to pay for it when it’s not in use, will you delete it? How will you know it has been deleted?
• If your big data is running in a public cloud, what tools, and strategies will you use to make that data available to customers and other applications (integration)?
• Will big data cause you to buy more WAN capacity?
• Will big data cause you to rethink your enterprise application strategy?

So what’s the solution to bring data to as many businesses as possible?

To make big data available to everyone we need quite a few things to happen. We need to figure out simple use cases for data to solve common problem sets. Then we must make those available to developers so they can build tools that make solving those set problems easy. We need to continue to push the boundaries of cost-effective disk storage and network capacity, or provide ecosystem environments that allow for direct access over a private network. In an ideal world we will do both.

We’ll know big data has arrived when the use of the service is integrated into common business software tools that are used by the majority of your businesses employees. Also key will be the ability of any knowledge worker to run their own questions/queries against internal and external data sources. The average business won’t be able to call big data truly successful or accessible as long as its usability is being defined and managed by a small disconnected team of IT scientists.

Mark Thiele is executive VP of Data Center Tech at Switch, the operator of the SuperNAP data center in Las Vegas. Thiele blogs at SwitchScribe and at Data Center Pulse, where is also president and founder. He can be found on Twitter at @mthiele10.

Interested in big data? Come talk to us at our Structure Data event next month in New York City.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Why the big data startup boom will likely be short-lived
• How HR can make the case for workforce analytics

Dropbox, Box and other cloud-based backup tools for desktop and mobile devices have already been wildly successful. Similar tools, such as Riverbed, StorSimple and Ctera, are available for the server market, but they take a hardware-centric approach that has failed to garner a large market.

When Amazon Web Services launched AWS Storage Gateway last month, the move seemed entirely logical, almost expected, given the enormous appetite that desktops have shown for cloud-based data sharing and backup.

But is this enough? Most companies have become pretty good at copying data. The real challenge is not restoring data, but going a level higher and recovering services when disaster strikes. That’s particularly important in the era of IT consumerization, where end users expect to access data through applications of their choice on the device of their choice. In other words, it is useless to protect data and not protect the applications that exploit it.

This is where current backup tools fall short. End users are concerned about how long the outage is going to last, not whether the data is safe (which they assume is a given). Disaster recovery is about minimizing downtime. The cloud has huge potential to make an impact here. By managing and scheduling all of the components involved in service delivery, the cloud could turn recovery time objectives into guarantees.

We are at a fork in the road: backup and disaster recovery are going to be two separate processes.

Backup will be used to quickly restore data — including files, mailboxes, attachments and database tables — and to keep auditors happy about long-term data availability.

Disaster recovery will focus on continuity and service recovery, not data restoration. It will orchestrate all of the components involved in service delivery — from storage to hypervisors, operating systems, databases, middleware and applications — across collaborating clouds. This means that when a cloud disappears, another cloud will be ready to take over at the push of a button. And we’ll be 100 percent certain about the maximum outage time. There will be no need to worry about doing disaster recovery exercises, because clouds will do them on their own, continuously and accurately.

Carlos Escapa is the CEO of VirtualSharp Software. Previously, he was a senior executive at VMware in Europe, where he managed VMware’s field operations in France, Portugal, Spain, Italy and Greece. 

Image courtesy of Flickr user Kevin Burkett.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• What Enterprise Software Vendors Could Learn from the Consumer Space
• Infrastructure Q1: Cloud and big data woo enterprises

Sure, we can talk about virtualization platforms, supported operating systems and SLAs (if SLAs even matter). However, I would argue that one of the most critical determinations of the placement of cloud workloads is also one of the weightiest: the law. I’ve called this out before, but I think there are some new elements worth exploring given recent controversy over applicable laws in the European Union and the United States.

Here’s the thing. For the most part, cloud-related laws on the books today or in the works right now are almost entirely about data, and data has “gravity,” as my friend Dave McCrory has pointed out. This is true in the sense that the more important it is, the more likely services and applications are going to move to the data, rather than vice versa.

As McCrory notes:

Consider Data as if it were a Planet or other object with sufficient mass. As Data accumulates (builds mass) there is a greater likelihood that additional Services and Applications will be attracted to this data. This is the same effect Gravity has on objects around a planet. As the mass or density increases, so does the strength of gravitational pull. As things get closer to the mass, they accelerate toward the mass at an increasingly faster velocity.

Services and Applications can have their own Gravity, but Data is the most massive and dense, therefore it has the most gravity. Data if large enough can be virtually impossible to move.

(He has since updated his theory to note that the mass is created by importance, rather than sheer volume, of data. For example, New York Stock Exchange trading data has huge “mass,” and traders routinely build and deploy applications and services as close to that data as possible. U.S. census data is huge, but its importance in most day-to-day activities is relatively less, so people draw census data into applications tied to other, more “important” data.)

So, where does the law fit in? Well, if the law dictates where data can be placed, the the law dictates where that “gravity” will reside, and therefore where workloads will be run to take advantage of that data. You can’t place a workload in a U.S. data center that requires highly personalized data from the EU, or you are breaking the law. So, if you want to “optimize” workload placement, EU law has dictated most of your options.

The irony of all this is that I predicted the importance of the law in the “flow” of applications across the globe in 2008, but it didn’t come to be as I expected. In one of my favorite all-time posts, I wrote:

If law will in fact have such an influence on cloud computing dynamics, it occurs to me that a new cost factor might outshine simple operations when it comes to choosing where to run systems; namely, legality itself. As businesses seek to optimize business processes to deliver the most competitive advantage at the lowest costs, it is quite likely that they will seek out ways to leverage legal loopholes around the world to get around barriers in any one country…

…So, run your registration process in the USA, your banking steps in Switzerland, and your gambling algorithms in the Bahamas. Or, market your child-focused alternative reality game in the US, but collect personal information exclusively on servers in Madagascar. It may still be technically illegal from a US perspective, but who do they prosecute?

It turns out that instead of taking advantage of loopholes and economic advantages in the law, business might find themselves being forced to distribute their applications in specific ways just to stay within the bounds of the same law.

Are there ways out of adherence to data “gravity,” and therefore legal restrictions to workload placement? There may be. Check out McCrory’s follow up post on Defying Data Gravity, for example. But even most of those options are about using distribution to ease the pain of data gravity from a pure technology perspective. They don’t generally address the pain of keeping data within political and legal boundaries while continuing to meet the performance and availability objectives of the users of that data.

As you think about the ways you operate cloud applications, consider the methods you will use to address legal restrictions on data placement, and the related effect that has on application operations. You’ll be way ahead of the game if you do.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Facebook’s tactical retreat on privacy
• Helix Nebula and the future of Europe’s cloud

Pentaho is moving its open-source business intelligence capabilities to the Apache license to make them more compatible with big data technologies. Pentaho’s Kettle extract, transform, load (ETL) technology was previously available under the LGPL or lesser Gnu General Public License.

Apache Hadoop, as its name implies, is offered under the Apache license, as are most of the NoSQL databases that are used to attack tons of structured and unstructured data from multiple sources. ETL tools are used in applications when data needs to be pumped out of (extracted) from a source repository; cleaned up or put into the required format (transformed); and then put into (loaded) the application that will manipulate it.

“We want to get our Kettle ETL engine embedded in big data solutions and this is a good way to do that,” said Doug Moran, co-founder and product manager big data for Pentaho.

Apache projects will not allow LGPL code to be mixed with their code, Moran said. “We partner with different Hadoop distributions, and they really strongly recommended — well they pretty much told us — to do this,” Moran said.

Generally, the difference between the Apache and LGPL licenses is that under the Apache model, a developer can put Apache software into a product and distribute it under any other open-source license as long as the embedded Apache-licensed code is unadulterated — the developer hasn’t “diluted” the rights. With GPL and LGPL licenses, a developer cannot distribute that derivative work under a less restrictive license, Moran explained.

Kettle 4.3, available under the Apache License Version 2.0, can ingest, output, manipulate and report on data from Apache Cassandra, Hadoop HDFS, Hadoop MapReduce, Apache Hive, Apache HBase, MongoDB and Hadapt’s Adaptive Analytical Platform, Pentaho said.

Pentaho’s BI tools compete with offerings from Talend, Qlikview, and  Tableau. The license change takes hold with the new Pentaho Kettle 4.3 release.

Feature photo courtesy of Flickr user opensourceway

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011

To take advantage of that potential, business leaders need to know what steps to take in order to make maximum use of their data asset. Business success with big data is not just about choosing the right cloud technologies or hiring smart data scientists – it’s about creating a business-centric approach that connects a company’s data to its business strategy, enables continual improvement, and follows through to impact processes, margins, and customer satisfaction.

In my experience with big data I have developed seven steps that can help any business leader drive more success with their big data initiatives.

1. Create a strategy for your data

Your data needs a strong strategy, one that connects to and underpins your business strategy and also integrates with department-level accountabilities. Develop a plan for where you want to be at each milestone, define your future capabilities clearly, and describe how your data capabilities will be utilized. Then hold your users accountable to where and how they will use the new capabilities, and what business impact they will drive.

For example, if you can use big data to improve in-store sales, you need to not only work with store managers to define capabilities that connect to their strategy, but also ensure the managers are held accountable to using the data capabilities correctly and delivering the intended impact. Doing so connects both your and their goals to the overall business strategy, helps create more usable capabilities, and ensures any needed iterations will be done jointly. A well-conceived data strategy will give you the most bang for the buck on your data investment.

2. Design for agility

Big data systems are just that…big, which means they tend to be inflexible. A great BI system, by definition, will cause a business to change, which in turn will require the BI system to change. Thus, your systems need to adapt quickly to keep pace with your business. Twelve- or 18-month release cycles are appropriate for certain parts of your system, but 3- or 6-month cycles may be appropriate for others. Carefully analyze each component of your big data system, and design for the right amount of agility you need.

You may decide to build higher levels of automation into the layers of your stack that change slowly, while reserving configuration-based approaches for layers of your stack that need to change quickly. In general, the top layers of your stack (e.g., user interfaces and reporting tools) need to be more agile than the bottom layers of your stack (e.g., data collection and storage), but many exceptions to this rule exist. Only careful analysis and understanding of your current and future uses of data will enable you to make the right decisions on agility. Designing for agility will enable your big data investment to keep pace with, and even lead, your business.

3. Understand latencies

Latency is a challenge in traditional BI systems, and big data only amplifies the problem. Big data solutions tend to be architected first as batch systems, with lower latency capabilities being addressed afterwards. Don’t save latency for last – analyze your key use scenarios in terms of latencies, and connect them clearly to business drivers. Focus on delivering the right latency for each need, including the value being driven, and let those needs drive your design.  Certain low latency needs may require bypassing your big data system temporarily, sharing directly between systems in order to deliver specific scenarios.

For example, if your customers tend to interact with system A and system B in parallel or in quick sequence, these two systems may need to share data directly. The data can then be written into the big data system in time to be used by other systems. Delivering data on a real-time or near-real-time basis can be very expensive; thus, it’s better to think in terms of “right-time” data targeted to each need.

Describe latency requirements in detail, and ensure the business justification is sound. Understanding latencies will enable you to deliver data exactly when it’s needed, while keeping costs under control.

4. Invest in data quality and metadata

Data quality in any system is a constant battle, and big data systems are no exception; however, big data systems require much more automation and advance planning.  You should first ensure that data quality is not treated as a project or initiative, but as a foundational layer of your data stack that receives adequate resourcing and management attention.  Second, build in multiple lines of defense – from data mastering (where, for example, you are creating customer accounts) to data collection (where you are recording all of that customer’s interactions with you) to metadata (where you are organizing and dimensionalizing the data to aid in future reporting and analysis).  Third, automate both the processes that identify and elevate data quality issues, and the measurement and reporting of data quality progress.  Empower your data quality team with tools that solve problems at high scale, such as diagnostic and workflow tools.  Efficient data quality practices will enable your big data system to earn its place as a trusted input for key business processes.

5. Get good at prototyping

The data sizes in most big data systems are too large to work with all at once, so it’s typically wiser to build small-scale prototypes to iron out the wrinkles and ensure you are meeting customer requirements. If you are building complex data integrations, online algorithms, or user interfaces, prototyping allows you to learn at a smaller and less costly scale. What’s more, prototypes can be shared early with your user base, which generates valuable feedback as well as excitement.

Prototyping requires somewhat unique skills that you will need to build and refine over time. Prototypers need to be able to move quickly, figure out new designs and technologies, understand user scenarios, actively solicit feedback, and not be afraid to fail. They need to be creative in their approach to solving problems, while still rooted in sound data mechanics. Why is prototyping better than wireframes or feature lists? Since prototypes are “real,” your users will give you better feedback; at the same time you will also understand some of the challenges you will face as you build the full-scale version. Building a strong prototyping capability will help you increase innovation and speed, while reducing the cost of mistakes.

6. Get great at sampling

Sampling will save you a lot of time if you learn how to do it correctly. There are many use cases for which sampling is an effective alternative to using full census (100 percent) data. Certain needs such as creating personalized experiences for each customer, or calculating executive accountability metrics, are not appropriate for sampling. But for many other needs, sampling is a viable option.

For example, understanding product or feature performance, looking at patterns and trends over time, and filtering for unexpected anomalies can typically be done on sampled data. One approach is to collect 100 percent of the data, but do most of your analysis on samples, and then confirm important conclusions on the full data set. Once you establish process flows to pull sampled data into standard tools such as Excel and/or SQL, you will see analyst productivity increase substantially, which will save you time and money and increase the job satisfaction of your analysts.

To get great at sampling, you need to do three things: first, develop standard sampled data sets that help your analysts address large swaths of business questions, updating them regularly; second, make sure you have at least one highly qualified individual (i.e. a statistician) who can ensure the data is being sampled correctly and results are not misapplied; and third, educate decision makers on the benefits and limitations of sampling so they can get comfortable making decisions with sampled data. The effective use of sampling increases productivity while delivering equivalent business value.

7. Ask for regular feedback

Big data is a learning process, both in terms of managing the data and in driving business value from its contents. Your internal user base is a valuable source of feedback and integral to your learning and development process. Your prototyping program will be a source of feedback, but you should also survey your users and benchmark your progress over time. Areas such as usability, data quality, and data latency are all categories within which users will give you feedback. In addition, you should ask for ad hoc feedback from every level of your stakeholder organizations so they see your commitment to making their business better.

As your data asset’s reputation grows, your stakeholders will give you more and better feedback, which will allow you to develop integrated goals and roadmaps, and drive more business benefit as a result. Regular feedback ensures your big data system is tightly integrated into business decision making, so it can play a lead role in business improvement.

Following the above steps will help you build more effective big data capabilities, saving you time and money, and driving maximum ROI for your business. The big data frontier is here; breaking through it requires an understanding of which steps will help you drive the most impact.

Chad Richeson is the CEO of Society Consulting, a Seattle-based analytics and technology consulting firm that provides business-driven data strategies, solutions, and analytics for its clients. Before joining Society Consulting in 2011, Chad spent 12 years at Microsoft driving analytics and big data solutions for Bing, MSN, Mobile and AdCenter.

Image courtesy of Flickr user Susan NYC.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How HR can make the case for workforce analytics
• Infrastructure Q1: Cloud and big data woo enterprises
• The fourth quarter of 2012 in cloud