GigaOM » Structure:Data 2012

Eventbrite: Hadoop isn’t for everybody

Kevin Fitchard — Thu, 22 Mar 2012 22:52:43 +0000

Open source data warehousing models have a lot of advantages, the ability to scale horizontally and cheaply among them, but traditional warehousing techniques have their strengths as well, said Vipul Sharma, principal software engineer and engineering manager at Eventbrite, which itself manages terabytes of event and user information and uses a combination of MySQL and Hadoop databases, at GigaOM’s Structure:Data.

Open source solutions like the Hadoop distributed file system and HBase NoSQL database are justifiably the hot platforms in data warehousing, but MySQL and other more traditional enterprise data warehousing may still be optimal if a company has a good understanding of where it needs to scale and has big focus on security and reporting.

Hadoop’s open source community still hasn’t worked out the security kinks, and building the dashboards around reporting tools in Hadoop and Hbase is quite difficult, Sharma said. Security and reporting in enterprise data warehouse platforms are much better developed, he said.

Sharma also cautioned companies that they must be willing to commit manpower to support their Hadoop clusters, otherwise they’re better off using a traditional warehouse solution.

“Since [Hadoop and Hbase] are very new, the entire code base has to mature a lot,” Sharma said. “So you end up having to do a lot of debugging yourself. You end up getting into the code base to understand what’s going on. … If you’re not willing to invest in a team around it, then probably Hadoop is not the best for you because the ecosystem is still maturing.”

gigaombigdata on livestream.com. Broadcast Live Free

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The online future is personal, and that requires big data

Mathew Ingram — Thu, 22 Mar 2012 22:22:13 +0000

History shows that the most successful products and services — from the VHS video recorder to the Apple iPad — are the ones that are the most user-centric, and in order to construct those kinds of experiences companies need to use all the data that is at their disposal, WiBiData co-founder and chief technology officer Aaron Kimball told the Structure:Data conference in New York on Thursday.

The problem for many companies is that information about their users is spread across hundreds or even thousands of different fields in various databases, and it’s difficult to put it all together in real time. But doing that successfully is becoming more and more important as services become more personalized.

Kimball, whose company has been funded by Google chairman Eric Schmidt and Ron Conway’s SV Angel group, said that WiBiData’s solution is to put together a platform that combines big-data aggregation and analysis tools like Hadoop and Hbase and Apache’s Avro, which can pull in and make sense of data coming from multiple databases in one location, and then make it easy for companies to extract and make use of that information in real time. In addition to tools that combine those services, WiBiData also has a user interface and dashboard that allow companies to pick and choose which data to use quickly.

Online users in particular have grown used to services like Amazon and Netflix, which provide real-time recommendations based on what a user has done in the past or other elements of their user profile, Kimball said. This not only saves a user time, but exposes them to items they may not have even known they wanted to search for — and thus creates a better user experience.

So if your company is selling hardware and a user puts a hammer in their online shopping cart, you want to be able to suggest that they buy some nails as well. But this has to be done in milliseconds if it is to be useful, and that requires manipulating large amounts of data in real time to update or alter the user’s profile and pull in the new selection.

“We are now capable of collecting and analyzing orders of magnitude more information than at any other time in history,” thanks to the database tools we have, Kimball said, but “the one constrained resource is understanding.” Users of online services now expect to have things personalized in real time, and if your company can’t provide that then you are going to be behind the curve, he said.

Watch live streaming video from gigaombigdata at livestream.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Mobile data a fascinating and scary opportunity

Tom Krazit — Thu, 22 Mar 2012 21:44:52 +0000

We’re walking around with sensors in our pockets: those of us carrying smartphones, anyway. There are huge opportunities for companies to improve existing services and create new ones with the huge amount of data provided by mobile computers, assuming, of course, that you avoid freaking out your users.

“What’s so interesting about mobile data is how personal it is,” said Michael Driscoll, CEO at MetaMarkets, during a session at Structure:Data. It’s not just what you’re doing on your phone or tablet, but when you’re doing it and where you’re doing it: those are variables that aren’t necessarily as important to the traditional desktop-and-Web-based model of computing.

And “the accuracy of what you can capture in a mobile app goes beyond” other data sources, said Raj Aggarwal, CEO and co-founder of Localytics. That’s going to increase as phone makers put more sensors in their devices–say, for health monitoring–and software developers come up with more sophisticated ways to interact with those sensors, he said.

But as we’ve seen time and time again, there are privacy minefields involved with any type of collection of data from mobile devices, the most personal computing device most of us own. Aggarwal advised anyone working with mobile data to give users an easy way to opt out of those services and to be transparent about how data is collected and scrubbed of personally identifiable information.

Watch live streaming video from gigaombigdata at livestream.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Flash, an option for big data performance?

Tom Krazit — Thu, 22 Mar 2012 20:49:07 +0000

Flash memory is not cheap, but it can play a valuable role in helping companies crunch big data faster and might even save them money in the long run.

Scott Metzger, vice president of analytics at flash-memory array maker Violin Memory, argued Thursday at Structure:Data that putting flash memory at the heart of big-data infrastructure is a must for any business that is worried about how long it takes to get results from data analysis. Metzger’s got a dog in this fight, of course–Violin Memory makes storage arrays designed around flash memory–but it’s true that flash memory operates faster than hard disks or tape storage.

In one case study he presented, Metzger said that flash-memory storage improved the latency–the delay occurring when a signal moves from one point to another–by up to 50 times faster compared to storage-area networks based on disks. And because flash memory also operates at a far lower temperature than disks, companies that use flash-memory storage can reduce their power and cooling requirements by 80 percent compared to traditional requirements.

This probably isn’t for everybody: Metzger acknowledged that customers just looking for archive and backup storage don’t need to pay the up-front costs of investing in flash memory.

Watch the livestream of Structure:Data here.

gigaombigdata on livestream.com. Broadcast Live Free

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Another big obstacle to exascale computing: resilience

Kevin Fitchard — Thu, 22 Mar 2012 20:17:13 +0000

Los Alamos National Laboratory is trying to build to an exascale computer, which would be 1000 times faster than Cray’s Jaguar supercomputer and could process one billion billion calculations per second. The man in charge of executing that vision, however, sees a big obstacle toward building a computer with 1 millions nodes, running between 1 million to 1 billion cores. That problem is resilience.

Speaking at GigaOM’s Structure:Data conference, Los Alamos HPC deputy division leader Gary Grider said that the exascale computer has so many parts, that some element will constantly be failing.

“It wouldn’t be worth building if it didn’t stay working for more than a minute,” Grider said. “Resilience is absolutely a must. The way you get answers to science is you run problems on these things for six months or more. If the machine is going to die every few minutes, that’s going to be tough sledding. We’ve got to figure out how to deal with resilience in a pretty fundamental way between now and then.”

Grider and Los Alamos’s technology partners have between 6 and 10 years to work on the problem, and the national lab won’t be alone. According to inside-Data president Rich Brueckner, who moderated the “Faster Memory, Faster Compute” panel Grider spoke on, countries from all over the world are in an exascale race. Brueckner said it’s just as likely as Russia, Japan, China, India or the European Union develops the exascale machine as the U.S.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

What big data really needs is security

Ryan Lawler — Thu, 22 Mar 2012 20:10:59 +0000

There are plenty of benefits from making data available to large repositories. But Dave Asprey, VP of Cloud Security at Trend Micro, said one thing holding enterprises back from putting their data in the cloud is the lack of security of what they’re sharing.

At GigaOM’s Structure:Data conference, Asprey said that when individuals or enterprises think about adding their data to large repositories, there are many things that they need to be able to control who has access to it. That demand largely means adding a layer of policy management that isn’t really available today.

Asprey gave the example of sharing genetic data through a service like 23andMe. The genetic testing could help drug or science research labs, which is great — but Asprey wonders, “Do I really want to share that data with [insurance provider] Allstate?” In that case, there’s less of an incentive to do so.

In addition to policy management, there’s also the issue of ensuring the reliability of data that’s put in the cloud. Asprey gave the example of a diaper manufacturer that did a field trial of a new product in a certain city, whose competitors bought out the product event though there was little actual consumer interest. That hacking, or poisoning, of the data led to the launch of a national campaign for a product no one really wanted.

“If big data isn’t properly secured,” Aprey said, “You can stuff the ballots.” That’s bad news when the data is being used not just to make advertising or product decisions, but when it’s applied to health and national policy decisions.

Watch the livestream of Structure:Data here.

gigaombigdata on livestream.com. Broadcast Live Free

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Never mind the hardware, it’s the algorithms

Tom Krazit — Thu, 22 Mar 2012 19:46:29 +0000

It’s easier to crunch massive amounts of data when you don’t have to reinvent the wheel for every scenario. Sultan Meghjji and his colleagues at Appistry are hoping to make it easier for those who are too busy with their primary jobs to take advantage of the insights that can emerge from sophisticated analysis of big data, he told attendees at Structure:Data on Thursday.

Appistry isn’t a household name, but FedEx, one of the company’s most famous clients, certainly is. FedEx uses Appistry’s computer infrastructure and algorithms to help its employees deliver overnight packages as reliably as possible. And there’s a simple philosophy behind company’s approach: “we’re about moving the work to the data instead of moving the data to the work,” Meghji said.

That type of approach requires sophisticated algorithms, and Meghji bemoaned the fact that graduate students are the ones developing many of these algorithms as research projects. The world of big data needs better algorithms far more than it needs better hardware, he said.

We might get closer to that goal if more people were taught at an earlier age to appreciate and respect the value of big data and data analysis, he said. “Across the professional spectrum, the understanding of data–the value of it, what math matters, what it implies… we do not teach that anywhere with any seriousness.”

Watch the livestream of Structure:Data here.

gigaombigdata on livestream.com. Broadcast Live Free

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why you’ll likely have your DNA sequenced in 5 years

Kevin C. Tofel — Thu, 22 Mar 2012 19:21:56 +0000

Updated. The costs of DNA analysis and genome sequencing are fast decreasing while the amount of data created by such activities is nearing the one petabyte — or a thousand terabytes — barrier. Is it possible then that most people will be likely to have their DNA profile within the next 5 years? Yes it is, according to one expert.

Andreas Sundquist, CEO and co-founder of DNAnexus, suggested this to the entire audience at Structure:Data in New York City on Thursday, adding that “Ten years from now, sequencing will be standard. The test will just be a run of some software.” What makes Sundquist’s claims possible is both the advances in big data techniques and heavy use of the cloud.

Thanks to these two solutions, it now costs a few thousand dollars ~~fewer than $1,000~~ to sequence a genome; the cost had decreased by a million-fold over the years, even as each genome sequence creates a terabyte of data. So far, around 20,000 genomes have been sequenced world-wide, but the pace increases as the cost falls. And next year, Sundquist expects that 100,000 genomes will be sequenced.

Helping to speed up sequencing efforts is access to and the scalability of the cloud. Most hospitals and medical centers don’t have the budgets to create massive data centers to process or store genome data.

By sending data directly to the cloud from a sequencing machine, there’s no need for medical facilities to manage or process it. And it’s easier to collaborate on DNA data; Sundquist says “Sharing data with other people is simply switching a few bits by adding access, instead of moving the data itself.” That’s important given that he sees the amount of sequencing to grow 10-fold every 2 years.

In the not-too-distant future then, children with a genetic disorder or people with a cancerous tumor could quickly and cheaply have sequencing tests run. And beyond those types of immediate needs, everyone can benefit over the next 5 to 10 years as we use the power of big data and cloud to crack the secrets of our genetic code.

Watch the livestream of Structure:Data here.

Watch live streaming video from gigaombigdata at livestream.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How Wordnik moved its database to the cloud

Ryan Lawler — Thu, 22 Mar 2012 19:07:29 +0000

When running databases, how do you get the speed you want while offering the flexibility and cost savings of the cloud? At Structure:Data in New York City Thursday, Wordnik co-founder Tony Tam talked about how his company was able to move its relational database from dedicated hardware to the cloud.

Wordnik provides a real-time engine for understanding the meaning of words that you read among various publisher sites. The best example of this might be a recent integration with SmartMoney, which provides relational links to a glossary that the publisher has assembled.

But managing dynamic, real-time understanding and meaning of words is no small task. To handle the task, Wordnik has gone through various iterations of its product in order to get to the speed and capacity that it needed. Wordnik started out on with a MySQL database on Amazon EC2, but found that setup wasn’t processing its data quickly enough. So it moved to some dedicated hardware and MongoDB, which solved its processing problem — it went from processing about 50 records a second to processing 1,000 records a second with the change, Tam said.

That was great from a performance point-of-view, but it came at a huge cost. In order to run at scale and to be ready for peaks in usage, Wordnik had a whole lot of excess capacity that was running idle most of the time.

“I’m not going to tell you that EC2 is as fast as raw metal, because it’s not,” Tam said. But what it lost in pure performance it was able to make up in part by using management tools to quickly bring up and shut down EC2 instances as needed and splitting processes across a number of clusters. Doing so cut costs by about half, when compared to owning servers with a bunch of capacity that was going unused most of the time.

To help with that, Wordnik relies on 10gen, which develops MongoDB and provides service and support to clients around it. 10gen founder Dwight Merriman said that the idea behind the database was in part to enable a cloud-based instances. But while MongoDB is cloud-friendly, Merriman said a number of clients still run it on dedicated hardware as well.

“Databases are not cloud-friendly. They’re one of the hardest parts of stack to get into the cloud,” Merriman said. “We’re big believers that this tech needs to be able to run anywhere… We don’t want it to be constrained.”

Watch the livestream of Structure:Data here.

Watch live streaming video from gigaombigdata at livestream.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why big data needs real-time intelligent systems

Ryan Kim — Thu, 22 Mar 2012 18:38:03 +0000

Zubin Dowlaty was watching the movie the Fifth Element when he saw the future of real-time intelligence laid out in a pithy quote: “Anticipation denotes intelligence.” Dowlaty, vice president and head of innovation and development of analytics-outsourcing firm Mu Sigma, said that’s what companies need to be striving for and in this era of big data, the barriers to achieving that have fallen away.

Speaking at GigaOM’s Structure: Data conference, Dowlaty gave an overview of how companies can stitch together their own real-time intelligent systems that can handle a growing stream of big data. He said companies first need to get into the right mindset and that means worrying more about consumption of analytics and less about the models and technology used.

He said the bigger goal is to help companies create predictive intelligent systems that can handle real-time data and shrink the amount of time it takes to act on events. He said by observing the companies that are successfully building these systems, often high-frequency traders, the necessary components come down to: messaging-oriented middleware; an advanced analytics engine; a business process modeling system; and a rules engine. Assembling these parts is becoming easier now, but companies still need to make the move.

“The biggest mistake is not doing anything,” Dowlaty told me after his talk. “Real time is hard but it’s not that hard. Big data is removing some of the excuses from a computational perspective. The technology exists, we just need to stitch it together and we’re off to the races.”

He said if pursued, this kind of intelligent systems can be applied to fraud detection, exception reporting, offer optimization, quantitive trading and sensor fusion. With the expected onslaught of sensor data coming from smart devices, there’s going be even more data to handle. But if you can anticipate what’s coming next, you’re on your way to becoming intelligent.

“It’s about man plus machine, heuristic and algorithmic coming together to operate analytics on a much quick scale,” he said.

Watch the livestream of Structure:Data here.

gigaombigdata on livestream.com. Broadcast Live Free

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.