GigaOM » data processing

Real-time query for Hadoop democratizes access to big data analytics

George Gilbert — Wed, 07 Nov 2012 07:55:09 +0000

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why you should care about data-flow computing’s big comeback

Ki Mae Heussner — Wed, 17 Oct 2012 15:37:15 +0000

Data-flow computing was developed about 30 years ago as a way of solving the parallel processing problem and then faded away over time. But, Damian Black, CEO of SQLstream, said Wednesday at GigaOM’s Structure: Europe conference that the decades-old technology is making a comeback. On stage with GigaOM senior writer Derrick Harris, Black talked why data flow computing is experiencing a rebirth and what it could mean for scaling in the cloud.

Check out the rest of our Structure Europe 2012 live coverage here, and a video recording of the session follows below.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

We need to prevent insights from dying in the big data avalanche

Anukool Lakhina, Guavus — Sat, 06 Oct 2012 17:30:50 +0000

Most enterprises think they know how promising their data is. The truth is, they don’t realize just how much value is hidden in the massive amounts of data they sit on — even as more data rolls on in. And because of this, the best insights – the ones that can be harnessed for transformative change – are at risk of getting buried in today’s data avalanche.

Analyze your data in the now.

Back when I was working on my PhD, I worked in the lab of a major telecommunications operator. My job was to run algorithmic possibilities on sensor-generated data so I could identify valuable trends and clues to network performance.
The best part of the day was when the FedEx truck arrived and I could get my hands on boxes of network data storage drives with mountains of months-old data generated by those sensors just waiting to be analyzed. Talk about timely insights being dead on arrival!

My employer had no idea what insights lay buried inside those drives. And yet, collecting, storing and sending the data out for analysis was the only option they it at the time. At that point I realized the model for data analysis had to change on a fundamental level, especially if data was going to continue its exponential growth curve. Businesses needed to analyze data as the avalanche roared in, and it was going to take some sturdy tools to do it.

The smartphones and tablets we rely on today contain a wealth of information on us — our preferences, our habits, our behavior. And this is just one kind of machine interface. there are also cars, for example, which now come equipped with an array of sensors to gauge everything from driving styles to road conditions and wear-and-tear, all in the interest of making driving safer and more enjoyable. Meanwhile, cities are deploying wireless sensors in stoplights for improved traffic surveillance. In disaster-prone regions, bridges and buildings can even evaluate their own stress points.

This phenomenon gives us an extraordinary opportunity — one that no civilization has had before — to know the now. If businesses act fast enough, they can distill that knowledge into timely, intelligent, data-driven insights for more agile operational and business processes. For example, an auto collision warning that pops up three weeks after the crash itself is useless. It’s the immediacy of insight that can then be translated immediately into action that safeguards us against disaster.

We need tools for real-time analysis

So now that the ability to gather such immediate data from a variety of devices and places exists in our world, it’s imperative we put it to work for our advantage. How can businesses parse data in a timely manner to identify trends, glean new insights into customer behavior, and respond immediately to changing market dynamics or customer habits? How can we best take divergent sources of data and dynamically fuse them together so people, machines and processes make optimal responses at any given moment in time?
In order to save this data from a premature death, and catapult it into a driving force for a data-driven global economy both the enterprise and the analytics architecture must rise to the occasion. Enterprises need a new approach to analytics where contextually-aware applications are based on specific use cases, built on a new data processing stack and backed by a new economic model.

As it stands today, big data analytics technology is comprised of many disparate toolsets and technologies. What’s missing is a foundational architecture to support all these individual tools and technologies — a complete, holistic stack that can help organizations get from data ingestion to data decisions in one fell swoop. This new architecture must recognize that a sensor-rich world creates data continuously, and in order to take immediate action, the analysis too must also be done continuously, rather than after-the-fact, once the data is stored away. This new architecture must also combine a variety of data sources instead of keeping them in silos. And it must elastically scale to the petabytes of structured and unstructured data that are now generated on a nonstop basis.

Equally important is the need for a new economic model for data processing. Today, enterprise customers spend tens of millions of upfront dollars on data projects, the majority of which goes toward capturing and storing the data. They must then wait a year or more to start seeing value from their data assets.

Our data-rich world therefore needs a new paradigm where enterprises first spend on analytics — not storage — with an agile, iterative approach that proves out the value of a particular idea in the first days and weeks of deployment. Once proven, this use case is swiftly rolled out as an application that any business manager can use to make decisions. This business value-led approach to big data can then be scaled across other functional areas of the business and power data-driven decision-making across the enterprise.

Once enterprises embrace this new approach, big data’s vast potential will no longer be crushed by its own weight. If data is at risk of being lost in the avalanche, our analytics platforms should serve as first responders to the emergency.

Anukool Lakhina is the CEO of Guavus.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Cloud computing and trickle-down analytics

Thu, 05 Jul 2012 06:55:01 +0000

Some people predict 2013 will be the year Hadoop becomes mainstream. Such an occurrence will only be possible if the technology trickles down to a broader base of users and lowers many of the barriers to adoption it carries today. A major limitation of big data, after all, is that the technologies used to analyze it are not easy to learn. It doesn’t have to be that way, and this research note looks in detail at how components of technologies like Hadoop are finding their way into tools that target less-sophisticated users — from business users to receptionists to high school students. Thanks to cloud-based services, data visualization tools and more, analytics can be made easier, and maybe even fun.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Creating value out of machine-driven big data

Mon, 09 Apr 2012 06:55:43 +0000

If your organization doesn’t have a strategy for big data now, you will need one in the future. This paper explains how business executives working with their CTOs or CIOs and other tech management can use big data within their organizations. It explains what big data is and how it differs from traditional business intelligence. The different considerations executives should take into account as they plan their big data strategies are also discussed. For example, is it better to build your own system or to buy one? Should you run your system on premises or in the cloud? Finally, the paper also includes examples of how some companies are putting their operational data to creative use.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

A near-term outlook for big data

Krish — Wed, 21 Mar 2012 06:55:20 +0000

Big data now touches everything from enterprises and hospitals to smart-meter startups and connected devices in the home. Hadoop, meanwhile, is fast becoming the leading tool to analyze that data, and there is the ever-lingering question of privacy and how we, the technology industry, are responsible for teaching ethical ways to collect and regulate our data. This report, composed of eight different sections each written by a GigaOM Pro analyst, offers insights on what to consider when it comes to big data decisions for your business.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Forget consumers, gigabit networks are ready for business!

Stacey Higginbotham — Tue, 07 Feb 2012 19:13:32 +0000

Consumer applications have driven the rapid take up of faster broadband services in the U.S. in the last decade as people downloaded iTunes songs and apps and watched streaming movies via Netflix. But as Google and others build gigabit networks to see what can be done with them, maybe it’s time to bring businesses back into the innovation cycle.

In Chattanooga, Tenn. the creation of a gigabit network has led to the formation of an incubator that wants to attract startups to the city this summer to play around with the nation’s first gigabit network. I spoke with Jack Studer, the managing partner at Lamp Post Group, which is the incubator hosting the contest, on what kinds of applications might drive people to get a gig.

Studer explained that while consumer applications were fun, the lack of other gigabit networks around the country made it a bit difficult to justify building a startup or business that needs a gigabit connection. Even if Studer has the bandwidth to receive a massively fat 3-D holographic image of me for a video conference, I couldn’t reciprocate on my 60 Mbps cable connection (that really delivers 30 Mbps) so building a consumer 3-D holographic web conferencing business is probably a long shot. Other similarly bandwidth-intensive ideas are also out … for now.

“Startups that require a gig — well, that business plan would suck. It’s like building up a business based on teleportation. It doesn’t exist yet,” said Studer.

Where the gigabit network really shines is business productivity says Studer. He points out that he can do things between his offices in Chattanooga that are truly business-changing such as real-time and continual data backups and replication. And that’s just the beginning. Studer has ideas around connecting distributed compute nodes around the city that could essentially turn Chattanooga into a giant supercomputer.

Gigabit speed, and the fact that no applications today require such speeds, mean a variety of services that now run on the computer might run in the network instead without it affecting the end-user. That has implications for data processing and the creation of new services based on an intelligent network. Such services might even become necessary as we connect more devices to the network.

For example, if we have a smart home where our computers, CE devices and even our lighting or appliances are connected to the network, we have to think about securing all of those endpoints. The current model of having antivirus software sitting on a PC no longer makes sense, but what about putting it on the network? A fast network means one could run services such as antivirus on the network without the user noticing.

But to bring the future to life, Studer needs students, venture capitalists and entrepreneurs to come to Chattanooga to play around with the network. Who’s up for the challenge?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

GigaOM » data processing

Real-­time query for Hadoop democratizes access to big data analytics

Why you should care about data-flow computing’s big comeback

We need to prevent insights from dying in the big data avalanche

Analyze your data in the now.

We need tools for real-time analysis

Cloud computing and trickle-down analytics

Creating value out of machine-driven big data

A near-term outlook for big data

Forget consumers, gigabit networks are ready for business!

Real-time query for Hadoop democratizes access to big data analytics