Retwact — whose full name is Retweet Retraction — is the brainchild of a programmer named Stonly Baptiste, a developer in Pennsylvania. In a nutshell, the service archives your incorrect tweet with a correction or apology of your choosing, then shoots a link out to all of your followers to try and encourage them to read the corrected version. In addition, it also sends an @ mention and link out to the first 100 people who retweeted your original incorrect message, in the hope that they might also help spread the correction.

Correcting tweets would be complicated

As it turns out, this latter feature appears to have run afoul of Twitter’s terms of service, which are designed to prevent spam accounts, and Retwact’s account was suspended on Thursday. Baptiste says that he plans to go ahead with the other features regardless, and may even make his project — which received a lot of support on Y Combinator’s Hacker News forum — open source.

The impulse behind a tool like Retwact is an obvious one: as Wired writer Mat Honan notes, there is a sinking feeling whenever you tweet or retweet something that is incorrect (or turns out to be incorrect), and it would be nice to be able to retract or remove not just that tweet but all the subsequent retweets of it as well, to clear up the public record. Honan joins a growing chorus of critics asking for a correction mechanism (or trying to design one, as some members of this post-Boston Branch discussion did).

Adding that kind of editing or retraction/clarification ability seems to be something that is within Twitter’s grasp: in the same way that it has built hooks into Twitter’s code so that media companies can embed video clips and other data within its “Cards” or expanded tweet feature, it would theoretically be possible for Twitter to add a hook that would connect a mistaken tweet with its subsequent corrected version, so that both would follow each other around the social web.

As Twitter engineer Nick Kallen has explained, however, the likelihood of Twitter actually building in this feature seems somewhere between slim and nil — in part because they driving force behind most of the company’s changes over the past year or so (with the exception of expanded tweets) has been to strip functionality and features away rather than to add them. An editing or correction function could also theoretically be abused in a number of ways.

Twitter is a real-time stream

But more than that, I think Kallen puts his finger on the problem when he says that adding correction features would change the nature of what Twitter is in a fairly fundamental way. The whole point of the service is that it is a stream of content that never stops — and the only way to correct a tweet is to send out another one. In that sense, it mimics conversation, which is also inherently un-correctable except through more conversation. It may be flawed and messy, but that’s the way information works now, for better or worse.

And yes, this has obvious flaws, because the correction never travels quite as far as the original mistake (as Craig Silverman of Regret The Error has pointed out). But over time, I firmly believe that Twitter becomes what Sasha-Frere Jones of the New Yorker called a “self-cleaning oven” for news.

On top of that, I don’t think adding an editing or correction function like Retwact would actually help all that much. People would continue to believe whatever they want to believe — as wrong as that might be — and no matter how thorough the mechanism was, it wouldn’t stop those who created their own manual retweets or retweets of retweets. I also think that having errors emerge and get stamped out over time is a positive process that creates more skepticism about real-time news, something that we need to encourage. It is a process, not a finished product.

So as much as I cringe internally whenever I send out a mistake — which I have done, and will no doubt continue to do — I hope Twitter ignores the requests of its critics to implement an official editing or correction function.

Post and thumbnail photos courtesy of Shutterstock / Hirurg

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Players and Strategies for Real-Time In-Stream Advertising
• A 2011 NewNet Forecast
• Communications, Platforms, Privacy Ruled NewNet in Q4

IBM considers both efforts part of its Smarter Planet and Mobile First strategies. The box is called the IBM MessageSight, and it combines the ability to process a lot of information in real time — which experts believe the internet of things will need. The logic is that billions of sensors sending trillions of bits will need some type of special equipment to process the incoming information in real-time and send instructions back to a human or a device.

IBM uses the example of the hundreds of sensors in your car recognizing a problem, turning on your check engine light, and then notifying the dealer so it can do remote diagnostics. As someone who is heading to the dealer tomorrow for a check engine light, this example caught my eye. Yet, I’m not sold on the need for a special box over more intelligence at the sensor, or perhaps a mesh network with nominal “intelligence.”

The internet of things exaflood is coming!

The idea is compelling, but it also grossly simplifies the flow of data inside the internet of things. For example, it assumes all sensor data must be processed in “real time.” It also assumes all the data must be processed. Both of these are untrue, especially in the early days of the internet of things. But IBM is looking ahead. From its release on the MessageSight appliance:

Over the next 15 years, the number of machines and sensors connected to the Internet will explode. According to IMS Research, there will be more than 22 billion web-connected devices by 2020. These new devices will generate more 2.5 quintillion bytes of new data every day, while every hour enough information is consumed by Internet traffic to fill seven million DVDs.

It’s the same exaflood of data that telephone companies were so fearful of a decade ago. And like the telephone companies, IBM is hoping to cash in on these fears — with its box. IBM’s appliance can totally stand up to this tsunami of information, or so goes the pitch. A release from Big Blue noted that the machine can handle up to 1 million concurrent sensors and can scale to manage up to 13 million messages per second.

The new protocol for the internet of things?

Inside this magic data-defying box will be a protocol called MQTT (Message Queuing Telemetry Transport), which the OASIS standards organization recently proposed for the internet of things. The standard, which is backed by Kaazing, Red Hat, TIBCO, Cisco and IBM, is a lightweight messaging transport system for communication in machine to machine and mobile environments.

The idea is that such a lightweight protocol will allow sensors to communicate wirelessly without needing massive batteries to support a fully functional wireless radio. I’m unclear on what radio protocol one might use, but have reached out with questions. As for MQTT, it’s already in use for satellite transmissions and in medical and industrial settings where low-bandwidth communications are essential.

IBM said “sensors can use MQTT to send messages wirelessly using 10 times less battery power and 93 times faster than before, making it possible for a sensor to send real time updates that can be acted upon immediately.” IBM is positioning MQTT as the same enabler for the internet of things, as HTTP was for the web. I’m not sure I’d go that far, but it’s worth watching to see how the standard evolves.

As for IBM’s appliance, I’m pretty sure people can build connected homes, buildings and possible cities without it, but IBM’s marketing will snag customers, especially as part of an overarching integrated smarter cities deployment.

Updated at 11:41 PT: This story was updated to clarify that the MQTT protocol is new for use in the internet of things. The protocol itself is not new.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Analyzing the wearable computing market
• The Internet of Things: What It Is, Why It Matters
• Call it Real-Time, Squared, or NewNet, The Web Is Changing

Facebook is going to be launching some new features for its feed on Thursday, which may include new ways of filtering specific kinds of content and possibly new advertising features. Meanwhile, Twitter continues to show you everything, without filtering or ranking it in any way. Which method is better? That depends on how and why you are using it.

Much of Bilton’s criticism revolves around what some call the “subscribe” function, which allows users to get updates from others without having to ask their permission. When it launched in the fall of 2011, it was widely seen as an attempt to copy Twitter’s “asymmetric following” model, since Twitter lets users get updates from whoever they wish — whereas Facebook’s model has always been symmetric, in the sense that users must agree to be friends before they can see each other’s updates. Late last year, Facebook changed the name of this feature to “follow,” which made the similarity to Twitter even more obvious.

Do you want to see everything in your feed?

As an attempt to copy Twitter, the follow feature seems to be largely a failure — at least if the experiences of Bilton and others who have complained about Facebook’s newsfeed, such as billionaire entrepreneur Mark Cuban, are anything to go by. They say they don’t get much engagement, which makes them question whether their content is even reaching their subscribers, and whether Facebook is tweaking their feed so that certain kinds of updates don’t show up as frequently.

The last time this topic came up, when Cuban and actor George Takei were criticizing the network because of the lack of engagement from subscribers, a number of Facebook users attacked the company for filtering their feeds and not showing them all of the updates from pages or individuals they were following. Some users said the equivalent of: “If I subscribe to someone, I want to see all their updates, not just the ones that you choose to show me.”

swipe at facebook: 'there's no algorithm standing between you and your audience' - @adambain gets feisty.

—
Brian Morrissey (@bmorrissey) March 07, 2013

In a nutshell, this is the fundamental difference between Twitter and Facebook: the former doesn’t apply any filters to the stream of updates users get, apart from those required by law — if you follow a couple of thousand users, as I do, then you get all of the updates from all of those users, and they flow past you in a giant river of undifferentiated tweets, in reverse chronological order.

Facebook, however, applies all kinds of algorithmic tweaks to a newsfeed based on what some call EdgeRank (although this isn’t a term Facebook uses internally, according to Anthony De Rosa of Reuters), and therefore some updates are more prominent than others, and in some cases updates may never appear at all. Users have control over some of the knobs and dials that will hide or reveal certain kinds of posts, but there is also a lot of filtering that goes on behind the scenes, which makes Facebook a bit of a Google-style black box.

It’s hard to know what you’re missing

Depending on how you see them, these two different approaches can be a good thing or a bad thing: Twitter’s method is theoretically more transparent and comprehensive, since it is completely unfiltered — but it can also be overwhelming, and the network has worked hard to try and help users cope with this vast stream of content, via things like the Discover tab. Facebook’s method seems a lot more invasive and secretive, but at the same time it can make it easier to cope with the never-ending ocean of content — an average of 2,000 posts a day for each user.

Danny Sullivan of Search Engine Land has a useful analogy for the difference between the two: Twitter is a little like real-time TV news, while Facebook functions more like a DVR that lets you watch things after they have happened (although to some extent the network chooses what to show you, which your DVR doesn’t). They are two very different experiences of a social stream.

While Facebook users might complain that they want to see everything their social graph posts, the reality is that they likely wouldn’t see everything anyway — unless they sat on their computer all day long reading everything that was posted. Most die-hard Twitter users likely don’t see everything their followers post either, unless they watch the network 24 hours a day, and many use lists (as I do) to try and cope with the volume of content that is posted, or services like Paper.li that allow them to “time shift” that content and catch up with it later.

In the end, the question hangs not just on how you want to handle that stream of updates from your social graph, but who you trust to do that management for you: in the case of Twitter, you are pretty much on your own, and that can be chaotic — but there is a certain purity to it. With Facebook, you have some tools at your disposal to manage that content, but the network itself also does a lot behind the scenes without telling you much about how it works. Facebook says it’s for your own good, but how do you really know what you are missing?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Examining the rise of crowd labor platforms in 2012
• Social third-quarter 2012: analysis and outlook
• Social media in Q1: commerce and discovery dominated

One of the things that made my iPhone into an extension of my arm for the three years that I used one was the ability to see at a glance anything that required my attention, whether it was email or Twitter, or Instagram, or Path, or one of a dozen other social networks and services that I have signed up for. At first I thought this was a great feature — but I’ve changed my mind.

A profusion of bubbles, banners and popups

Not only did certain apps (like Twitter) wake up the iPhone screen even when the device was sleeping to flash a message, but every icon for every app also had mini-notifications built in, so that I could see at a glance how many emails had come in since the last time I had checked, or how many Facebook messages, etc. Each icon had a little number next to it that wouldn’t go away until I opened the app and dealt with the messages or updates (there are also banner updates that can be individually configured for different apps).

If you need to stay on top of things like email, this is a really great feature. If you are somewhat obsessive or have something approaching attention-deficit disorder, however, it’s like a never-ending game of whack-a-mole that you play with your phone: open the app and click through the emails so that the number next to the icon goes away, and five minutes later there are a hundred more waiting. Twitter is the same, and so is Facebook.

(Note: I know that you can turn these off on the iPhone, as some commenters have pointed out. I am just describing my experience of the default settings, not making a blanket statement about the value of the iPhone as a whole).

To me, those numbers became a nagging indicator of my failure to stay on top of everything I was supposed to be paying attention to. Which is why I noticed when I switched to Android that there weren’t any notification bubbles next to the icons, and nothing woke up my phone. There was a small LED at the top of the phone — a Motorola Razr HD — that changed color based on certain input, but that was it. And when you wake the phone up, there are some small icons at the top that indicate new emails, etc. All very easy to ignore.

How can something that’s missing be positive?

Many iPhone fans are probably going to see what I’m describing as a negative rather than a positive. After all, I’m talking about how the Android actually *lacks* certain features that the iPhone has — how could that be seen as a good thing? And that’s what I wondered when I started using the Android.

In fact, I spent a fair bit of time looking for ways to reproduce the same kind of notification experience I got with the iPhone. I tweaked the settings — which don’t really give you the same kind of granularity that you get with the iPhone (or at least not in my experience) — and I even downloaded a bunch of apps that were designed to replicate the iPhone notifications somehow, right down to the noises they made, which were programmed into my subconscious.

Nothing I tried seemed to reproduce the kind of notifications I got on the iPhone, however, or at least not in a way that seemed to fit my needs. So I basically stopped trying. Now the light on my phone blinks from time to time, but it’s really easy to ignore — and it chirps sometimes, but there’s no flashing on-screen message to tell me what it is. I have different rings for texts and phone calls from important people and that’s about it.

It’s not you, iPhone — it’s me

When I open my Android phone up from sleep mode, there are no tiny numbers beside any of the icons. There’s a widget that shows the first few subject lines of emails, so I can see whether there’s something hugely important, and another widget with a small calendar view. And when I want to see notifications from all the various apps and services, I can swipe down on the screen (a feature Apple borrowed from Android, I believe) and see a list.

Not having better notifications may be a downside for some, but I guess for me it has been a blessing in disguise — I was trying to be more disciplined about my real-time updates, the way some others like Om have described, and turn off all the notifications one by one, but I am weak. Maybe it took a switch to a different platform and an unfamiliar user interface for me to make the decisions I should have made before to make my life a little less hectic.

Believe me, I’m not trying to say that the Android phone is better than the iPhone in every circumstance or for every person, or that Google is better than Apple. I’m just trying to describe my usage of both and how I came to the conclusion that for me, fewer notifications (or more subtle ones) is actually a good thing.

Thumbnail image courtesy of Brosix

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• What the Google-Motorola deal means for Android, Microsoft and the mobile industry
• Mobile Q2: Smartphone growth surges; iPad’s rule continues

It’s not surprising HStreaming would be drowning in interest. Ask anyone how they’d like to see Hadoop evolve beyond its current status as a batch-processing platform, and you’ll likely hear “real-time” as one of the answers. In fact, this is a topic we’ll discussing a lot at Structure: Data next month with companies trying to turn Hadoop into operational databases and various types of OLAP engines, and those companies just generally struggling with unceasing streams of machine data.

Presently, companies trying to incorporate a real-time component into their Hadoop environments, in order to process data as it streams into the system and before it hits the disk, are bolting on open source technologies such as Storm and Kafka. While these certainly aren’t toy technology, Uhlig thinks the open source versions are rudimentary (Storm, for example, can pretty much just classify each piece of data as it hits) and notes that they’re not part of a full-on analytic system.

HStreaming, on the other hand, has built a complete system that incorporates its real-time engine for processing streams of video, server, sensor and other machine-generated data, but also is wholly compatible with Hadoop as an archiving and batch-processing system. It also plugs into a wide variety of existing BI tools for analytics, Uhlig said.

Better yet, for Hadoop users, nothing has to change. HStreaming can do its stream processing by re-using the same MapReduce algorithms and Pig scripts that customers have already written. In practice, Uhlig said, users can move from a batch-only system into a real-time system in just days.

She said major interest thus far has come from governments (especially around video analysis, where HStreaming can stitch together images from thousands of cameras in real-time), telcos and advertising. The company plans to step up its focus on utilities and financial services, too.

Former Vertica CEO and Atlas Venture partner Chris Lynch said telcos should be particularly excited about technologies like HStreaming for the purpose of network arbitrage. Without stream processing and some semi-complex algorithms, he explained, it’s impossible to tell what traffic belongs to which users so network providers can charge other carriers or ensure that premium customers don’t experience degraded service while others are doing fine.

“We tried this unsuccessfully at Vertica,” Lynch said. “… We weren’t fast enough.”

HStreaming certainly appears to have the technical chops to do what it promises. CEO Uhlig is joined by CTO Volkmar Uhlig (Jana’s husband) and Chief Software Architect Jan Stoess, both of whom hold Ph.Ds in computer science. Volkmar was lead architect on the L4 microkernel, has built high-frequency trading systems and spent five years a IBM’s TJ Watson Research Center working on stream-processing technologies.

And while one expects investors to be supportive of their portfolio companies, Lynch is effusive in his praise for what HStreaming is doing. “The technology has no practical limitations, but we have to market the company,” he said. “… Watch what happens in the next six months. … I sold 800 customers at Vertica before I sold the company [to HP], and every one of them is going to want HStreaming.”

Feature image courtesy of Shutterstock user MrJafari.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Deploying big data: 2012 strategies for IT departments
• Creating value out of machine-driven big data

This is why, in 2006, Lakhina founded Guavus, a company that has built a software platform for real-time analysis of streaming data. The company claims its proprietary platform can handle massive amounts of real-time data, but also that it can match that incoming data to existing warehoused data as well. In its latest product, aimed at the telco market, Guavus lets operators process about half a petabyte of data every day and match that real-time data stream against subscribers.

This means your wireless provider can match your YouTube viewing to you while you are viewing it. Lakhina calls this “DPI-level insights.” Subscribers might not see the value but operators do. Lakhina says that a Tier 1 operators in the U.S. used his company’s software to catch a customer who was violating the terms of its contract with the operator and was subsequently flooding the network in New York.

Lakhina says the carrier noticed that during business hours in New York its network traffic had suddenly jumped by 10 percent for no clear reason. By looking at the traffic patterns at an individual level using Guavus’ software it matched that jump in data to a new customer that had sold an M2M payment processing product to taxi cabs. The cab drivers were supposed to use the cellular network for processing payments, but instead were using it to stream video inside their cabs, a violation of the terms between the payment platform provider and the carrier. The operator renegotiated the terms of the agreement.

This has obvious benefits for the carrier, which can now police contracts and can also use the software for planning where it puts capacity. For example, when AT&T got the iPhone it had no idea how it would affect its network, and until things started breaking and calls began dropping it was unable to predict where and when problems would arise. At the time, network engineers dealt with capacity problems in quarterly or monthly reviews, but in today’s world capacity planning isn’t something static enough that a monthly or even weekly adjustment works.

Or at least, that’s the pitch from Guavus. For the end user the use of big data and real-time analysis by operators is less compelling. Lakhina says that operators could use it to help reduce the number of customer care calls. For example, instead of being surprised by an overage charge or being unaware of where such a change came from, an operator might proactively notify a subscriber or could at a minimum explain to them after the fact what behavior caused them to go over their current data cap.

Guavus has two of the top carriers in the U.S. as customers of its NetReflex software, and plans to take its underlying data platform to other industries that could benefit from the mix of real-time data analysis.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Carrier IQ and the continued erosion of operator trust

The idea is that Boundary has analyzed the performance and habits of a customer’s network so when that pattern changes, Boundary can send an alert. The company is ingesting 350 Mbps of inbound networking traffic on its 550 customers which it then processes in real time. Boundary has its own custom-built, real-time data processing engine, that takes this influx of second-by-second data and combines it into dozens or hundreds of terabits per second of traffic for analysis. A Boundary spokesman notes that this current rate represents a fraction of the startup’s total processing capacity.

Indeed, both the product launch and the data flows are topics I’ve discussed with Boundary CEO Gary Read (pictured) in the past. We’ve outlined the merits of real-time network monitoring that can track multiple data points every second as well as discussed what that will mean in terms of the amounts of data that Boundary will have to process. With 550 customers, Boundary is analyzing 600,000 events every second and the company is adding about 10 to 12 paid customers each month.

To analyze this data, servers monitored by Boundary report data to its SaaS platform, sending up information about the network and application flows they see passing through. So the 600,000 events per second refer to the network flows that Boundary observes each second for our customers inside their networks, and as they communicate with their customers. As Boundary grows to observe more flows between data centers to end users, and through mobile networks, it gains a broader picture of the health of cloud providers and the Internet as a whole. It’s somewhat similar to what startup DeepField Networks is also proposing, and is an essential step in creating accurate monitoring for federated applications built in cloud environments.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Dissecting the data: 5 issues for our digital future
• Is Tiered Data Access Unfair, or the American Way?

DataSift is more than just a collector of Twitter data, however. It also takes in data streams from dozens of other web and social media sources, and can handle corporate data, as well. And the company’s real value comes in the analytics it applies to all that data, letting users filter and correlate across myriad different factors.

Here’s an interview I did with DataSift Founder and CTO Nick Halstead during our Structure: Europe conference in Amsterdam last month, in which he describes how DataSift built an infrastructure capable of handling so much real-time data and how companies can use such data effectively.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Finding the Value in Social Media Data
• Facebook’s IPO filing: ideas and implications
• NewNet Q2: Google closes the quarter with a bang

Below is the video of their efforts, set to somewhat over dramatic music for an election (maybe if it were 2000, y’all). And here is where you can see the live trending data showing sentiment on Twitter right now.

The two researchers, Kalev H. Leetaru of the University of Illinois and Shaowen Wang of the CyberInfrastructure and Geospatial Information (CIGI) Laboratory at the University of Illinois at Urbana-Champaign, also tracked tweets related to Hurricane Sandy as part of what they call the Global Twitter Heartbeat Project. Other than cool, real-time heat maps, the effort shows how scientists (and eventually marketers) can use high performance computers to track real-time unstructured data. It is something people can do today. From a release on the project:

The Global Twitter Heartbeat project performs real-time stream processing of ten percent of Twitter’s 400M daily tweets as they are posted. The project analyses every tweet to assign location (not just GPS-tagged tweets, but processing the text of the tweet itself), and tone values and then visualizes the conversation in a heat map infographic that combines and displays tweet location, intensity and tone. With SGI UV, the entire process from data analysis to heat map was produced once per second.

The SG UV 2000 is an impressive machine with a maximum of 4,096 cores that can scale to 64 terabytes of cache-coherent shared memory at a peak I/O rate of four terabytes per second. Presumably one could use another high-performance machine with a lot of parallel processing and fast IO, but Leetaru and Wang apparently had an SGI machine on their desks. In the release, Leetaru likened the process of looking at all this data to peering into a telescope focused on the “post-demographic world” where individuals could be processed directly in the flow of information rather than forcing them into a specific demographic cohort.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Readers weigh in: future prospects for Twitter
• A near-term outlook for big data
• 4 iPad apps to help wrangle data

Analyze your data in the now.

Back when I was working on my PhD, I worked in the lab of a major telecommunications operator. My job was to run algorithmic possibilities on sensor-generated data so I could identify valuable trends and clues to network performance.
The best part of the day was when the FedEx truck arrived and I could get my hands on boxes of network data storage drives with mountains of months-old data generated by those sensors just waiting to be analyzed. Talk about timely insights being dead on arrival!

My employer had no idea what insights lay buried inside those drives. And yet, collecting, storing and sending the data out for analysis was the only option they it at the time. At that point I realized the model for data analysis had to change on a fundamental level, especially if data was going to continue its exponential growth curve. Businesses needed to analyze data as the avalanche roared in, and it was going to take some sturdy tools to do it.

The smartphones and tablets we rely on today contain a wealth of information on us — our preferences, our habits, our behavior. And this is just one kind of machine interface. there are also cars, for example, which now come equipped with an array of sensors to gauge everything from driving styles to road conditions and wear-and-tear, all in the interest of making driving safer and more enjoyable. Meanwhile, cities are deploying wireless sensors in stoplights for improved traffic surveillance. In disaster-prone regions, bridges and buildings can even evaluate their own stress points.

This phenomenon gives us an extraordinary opportunity — one that no civilization has had before — to know the now. If businesses act fast enough, they can distill that knowledge into timely, intelligent, data-driven insights for more agile operational and business processes. For example, an auto collision warning that pops up three weeks after the crash itself is useless. It’s the immediacy of insight that can then be translated immediately into action that safeguards us against disaster.

We need tools for real-time analysis

So now that the ability to gather such immediate data from a variety of devices and places exists in our world, it’s imperative we put it to work for our advantage. How can businesses parse data in a timely manner to identify trends, glean new insights into customer behavior, and respond immediately to changing market dynamics or customer habits? How can we best take divergent sources of data and dynamically fuse them together so people, machines and processes make optimal responses at any given moment in time?
In order to save this data from a premature death, and catapult it into a driving force for a data-driven global economy both the enterprise and the analytics architecture must rise to the occasion. Enterprises need a new approach to analytics where contextually-aware applications are based on specific use cases, built on a new data processing stack and backed by a new economic model.

As it stands today, big data analytics technology is comprised of many disparate toolsets and technologies. What’s missing is a foundational architecture to support all these individual tools and technologies — a complete, holistic stack that can help organizations get from data ingestion to data decisions in one fell swoop. This new architecture must recognize that a sensor-rich world creates data continuously, and in order to take immediate action, the analysis too must also be done continuously, rather than after-the-fact, once the data is stored away. This new architecture must also combine a variety of data sources instead of keeping them in silos. And it must elastically scale to the petabytes of structured and unstructured data that are now generated on a nonstop basis.

Equally important is the need for a new economic model for data processing. Today, enterprise customers spend tens of millions of upfront dollars on data projects, the majority of which goes toward capturing and storing the data. They must then wait a year or more to start seeing value from their data assets.

Our data-rich world therefore needs a new paradigm where enterprises first spend on analytics — not storage — with an agile, iterative approach that proves out the value of a particular idea in the first days and weeks of deployment. Once proven, this use case is swiftly rolled out as an application that any business manager can use to make decisions. This business value-led approach to big data can then be scaled across other functional areas of the business and power data-driven decision-making across the enterprise.

Once enterprises embrace this new approach, big data’s vast potential will no longer be crushed by its own weight. If data is at risk of being lost in the avalanche, our analytics platforms should serve as first responders to the emergency.

Anukool Lakhina is the CEO of Guavus.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Big data 2013: key trends and companies to watch
• Health care and big data in 2012