(L to R): Ashok Srivastava, Venture Advisor for Trident Capital and Chief Data Scientist, Verizon; Silvius Rus, Director, Big Data Platforms, Quantcast; Todd Papaioannou, Founder and CEO, Continuuity; Bhaskar Ghosh, Senior Director of Engineering, Data Infrastructure, LinkedIn; Michael Driscoll. CEO, Metamarkets Structure Data 2013 Albert Chau itsmebert.com

Last week it was Sawzall‘s time to shine. The language, named after the popular saw that cuts through anything (and I mean anything), comes out of Google.

Silvius Rus, director of big data platforms for Quantcast, gave Sawzall a shout-out during a Structure Data Guru panel last week. ”It’s a lightweight language developed by Google that ridges procedural and interpretive languages,” Rus said.

Michael Driscoll, CEO of Metamarkets and moderator of the panel, later explained why that’s important. With a declarative language, the programmer tells the computer what to do in almost English-language-like sentences. To tell the computer to draw a circle, a declarative or imperative programmer might say “draw.circle with a size attached,” Driscoll said.

Procedural languages, on the other hand, are much more detailed step-by-step instructions — they sound more like math. A procedural approach would “define the actual pointer and tell it to move one degree to the left and one degree up and the square root of 2 up to the diagonal and repeat X times,” Driscoll said.

Sawzall is a nice blend between a declarative language that might be too high level to do all of what the programmer really wants and procedural, “which is way too in the weeds” to be fully productive, Driscoll said. More broadly, Sawzall is a powerful and compact language for log data aggregation and transformation. And, he added, it plays well with Hadoop MapReduce.

New toolsets for webscale computing

Another tool ranking high on the hit list was YARN (or Yet Another Resource Manager) aka MapReduce 2.0, cited by Todd Papaioannou, founder and CEO of Continuuity is a fan.

Yarn was built to “just think about mass-produced jobs.” Continuity is building a real-time streaming engine called Big Flow and using Yarn for all the resource deployment and management.

He also gave kudos to Weave, a higher-level framework. Weave “allows you to build a much wider class of applications on top of Yarn. So,t Yarn is  … something that we will be going forward with for at least the next half a decade [and] Weave allows you to actually build more wide scale applications on top of that.”

Bhaskar Ghosh,  senior director of engineering at LinkedIn, touted Helix, a generic distribution cluster manager developed at LinkedIn and which is now an Apache incubator project.  Helix simplifies distributed system development by separating cluster management from the primary component tasks of a distributed system, according to LinkedIn.

Kafka, Storm slake the thirst for real-time frameworks

Driscoll also sees traction for Kafka, a real-time framework for ingesting and managing data streams and Storm, out of Twitter, for processing those streams. “Think of Kafka and Storm as the HDFS and MapReduce analogs but for real time — Kafka for storage and Storm for compute,” Driscoll said.

On its blog, LinkedIn describes Kafka as a distributed publish-subscribe messaging system — also now an Apache project. Kafka is used by Twitter and Square for log aggregation, queeuing, and real-time monitoring and event processing.

This list is by no means complete. When I spoke with Github co-founder Tom Preston-Werner a few weeks ago, he said Clojure, heretofore a rather obscure dynamic programming language, is gaining momentum. “It’s getting a lot of buzz round on the enterprise side,” Preston-Werner said.

The continued popularity of the Java Virtual Machine has breathed new life into languages like Clojure and Scala, he added. Indeed, the JVM remains nearly ubiquitous and that is a huge advantage for languages that support it. If you’re a developer, you want the widest possible audience.

“The JVM is still the modern foundation that lets you run everywhere and Clojure has benefited from that,” Driscoll agreed. “It’s certainly gained steam among an elite set of programmers in Silicon Valley.”

Sawzall photo courtesy of Flickr user Charles & Hudson

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang
• Social 2013: The enterprise strikes back

(c) 2012 Pinar Ozger. pinar@pinarozger.com

“What’s so interesting about mobile data is how personal it is,” said Michael Driscoll, CEO at MetaMarkets, during a session at Structure:Data. It’s not just what you’re doing on your phone or tablet, but when you’re doing it and where you’re doing it: those are variables that aren’t necessarily as important to the traditional desktop-and-Web-based model of computing.

And “the accuracy of what you can capture in a mobile app goes beyond” other data sources, said Raj Aggarwal, CEO and co-founder of Localytics. That’s going to increase as phone makers put more sensors in their devices–say, for health monitoring–and software developers come up with more sophisticated ways to interact with those sensors, he said.

But as we’ve seen time and time again, there are privacy minefields involved with any type of collection of data from mobile devices, the most personal computing device most of us own. Aggarwal advised anyone working with mobile data to give users an easy way to opt out of those services and to be transparent about how data is collected and scrubbed of personally identifiable information.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How tomorrow’s mobile-centric data centers will look
• Dissecting the data: 5 issues for our digital future
• Report: How Mobile Cloud Computing Will Change Tech

Michael Driscoll, CTO of Metamarkets, (see disclosure) made this pitch in his blog earlier this week. Lots of people designate their organs for donation after their demise, but in his blog, Driscoll argues:

…why shouldn’t you be able to give your DNA sequence, your diet, and your disease diagnoses to science while you’re alive?  Unlike your organs, you can donate your data away and yet still keep it.

Talk about the ultimate good use of big data. And HIPAA concerns aside, he thinks many people would go for it, provided it was easy for them to do so and the recipient organization was trustworthy. Devices like Jawbone Up and Fitbit (see disclosure), other body-born sensors, or even cell phones could send the data to construct what “could be the beginning of a valuable data bank,” Driscoll wrote. He attributed the idea to Gil Elbaz, founder and CEO of Factual.com.

Driscoll said sites such as Patientslikeme, which serves more than 122,000 users who share their own health information and Ginger.io technology which enables cell phone sensors to collect and send people’s health data, show what can be done.

There will be the inevitable knee-jerk reaction, but what better use of big data is there than to help existing and future patients with similar conditions? Of course, privacy advocates have a point: Misuse of personal data is dangerous and misuse of personal health data could be worse. Private health data, if it ends up with a potential employer, for example, might cost someone a job. It’s not right, and may not be legal, but it could happen. For that reason, this data would have to be anonymized. For real.

And, unlike past bad practices — the medical industry’s use of Henrietta Lack’s  ”HeLA” cells without her knowledge is an egregious example — this model must mandate informed consent.  (HeLa cells were taken from Lack, a cancer patient in the early 50s and are still in use today. They were termed “immortal cells” by researchers. But neither Lack, who died soon after the cells were harvested, or her children, were ever told that they were taken or that they were used in research.)

The key to medical data donation in the age of big data is full, informed and voluntary consent by people like you and me and ethical, trustworthy and technically-sound decisions about how to store, anonymize, analyze and manipulate data on the back end.

Disclosure: Metamarkets and Fitbit are both backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

Photo courtesy of Flickr user KWB57

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Health care and big data in 2012
• Takeaways from the second quarter in cloud and data

Both events highlight that when it comes to Big Data, analytics and cloud computing, Oracle is on the wrong side of history.

To glimpse the future of the data stack, Oracle need look no further than its own backyard, to what Silicon Valley start-ups are embracing: the distributed processing ecosystem of Hadoop, NoSQL data stores like MongoDB, and cloud platforms like Amazon’s web services.  As Marc Andreessen said last week, “Not a single one of our startups uses Oracle.”

The challenge for Oracle, which did $36 billion in revenue last year, is that they sell to big enterprises and selling technology to start-ups doesn’t move the needle.

Worse, Oracle’s support for the kind of technology stacks embraced by startups — open-source software, elastic architectures, commodity hardware grids — cannibalizes revenue from their existing lines of business.

“I don’t care if our commodity X86 business goes to zero,” Ellison said in Oracle’s last earnings call, “We don’t make money selling that.”

This commoditization wave may have sent others, including HP, fleeing from hardware, but it has driven Oracle into the breach: they are attempting to capture higher margins on sales of their Sun-acquired SPARC architectures.

The buyers of these big boxes are enterprises struggling with sharp increases in data volumes, and willing to pay top dollar for what Ellison dubs a “100 percent upwardly compatible migration path,” referring to the SuperCluster T4-4.

But history is not on Oracle’s side.  Today’s startups are tomorrow’s Goliaths, and soon they will have to confront a future that, as William Gibson quips, “is already here… just not evenly distributed.”

Here are four realities that Oracle must face to maintain its unassailable position as the world’s leading data firm:

The future of data is distributed

“Lots of little servers everywhere, lots of little databases everywhere. Your information got hopelessly fragmented in the process.” – from Matthew Symonds book Softwar (p. 38).

This is how Larry Ellison described the technology landscape of the 1990s, and his personal jihad against complexity has deepened Oracle’s distrust of distributed computing.

But the tide of data isn’t turning back, and the scale is too large to contain in any box; Big Data, on the scale of hundreds of terabytes to petabytes, must be distributed across “lots of little servers.” The most viable tool available today for processing and persisting Big Data is Hadoop.

Whether at the data layer — or a level above, at analytics — firms must adapt to this distributed reality and build tools that enable parallelized, many-to-many migration of data between nodes on Hadoop and those on their own platforms.

The future of computing is elastic

Metal server boxes don’t bend or expand; they are inelastic, both physically and economically.  In contrast, the needs of businesses are highly elastic; as companies grow, they shouldn’t have to unpack and install boxes to meet their compute needs, any more than they should install generators for more electricity.

Computing is a utility, compute cycles are fungible, and firms want to pay for what they need, when it’s needed, like electricity.

The ability to scale storage and compute capacity up or down, within minutes, is liberating for individuals and cost-effective for organizations, but it is impossible with a “cloud in a box.”  It is only enabled by a true cloud computing infrastructure, with virtualization and dynamic provisioning from a common pool of resources.

The future of applications is not on the desktop

Despite that Oracle developed the first pure network computer in 1996 (or perhaps because of this), far too many of Oracle’s supporting business applications are delivered via the desktop, rather than via web browsers.

By comparison, Cloudera has created a rich web-based application for managing and monitoring all aspects of Hadoop clusters; Amazon Web Services has a fully-featured web console for interacting with its offerings; and Salesforce’s products are almost exclusively web-driven.

The expressivity afforded by web browsers has risen dramatically in the last two years, particularly with the emergence of Javascript as the lingua franca of web application development, and improvements in Javascript engines.

The same trend from desktop to browser also extends into mobile devices.  An increasingly large fraction of computing occurs on smart phones and tablets, and forward-thinking firms, like Dropbox, have built applications that cater to this reality.

The future of analytics is beautiful

The decades of disappointment with business intelligence tools isn’t due only to their lack of brains (such that they’ve now fled to the fresh moniker of “business analytics”), but also the absence of beauty. Data is beautiful, as any reader of Edward Tufte can attest.

When visualized thoughtfully and artfully, data has an almost hymnal power to persuade decision makers.  And when exploring data of high complexity and dimensionality, the kind that lives in Oracle’s databases, tools that accelerate the “mean time to pretty chart” are essential.

In addition, analytics tool users are right to expect a smooth user experience on a par with other tools, whether photo editing or word processing, when they are creating and exploring data visualizations.

Yet amidst all of Oracle’s presentations and marketing materials about big data and analytics, one finds not a single dashboard or visualization to stir the senses.

While Spotfire and Tableau are notable exceptions to this critique, on the whole, the tools that dot the Oracle landscape lack either brains or beauty.

Enterprises will be slow to wake up to these realities, and Oracle will continue to profit handsomely from their slumber.

However, the opportunities abound to chip away at the massive market share that Oracle now holds, providing data services to start-ups who refuse to pay Oracle’s prices, or helping medium-sized businesses migrate to new solutions.

Michael Driscoll is the CTO of Metamarkets (see disclosure), a data analytics firm.

Disclosure: Metamarkets is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

Image courtesy of Flickr user crazytales562.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010