The key is developing the tools and techniques to properly collect, harvest, and analyze that data. On Wednesday and Thursday this week, we’ve gathered some of the top minds in this emerging field in New York at Structure:Data to help you understand how data and data analysis can improve your business and your life. They include Paul Maritz, chief strategist at EMC and a leader of The Pivotal Initiative; data experts from companies like Facebook, Microsoft, IBM and a host of startups; and even the CTO for the Central Intelligence Agency, Ira “Gus” Hunt.

More information about the conference can be found here. If you can’t join us in New York, follow along with the livestream here, or join the conversation on Twitter with #dataconf. We’ll also post a roundup of all our coverage from what should be a very interesting two days of data.

Wednesday:

• Forget FICO: how data is changing the rules of credit and underwriting 
• Kleiner Perkins’ Michael Abbott: It takes two (teams) to build a successful app
• Without human input augmentation, algorithms alone are making us dumber
• SDN can turn the network into a big data “curator,” claims Juniper
• It’s not Skynet yet: In machine learning there’s still a role for humans 
• Data science is not enough. We need data intelligence too
• If you think big data is big now, just wait for the internet of things
• EMC’s Paul Maritz: it takes leadership to move companies toward a data-driven future
• How Aetna uses patient data to prevent diabetes and heart attacks
• Even the CIA is struggling to deal with the volume of real-time social data
• How coding contests can be better at solving problems than Harvard
• For big data achievements, IT and analysts need to work together
• Big data is still hard, but it gets better
• From Amazon’s top data geek: data has got to be big — and reproducible
• Beyond the Like button: Putting social networks to work for us
• Do we need internet exchanges for public cloud data portability?
• People will give up their personal info if you give them a good reason
• Six ideas from entrepreneurs for solving your big-data problems

Thursday:

• Why Nuance sees the semantic web as a key to smarter natural language interfaces
• Big data analytics is great but it’s no a silver bullet
• It’s not enough to just have information — intelligence requires context
• Hadoop: “It’s damn hard to use”
• Want a bigger/greener/more agile data center? Use the data
• Getting beyond the cult of big data
• How search can solve big data problems
• Hadoop applications abound, but Hadoop still needs improvement
• If the future of BI is Hadoop, SQL and the cloud are the glue
• Why Guavus analyzes lots of telecommunications data before storing it all
• Pursuing big data utopia: What realtime interactive analytics could mean to you
• Nasdaq on the virtues of the public cloud
• No, not every database was created equal. Here’s how they stand out

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Why the big data startup boom will likely be short-lived
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

• The history of Hadoop: From 4 nodes to the future of data
• The Hadoop ecosystem: the (welcome) elephant in the room (infographic)
• 5 reasons why the future of Hadoop is real-time (relatively speaking)
• Hadoop through the years: A GigaOM retrospective

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Don’t hold your breath for a single big data stack
• Big data budgets on the rise

Think about this like Hadoop’s greatest hits, but know that there will be more to come. Although the big data discussion is moving away from Hadoop somewhat, it’s still an integral — if not the integral — part of the discussion around data infrastructure. We have two great panels on Hadoop at our Structure:Data conference March 20-21 in New York (which include participants from Facebook, Platfora, Continuuity and EMC’s Pivotal Initiative (whose leader Paul Maritz will also be speaking), among others) and will keep up with all things Hadoop and data for the next 10 years.

The biggest news

1. Hadoop-focused startup Cloudera raises $5 million (March 15, 2009)
2. Friends on the move: Hadoop, AOL & PayPal  (Aug. 10, 2009)
3. Survey: Hadoop is great, but challenges remain (Sept. 29, 2010)
4. Yahoo suggests MapReduce overhaul to improve Hadoop performance (March 17, 2011)
5. Meet MapR, a competitor to Hadoop leader Cloudera (March 24, 2011)
6. EMC makes a big bet on Hadoop (May 9, 2011)
7. Exclusive: Yahoo launching Hadoop spinoff this week (June 27, 2011)
8. Microsoft’s Hadoop play is shaping up, and it includes Excel (Feb. 28, 2012)
9. VMware aims for Hadoop on VMs with ‘Serengeti’ project (June 13, 2012)
10. Cloudera makes SQL a first-class citizen in Hadoop (Oct. 24, 2012)

The best analysis

1. The data mining renaissance (April 10, 2009)
2. Is Hadoop champion Cloudera the next Red Hat? (Oct. 2, 2009)
3. Meet the big data equivalent of the LAMP stack (Aug. 1, 2010)
4. As big data takes off, the Hadoop wars begin (March 25, 2011)
5. Hadoop’s civil war: Does it matter who contributes the most? (Oct. 7, 2011)
6. 5 low-profile startups that could change the face of big data (Jan. 28, 2012)
7. What it really means when someone says Hadoop (Feb. 6, 2012)
8. Hadoop jumps through hoops, becomes mainstream (March 3, 2012)
9. Why the days are numbered for Hadoop as we know it (July 7, 2012)
10. A few stats, rumors and stories on Hadoop’s rapid growth (Nov. 9, 2012)

The coolest users … aside from Yahoo

Taking Hadoop to the stage

The Hadoop Meetup (May 1, 2008)

Cutting (center) flanked by Baldeschwieler and Om Malik at GigaOM’s Hadoop Meetup in 2008.

Next-generation data stores (Structure 2008; start at 57:00)

Hadoop, NoSQL and webscale data (Structure 2009)

The big data tsunami (Structure 2010)

Hadoop and beyond (Structure: Data 2011)

What’s next for Hadoop? (Structure: Data 2012)

Mike Olson on Hadoop (Structure: Data 2012)

Analyzing data with HBase (Structure: Data 2012)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

With more network end-points and more digitization, it goes without saying that the amount of data in our lives and at work is only going to increase. But the size of the data isn’t the issue; instead, it’s “what you do with the data” that will be the key to the success in the emerging future economy. The companies (and individuals) who don’t think accordingly will find themselves on the losing side. Let me tell you three personal stories that will illustrate my point.

Getty Images

Grounded

The first story involves an airline — Lufthansa, the German giant. I recently visited my parents in Delhi and a day before I returned I fell sick. I was quite feverish and somewhat of a pain to my fellow travelers. I developed a nasty cough and, well, I thought it might be a good idea to buy an upgrade and go to sleep on a lengthy (22-hour) flight. Instead of trying to use my points  — I know better — I offered to buy an upgrade. But a Lufthansa official declined to sell me the upgrade. It was not that there weren’t any empty seats — there were many. But since I had an unchangeable ticket, he refused. It was mildly irritating because I had been patronizing Lufthansa for nearly two decades and was hoping for a little compassion.

That episode made me wonder about why Lufthansa was so rigid and refused to use historical data they had on me to make a smart decision to appease a returning customer, especially since it allowed them to monetize empty seats. The inflexible policies basically lead the airline to leave money on the table.

In the age of big data and smart enterprises, how can a company not have a way to make smarter, real-time business decisions? I wonder if “out data-ing” will be the right way for a competitor to eat the German carrier for lunch. This lack of ability to not know the customer is going to be what I believe we will mean when we say “big, dumb company.”

I, for one would like my airline to know me, know my tastes and if possible have enough data on me to offer me a quasi-personalized experience. Yes, I do live in the future and sometimes get carried away about the possibilities of data, sensors and the notion hyper-personalization. But still, I am not talking mining on the moon — I am talking about tactics little companies like Uber are using in making smart customer decisions.

Disconnected

The second story involves a wireless company — Verizon Wireless. Every time I leave the country, I call them up and sign up for the traveling data plan, letting them know where I am traveling and for how long. I actually don’t mind doing that because it makes a life a lot easier when I land in a new country. A week later when I return, I get ominous-sounding SMS alerts followed by a phone call from one of their agents who in an alarmist tone asks me if I have my phone and what not. Most of these calls are at early hours of the day when I am trying to deal with jet lag.

I cannot figure out why the carrier cannot figure out — using location data it obviously has — that I am actually back in the United States and in my city and perhaps even in my own neighborhood. As a customer, it would certainly be more convenient. I mean, these guys are willing to sift through my location data and my phone calls to do targeted advertising, why can’t they reconcile my location with their other databases to automatically update the records?

Shoe Side Story

Now let me tell you another story about a little store in Manhattan. In sharp contrast to my Lufthansa experience, I was reminded that I had a great visit to a shoe store in New York on my last visit to the big city. It proved to be educational. I had about half an hour between meetings and I walked through Soho, where I spotted Varda.

The last time I was there it was about 10 years ago, a few weeks before I moved to San Francisco. I had bought a pair of boots at the store. I was surprised that the store had survived the test of time and it was still going strong (it had started in 1981). The fact that they made shoes that last forever might have something to do with it, I imagine.

As luck would have it, I was wearing the very same boots. They have given me excellent service and with the exception of being comfortable like old shoes can be, they are almost new. I decided to duck into the store — after all, I had a little bit of time. I saw that they had an identical pair to the one I was wearing, except they were made with suede of a different hue. And I am sucker for suede and boots.

The salesperson and I ended up in a conversation about shoes and when she swiped the credit card, she noticed that I had done business with the store previously on a couple of occasions. She gave me an instant discount — without me asking for it. It wasn’t a lot, but it was a nice feeling of being appreciated for my loyalty.

Data designs experiences

A small store like Varda created an experience that was all-encompassing and got my money. Lufthansa just alienated me, after twenty years of blindly buying from them. I don’t think there was any big data involved at the shoe store: the aging PC probably was pulling data off Excel or something similar. It was micro-data if there was any.

A few weeks ago, I wrote about how a brand experience is multitouch and multimodal. I don’t think large industrial-era dinosaurs like airlines such as Lufthansa and American Airlines quite understand that. And that is why it doesn’t matter how much data they have collected about their customers or how many millions of dollars they spend on their computing and data infrastructure. They don’t know one simple truth: it is not the data, it is what you do with it, stupid. 

Asking the right questions from the data and then creating an experience befitting customer happiness or drawing conclusions that are not obvious involves a level of humanity — something that is unfortunately missing from all the buzz about data. It is a pervasive problem across the industrial landscape.

As for me, I am shopping for a more-intelligent airline — one that values relationships and creates tailored experiences for me, the customer.

[Structure Data 2013: We will be discussing a lot about data and what you do with it at our upcoming Structure Data conference. For instance, we will have  Mohan Namboodiri, VP, Customer Analytics, Williams-Sonoma talk about how the lifestyle company uses data. The conference is being held on March 20 & 21 in New York. More details here.]

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Listening platforms: finding the value in social media data

A couple of points about the methodology might be valuable: The first is that these are products and projects that are built with Hadoop in mind and that aim to either extend its utility in some way or expose its core functions in a new manner. Another is that the “Hadoop Repackaged” category is used to characterize companies that are reselling Hadoop in some way (e.g., in an appliance or as part of their existing product suites) but that haven’t developed their own technology at the Hadoop level and instead rely on existing distribution software from companies such as Hortonworks or Cloudera.

This is the second installment of our four-part series on the past, present and future of Hadoop. Part I is the history of Hadoop from the people who willed it into existence and took it mainstream. Part III will look into the future of Hadoop and serve as an opening salvo for much of the discussion at our Structure: Data conference March 20-21 in New York. Finally, part IV will highlight some the best Hadoop applications and seminal moments in Hadoop history, as reported by GigaOM over the years.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Don’t hold your breath for a single big data stack
• Big data budgets on the rise

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Because LexisNexis has so much data that it needs to analyze and provide to clients for its legal and government services, Escalante said that the company began building its own internal data-handling platform almost a decade ago, before “big data” even became a buzzword. “We already run our business on this, end-to-end,” he said. Once it became obvious that Hadoop was becoming a popular solution, LexisNexis decided to open-source the project and use the knowledge of a community of users and developers to improve and expand it.

Escalante said the LexisNexis’ system offers a number of features that Hadoop doesn’t, including a big-data delivery engine, and that it is building a layer that will allow its system to handle data from Hadoop. In fact, he said in a recent test a single LexisNexis node was 20-percent faster than a multi-node Hadoop configuration. But the biggest advantage that LexisNexis has, according to Escalante, is that because it is a large company and has already been using the system internally for years, the banks and insurance companies that make up a majority of its clients are more likely to want to use it than Hadoop.

“We have most of the banks and insurance companies as clients, and we are doing proof-of-concept tests with many of them now, and I think they may be more comfortable working with a company that’s not a startup,” Escalante said. “Big companies want a neck to squeeze sometimes, and LexisNexis has a big neck.”

Watch the livestream of Structure:Data here.

Update: This post was updated to reflect that moderator Derrick Harris described the project as a “Hadoop killer,” not Escalante.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Your corporation is watching you, and it might be using Cataphora’s software, which mines employees emails, IMs and other electronic communications to determine how big of a risk a corporation might face from one bad apple.

Calling it software that can detect “people who are weird along many different dimensions,” Elizabeth Charnock CEO of Cataphora, claimed that the software isn’t intruding on an employee’s rights to privacy, because that right can’t really exist in today’s office environment where 90 percent to 95 percent of employer communication is electronic, and thus hidden to managers.

She compared today’s environment to a few decades past, when office workers would have to send communication in open mailers that passed through the mailroom and talk on phone calls in earshot of many other people. But today, someone can’t manage by walking around and trying to overhear problems. Now those problems have migrated to Facebook and email, so it makes sense that managers follow them there, she said.

“When we do hear people call it creepy, it’s a result of people being misinformed. Seventy percent of companies monitor employees electronics communications,” Charnock said.

Cataphora doesn’t just monitor those communications, however. The technology helps them filter those conversations and establish the digital tone of an employee, which is slightly different from looking for triggering keywords in an email or IM conversation. Charnock said this tool, like any tool, can be used for an employee’s benefit. She cited the example of a French company that is using it to help determine who the best managers are, despite that organization having a bunch of regional offices with few employees. There, now don’t you feel better?

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• AWS Storage Gateway jolts cloud-storage ecosystem
• A near-term outlook for big data

Pivotal has been involved with many web and mobile startups, including Twitter. The company’s agile development chops made it a favorite hunting ground for other companies such as Square. It is not clear to me why EMC acquired Pivotal, though a reasonable guess would be to take agile development and Pivotal methodology into the enterprise.

Changing Landscape

It would a different — and a smart — kind of services division for a company like EMC. As I have said before, large hardware vendors are facing a unique and new challenge from the likes of Amazon, Microsoft and Google, who are getting the attention of the next generation of businesses. In order for these hardware vendors to stay relevant, they need to offer their hardware as a service – storage or computing for example – and then attract others such as startups and enterprises to their platforms.

PS: EMC Greenplum’s Scott Yara will be speaking with me at our Structure Data conference in New York on March 21st & March 22nd. I guess he and I will have something to talk about!

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• AWS Storage Gateway jolts cloud-storage ecosystem
• A near-term outlook for big data

My web database comparison using Visual.ly.

On Monday, infographic hub Visual.ly began its quest to bring this idea to the masses, releasing a new feature that lets users instantly create infographics based on statistics from competing Twitter accounts or from their own Facebook accounts. While the tool’s current utility is questionable, Visual.ly Create could have some profound effects on the future of visualization as it opens its doors to different data sets.

Experimenting with the new Visual.ly Create feature as it currently works, it’s easy to spot the limitations. Showdown-style infographics comparing Twitter followers aren’t particularly useful for anyone not somehow connected to the social media field, and visualizing data from one’s own Facebook page isn’t exactly an ideal vehicle for spreading meaningful information. They’re fun, yes (so fun, in fact, that the system was hammered all day and resulted in some serious processing delays), but fun only goes so far.

However, co-founder and Chief Content Officer Lee Sherman told me the company is planning some serious customization options that I think could help introduce a generation of children and teenagers to working with data and visualization. We’re not talking about visualizations as unique or advanced as what you see certain data journalists doing — the type of stuff that pops up on Flowing Data every day — just a tool that makes it easy, fun and rewarding to spend a little time thinking about data.

The problem is that we need more people with math skills to meet growing employer demand for data scientists and data analysts. But how do you get started caring about data in the first place when the barriers are so high? Really working with data requires a deep understanding of both math and statistics, and Excel isn’t exactly a barrel of monkeys (nor are the charts it creates). Applications like Tableau make it a little easier and the results more visually stunning, but they’re still business intelligence products aimed at professionals. At our Structure:Data event next week in New York, we’ll talk about all sorts of new tools for making big data easier, but none are apps in the sense that today’s teens are using them.

This wasn't made with Visual.ly. (http://blog.thejit.org/2012/02/27/wind-motion-patterns/)

Think about every lame report you ever wrote or every essay you turned in that was based on little more than your own opinions. If it were relatively simple to produce a visually captivating graphic that used data to illustrate the point you were trying to make, or to pose a point of comparison for consideration, wouldn’t you have used it? Quantified-self apps are hot right now in part because people get addicted to data once it’s made easy to digest. But, as with visualizing your Facebook profile, that type of data isn’t very useful beyond your own body.

That’s where Visual.ly — whose founders came from personal finance startup Mint.com — comes in. Sherman said they’re trying to marry the ease of use that exemplifies Mint with the greater world of data out there. And while right now that means just making some instant graphics using Twitter and Facebook data, that will soon mean access to various APIs and publicly available data sets, as well as letting users upload their own data and even mashup data sources. Ultimately, Sherman said, users will be able to move away from prepackaged infographics and actually edit the fields themselves.

Whether it’s Visual.ly or something similar that ends up making infographics and other visualizations mainstream, it has to happen. The data revolution is too important to remain the haunt of only the best and the brightest. But it will take something to convince everyone else to give data a chance — and the ability to create a worthwhile visualization via an app could be that something.

Wind-motion patterns image courtesy of Nicolas Garcia Belmonte.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing and trickle-down analytics
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

The patent system is broken — on that almost everyone agrees. There’s a backlog of applications that results in exorbitant wait times to get a patent issued, and the merit of patents that do get granted is often questionable. If you’re forced to litigate a patent-infringement suit — an increasingly likely scenario – the costs can be crippling.

When it comes to software patents, the situation is particularly dire, which leads many critics to argue that software patents should be abolished altogether. Patent trolls are a widely cited nuisance, but there’s a more fundamental problem. Litigaton is expensive, but litigation is all too common because there are so many software patents out there, and it can be very difficult — and very expensive — to find out whether a new invention possibly infringes on even one of them.

As we’ll discuss in depth at our Structure:Data conference in New York later this month, techniques such as machine learning and natural-language processing are already having transformative effects in a number of fields. Why not the patent system, too?

Software patents don’t scale …

Timothy B. Lee, a Cato Institute fellow (and frequent Ars Technica contributor), and Christina Mulligan of Yale’s Information Society Project explore one big software-patent problem in a new research paper titled “Scaling the Patent System.” The gist of Lee and Mulligan’s argument is simple: software is such a wide-ranging and nebulous topic that it’s nearly impossible to index software patents in a manner that would make it easier to search for them. The system just doesn’t scale.

Current USPTO search engine

Property records are easily searchable because county recorders organize them in a logical manner based on geography. Even chemical patents, the authors point out, are relatively easy to search by chemical formula. With software patents, however, there’s no such luck:

[I]n the absence of a precise, standardized scheme for classifying software inventions, patent applicants are free to use any terms they like — or even make up new ones — to describe their software inventions. The scope of a patent’s claims will not always be obvious from a patent’s title or abstract. And a single software patent can claim multiple applications that are only loosely connected to each other.

Lee and Mulligan’s paper doesn’t even touch on the problems that arise with prior art, generally defined as “all information that has been disclosed to the public in any form about an invention before a given date.” It only compounds the issue of searching the USPTO database when attorneys or patent examiners are forced to search articles, presentations and anything else that might negate the novelty of a proposed invention.

Unfortunately, the authors conclude, “Only dramatic reforms — such as excluding industries with high discovery costs from patent protection, establishing an independent invention defense, or eliminating injunctions — can return the patent system to its proper role of promoting innovation.”

… but big data does

Looking outside the law, though, and into the world of big data analytics, one needn’t look too hard to find some methods for making it easier to search for patents. The answer lies in semantics. If the problem is that keyword searches aren’t effective, then build a search engine that addresses a wide variety of sources and that takes into account related terms based on how frequently they’re linked, or based on the ontologies present in different industries.

• A startup called Apixio is already doing something similar in the field of medical records. It uses natural-language processing, machine learning and sematic association to make its Medical Information Navigation Engine (MINE) as easy to use as possible. Describing the service last April, I wrote that “when a doctor types a patient’s name and ‘chest pain’ into the search box, MINE is able to find ontological references to chest pain that bear little resemblance to the actual term.”
• 

  Factually accurate, but irrelevant connections for Vanessa Laine

  Another method for doing this comes from Gravity, a startup that uses a hybrid man-machine process to personalize content for readers of sites such as the Wall Street Journal. Gravity’s system is complex to say the least (here’s a video tutorial that explains part of it), but the gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

• Even IBM’s now-famous Watson question-answering machine could prove beneficial if the USPTO were to leverage its capabilities. The system has actually been suggested as an aid to help judges better interpret statutes against the Constitution, but loaded with patent data, it could help identify potential infringements and even answer with some certainty which ones might be the most relevant to any given application.

Indeed, a startup company called IP Street is already attempting to bring the benefits of semantic technology to bear on the patent field. By analyzing the entire library of patents issued by the USPTO, Founder and CEO Lewis Lee told me IP Street is able to extract meaning from patents using information from the patent claims. A succinct explanation on the company’s website explains that “[the core] technology, known as LSI or latent semantic indexing, uses complicated mathematics and matrix decomposition (SVD) to identify similarities among documents. This allows you to enter an entire document (such as a product description, idea for a patent, etc.) and compare it to the universe of patents and patent applications—comparing across just the claims or the entire document.”

Big data won’t solve all the complaints people have about patents, but it could make life a lot easier for the inventors, attorneys and examiners tasked with determining whether a patent infringes a previous patent, or is even patent-worthy in the first place. The question now is whether the USPTO wants to leave simplification of the process in the hands of private parties like IP Street, or if the agency wants to bring a few big data experts on board to improve what it’s able to offer those who rely on it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• Takeaways from the second quarter in cloud and data
• Cloud computing infrastructure: 2012 and beyond