The White House Office of Science and Technology Policy will host a  live webcast Thursday at 2 p.m. eastern time to outline how the government can “help big data” with its Big Data Research and Development Initiative.

Here are three things the feds could do right off the bat to promote better use of big data:

• put the government’s own data sets into open formats
• push states to include a data or statistical literacy component in their education plans
• establish ways to continuously collect data on prescribed topics as opposed to relying on temporary snapshots

Speakers at the event will include John Holdren, assistant to the president and director of the White House Office of Science and Technology Policy; Subra Suresh director of the National Science Foundation, and Marcia McNutt, director of the US Geological Survey.

There has been a focus lately on bulking up big data skill sets to meet the burgeoning demand for data scientists and statisticians, even artists who can help visualize important big data findings in a way that makes sense to lay people. Northwestern University’s McCormick School of Engineering for example, will start offering a masters degree in analytics starting next year.  The big data skills gap was also a hot topic at GigaOM’s Structure: Data conference last week.

Photo courtesy of Flickr user DUCKofD3ATH

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Appistry isn’t a household name, but FedEx, one of the company’s most famous clients, certainly is. FedEx uses Appistry’s computer infrastructure and algorithms to help its employees deliver overnight packages as reliably as possible. And there’s a simple philosophy behind company’s approach: “we’re about moving the work to the data instead of moving the data to the work,” Meghji said.

That type of approach requires sophisticated algorithms, and Meghji bemoaned the fact that graduate students are the ones developing many of these algorithms as research projects. The world of big data needs better algorithms far more than it needs better hardware, he said.

We might get closer to that goal if more people were taught at an earlier age to appreciate and respect the value of big data and data analysis, he said. “Across the professional spectrum, the understanding of data–the value of it, what math matters, what it implies… we do not teach that anywhere with any seriousness.”

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

(c) 2012 Pinar Ozger. pinar@pinarozger.com

The data itself, which can be log files from servers, or from sensors attached to appliances or air pollution monitors,  is usually “gibberish,” said Erik Swan, CTO and co-founder of Splunk, a pioneer in aggregating and processing machine data, at the Structure: Data 2012 event in New York on Wednesday.  ”It’s not designed for end users.”

But that doesn’t mean end users aren’t using it.

During last year’s Fukushima radiation crisis when people felt the Japanese government was not providing enough information about radiation levels, an ad hoc network of citizens worked together to string together Geiger counters and link them up to Pachube.com.

“Within 10 days, there were more than 1,000 feeds updating several times per minute,” said Usman Haque, founder and CEO of Pachube.com, the British startup that aims to make this sort of  machine data more available and understandable to mere human beings.

But the data aggregation was just the start. People wrote apps — like the Android-based Winds of Fukushima that melded geo location and wind data to predict where the radiation would spike next, Haque said.

“Some were domain experts and some were just people who were willing and worried,” he said, but the upshot was that consumers were able to use this data for a very important purpose.

Swan agreed that the world is at the beginning of that growth curve.

“This is like the paradigm shift around e-commerce years ago when users didn’t buy much at first but then Amazon came along and showed [e-commerce's] power, then all companies became commerce companies and consumers expected everything to be available that way,” Swan said.

In the future, consumers will expect to have a “data relationship” with every company they do business with and to have a dashboard based on their prior transactions and other pertinent interactions to help them make decisions.

Some non-technology issues — like the ownership of all that data — still need to be resolved, however. ”Consumers are shocked to find they don’t own their data. In some cases someone else might own it and in some countries, no one can own it,” Haque said.

There is also a dearth of applications — the Fukushima app notwithstanding — that tap into that data.

Haque likened the current state of affairs to the 1995 “green-screen” era of computing . “We have no idea of what’s coming in 5 years… but we see people out there using these systems now, building services, even individuals automating their homes using the Nest thermostat. It’s just a question of providing discoverability into this world [of machine data] that’s already being built.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to Make Cloud Computing Greener
• Big data 2013: key trends and companies to watch
• Real-­time query for Hadoop democratizes access to big data analytics

(c) 2012 Pinar Ozger. pinar@pinarozger.com

For Pivotal Labs, it will be business as usual — only more so — after EMC’s purchase of the company . That’s according to Pivotal CEO Rob Mee. GigaOM broke the news of the acquisition last week.

There has been some concern over Pivotal’s future as part of the EMC behemoth, which Mee did his best to alleviate.

“We’re not going anywhere and we’ll keep getting better,” Mee told attendees at Structure: Data 2012 on Wednesday. Pivotal Tracker, the company’s lightweight  project management tool  ”is still going but will get more resources and more emphasis.”

Mee shared the stage with another EMC bigwig, Scott Yara, SVP of products and co-founder of Greenplum, the data analytics company EMC purchased two years ago.

Yara reiterated EMC’s intention to let Pivotal focus on what it does best. Greenplum and Pivotal worked together on a project over the past year, a collaboration which helped lead to EMC buy.

The world needs a smarter way to write software and next-generation apps — the problem Pivotal attacks, said Yara. “We want to bring Pivotal services worldwide … letting Pivotal be Pivotal,” he said.

Parent company EMC has “really interesting branding — almost a branding problem,” Yara said. “It’s worth about $60 billion and people still think of it largely as a storage company — which it is. But it bought [security leader] RSA eight years ago, then VMware before spinning part of it out, and then Greenplum in 2010.

EMC under CEO Joe Tucci is a “savvy acquirer — it acquires outside the storage marketplace and lets those brands be,” Yara said.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Updated. Spreadsheets don’t have to be limited, argues Robert Lefkowitz, director of web development of 1010data. In a presentation titled “The Trillion Row Spreadsheet,” Lefkowitz argued at Structure:Data 2012 for why desktop-based or browser-based spreadsheets aren’t the solution if you have tons of data you need to put to work.

You can’t get a trillion rows in a [traditional] spreadsheet, but if it’s a web-based spreadsheet you can totally get a trillion rows because the spreadsheet part isn’t in the browser or on the backend, it’s in the data warehouse,” Lefkowitz said.

Trillion, of course, is a rather arbitrary number, meant to stand in for some really large number. And Lefkowitz admitted that 1010data isn’t quite to the trillion number just yet — but they plan to get there.

“We have customers running 400 billion row tables,” he said. “We expect to get (to a trillion) in the next year or so.”

Watch the livestream of Structure:Data here.

Update: This story was corrected to include Lefkowitz’s proper title.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

(c) 2012 Pinar Ozger. pinar@pinarozger.com

As the former vice president of data strategy at Yahoo, Weir has plenty of history with poking around large amounts of data. From that experience, the biggest challenges, he says, are: being able to truly exploit all the data available to you, proliferating analytics throughout your organization and driving significant business value from it.

The data he gave as an example: the 800 million Google queries per day, 250 million tweets per day and 800 million Facebook updates per day that shows what customers are either searching for and what they’re saying about your brand online.

The best way to squeeze actual value out of all these tweets, updates and searches, he said, is:

• Understanding user intent
• Knowing how people generally feel about your brand
• Pinpointing context: where and when people are talking about or searching for your brand

He gave an example of what Yahoo has been able to do. They take their gobs of data they suck in every day from their millions of users and do behavioral targeting to get higher value CPMs, Weir said. “Yahoo constructs DNA for each user. They construct models that look at behavior, score users on a number of dimensions and target ads to those users,” he said.

Besides crunching all that data, there’s a lot of modeling involved, he noted. “Not just the intensity with which [someone] is looking at a topic, but recency matters” too, another example of how important context is for deriving value from having a lot of data.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Dissecting the data: 5 issues for our digital future
• NewNet Q2: Google closes the quarter with a bang

© 2012 Pinar Ozger. pinar@pinarozger.com

According to former CIA director James Woolsey, the utility companies don’t want to think about issues like data security, so it’s up to the people working on smart grid technologies to do that. Right now, he said, they’re more concerned with adding fun new features, but it won’t be so fun if the electric grid goes down for a few days. Seventeen of the United States’ 18 pieces of critical infrastructure rely on electricity, and anything from weather to the pulse from a nuclear weapon detonated in the atmosphere could take it out.

In that case, Woolsey said, “You’re not headed back to the 1970s pre-web — you’re back to the 1790s pre-electric-grid.”

If you’re wondering where data fits into all of this, Woolsey said there’s a lot that’s accessible for a skilled-enough hacker. “The web was put together … by a bunch of absolutely brilliant people who … like me, were part of the ’60s flower generation,” he explained, and they assumed everyone would be buddies. However, as the Wikileaks diplomatic-tubes controversy illustrated, anyone who’s part of a network can inflict a lot of damage. You wouldn’t want someone who doesn’t have your interests in mind getting access to grid data and taking it down, Woolsey said.

Although the grid is vulnerable to the whims of mad men, Woolsey said methods such as distributed generation, co-generation and improved battery technology could help alleviate some of the threat. The less we rely on centralized transmission and generation, the less vulnerable any given point on the network is. The easier it is to hack, the easier it is to bring down the electric grid: “If the data disappears, it’s not of much use.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The case for low-power servers in the data center
• Data center, meet the smart grid
• Locating data centers in an energy-constrained world