He recently published a book called The Signal and the Noise that looks at the role data plays in our daily lives and the way we can use it to better understand the world around us. But speaking at a breakneck speed in front of a large audience in Austin on Sunday, Silver addressed some fairly random topics and questions:

1. Why the competition matters

“I’ve tried to work in fields like politics and baseball where the competition is not very good.” Silver said the lesson is important for startups, too: if you’re doing the same service that 98 percent of your competitors provide, there’s less room for growth.

2. On moving

“There were too many cool people moving to Brooklyn, so I had to move back to Manhattan.”

3. Who would play Nate Silver if they make a FiveThirtyEight movie

“I really hope they don’t make a movie about me. But I would think Brad Pitt. In terms of appearances.”

4. How he got his start

“Initially when I was a ten-year old I really wanted to build my fantasy baseball league. And I kept drafting … really crappy players and I didn’t understand why I couldn’t win. So with me it’s often been about competition, about wanting to win my NCAA tournament pool or my fantasy league.”

5. On fame

“Ironically I think I’m getting way too much credit now,” Silver says. “We’re very results-oriented as a society.” Silver said he’ll become uncomfortable if FiveThirtyEight ever discourages people from voting as the predictions improve, something he’s discussed before.

6. On the best way to pick a March Madness bracket:

“Travel distance matters a lot to performance.” Silver said that FiveThirtyEight does March Madness bracket predictions, but noted that people filling out their own should remember the correlation between travel distance and performance.

7. On one of the hardest things to predict:

“We had a formula that tries to predict the numbers and it’s only about 70 percent right,” Silver said of the Oscars. “That’s a field where you don’t have very good data available.”

8. Why he’s misunderstood by critics

“I think sometimes people mistake what I do as someone who’s saying everything we do is predicable…. Whereas really I’m more of a skeptic of prediction,” he said. “What I’m actually doing is taking polls and averaging them and the fact that it’s so surprising says a lot about where we have to go in terms of science and math.”

9. The strangest model he’s ever been pitched on

“I got pitched one time by a guy who works for a cricket team in India and thinks there can be like a Moneyball revolution in Indian cricket. I wasn’t too psyched about that.”

10. One piece of data he uses to make decisions in everyday life

“Don’t fly out of JFK in New York in the evening in the summer. If you fly out in the morning, and I’m not a morning person, but you probably have a much better chance of not having delays cascade across the system.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Pervasive Software retools for cloud, big data: will it be heard?
• AWS Storage Gateway jolts cloud-storage ecosystem

The White House Office of Science and Technology Policy will host a  live webcast Thursday at 2 p.m. eastern time to outline how the government can “help big data” with its Big Data Research and Development Initiative.

Here are three things the feds could do right off the bat to promote better use of big data:

• put the government’s own data sets into open formats
• push states to include a data or statistical literacy component in their education plans
• establish ways to continuously collect data on prescribed topics as opposed to relying on temporary snapshots

Speakers at the event will include John Holdren, assistant to the president and director of the White House Office of Science and Technology Policy; Subra Suresh director of the National Science Foundation, and Marcia McNutt, director of the US Geological Survey.

There has been a focus lately on bulking up big data skill sets to meet the burgeoning demand for data scientists and statisticians, even artists who can help visualize important big data findings in a way that makes sense to lay people. Northwestern University’s McCormick School of Engineering for example, will start offering a masters degree in analytics starting next year.  The big data skills gap was also a hot topic at GigaOM’s Structure: Data conference last week.

Photo courtesy of Flickr user DUCKofD3ATH

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

In the broadest definition, a data scientist is someone who enables the exploration and discovery of what is this massive data flow is telling us.

I spoke with Anjul Bhambhri, IBM’s Vice President of Big Data Products and the 2009 recipient of the YWCA of Silicon Valley’s “Tribute to Women in Technology” Award. She has 23 years of experience in the database industry with engineering and management positions at IBM, Informix and Sybase and very broad view of what a data scientist can be:

What we are seeing here is that this [data flow] has created a role that needs a discipline — data scientists who explore what is happening outside the organization and gain insights to the business and pass it on to decision makers and other interested parties. We need to make this a part of our regular exploration.

In the old days, and for many organizations today, business analysts would ask a question and IT would provide the answer after figuring out how to structure the queries. Bhambhri says:

Now the IT group has to make sure that their data platform is all inclusive (not just internal databases and repositories); they must integrate data from all sorts of sources — but in this case they don’t know what the questions will be. IT has to provide data without knowing what the business folks are going to ask. And the business folks need the ability to explore, play around, ask ad hoc questions, and then see trends — maybe then they go back to IT with set questions for formal reports.

The new role of data scientist is helped by a background in statistics and math, but Bhambhri does not think it is mandatory. Advancements in available tools that expose the data and allow for visualization of the data have opened the process such that people can focus on their own business domain expertise as they formulate their questions. (See Ryan Kim’s coverage on some of these big data tools.)

Are you ready to be a data scientist? What does it take?

Pete Warden is a frequent commentator on Big Data issues.  He says:

There is no widely accepted boundary for what’s inside and outside of data science’s scope. Is it just a faddish rebranding of statistics? I don’t think so, but I also don’t have a full definition. I believe that the recent abundance of data has sparked something new in the world, and when I look around I see people with shared characteristics who don’t fit into traditional categories. These people tend to work beyond the narrow specialties that dominate the corporate and institutional world, handling everything from finding the data, processing it at scale, visualizing it and writing it up as a story. They also seem to start by looking at what the data can tell them, and then picking interesting threads to follow, rather than the traditional scientist’s approach of choosing the problem first and then finding data to shed light on it. I don’t know what the eventual consensus will be on the limits of data science, but we’re starting to see some outlines emerge.

These are exciting times as advances in technology are opening up new roles for people in organizations. Are you one of the many who are asking for more and better data to do your job?

Image courtesy of Flickr user x-ray delta one.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Listening platforms: finding the value in social media data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Our companies have strong appetites to learn more ways to leverage data as a competitive tool.

Greylock Partners’ Reid Hoffman has been talking about the primacy of data for a while now. With former LinkedIn colleague DJ Patil‘s appointment as data scientist in residence, Hoffman is taking steps to ensure that Greylock Partners’ portfolio investments maximize the potential of the data that they collect. Hoffman and others recognize that timely and effective analysis of data can offer real competitive advantage, even in mature markets like retail.

Within Greylock’s portfolio, companies such as Zipcar might analyze data to ensure that its cars are parked in optimal locations, and Cloudera could tailor the products and advice it offers in order to make it even easier for its customers to work with data at scale. This new thinking about data and data scientists illustrates the broader shift that is slowly under way, a shift from simply collecting big data toward gathering and using data in order to make businesses better informed.

For more on the value of data and how companies are harnessing it, see my latest weekly update at GigaOM Pro (subscription required).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data is real, but only when you ask the right questions
• The importance of putting the U and I in visualization
• A near-term outlook for big data