Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

In the world of technology, especially, the legislative progress is just too slow, too confining and, ultimately, too shortsighted. For proof, one need look no further than all the hubbub on Thursday about a forthcoming amendment to the Video Privacy Protection Act allowing video providers to share data regarding customers’ viewing habits. In this case, the amendment was spurred by Netflix’s desire to enable “frictionless sharing” between itself and Facebook.

More than a year later, an amendment, which just awaits a signature from the president, will give Netflix and Facebook what they want — no more, no less. (Congress didn’t even bother to change the term “video tape provider” to something more current and all-encompassing (although, in its defense, the term is defined broadly).) And, as my colleague Jeff Roberts has noted, now that Facebook doesn’t appear to be the kingmaker it once was, it’s probably too late for this bill to matter.

But the VPPA situation is just a microcosm of the bigger problem facing a technology industry that’s moving at a speed Congress and federal agencies can never hope to match. Laws can no longer address a single technology or a single practice at a single point in time, but must be visionary enough to remain relevant as technologies, social norms and business models evolve.

Technologically speaking, what’s popular today might be unrecognizable five years from now, so it’s the ideas that really matter. The bigger issue behind frictionless sharing isn’t just video data, it’s the social sharing of data among web applications in general. The bigger issue behind email privacy isn’t just email, it’s about the perception of private communications whether they’re done via SMS, Twitter or some yet-to-be-developed platform. The bigger issue around file-sharing isn’t so much copyright as it is entirely new methods for consuming content.

Sen. Al Franken (D-MN) is at least pushing the discussion of new privacy concerns

But issues such as web privacy and intellectual property are actually relatively easy to wrap our minds around. We’re headed down a path toward some real science-fiction stuff, though — driverless cars, facial recognition, connected glasses, $1,000 genomes, nanotechnology, etc. – and one has to wonder how the legal system will keep up. They’re novel enough individually, even more so when you consider the possibilities of combining data from all these sources to try and uncover some never-before-detectable insights.

Who knows what life-altering technologies will pop up that we don’t want stifled by archaic laws — or what scary use cases will pop up that we will want to regulate even if we don’t have laws to do it. Who will be able to draw the connections between technology and other cultural realms or personal liberties, and to extrapolate the potentially complex relationships between tech laws and these areas?

Even if some of its members weren’t seemingly out of touch and reactionary when it comes to technology, Congress can’t even pass a budget on time. Now we’re supposed to trust it to regulate a largely digital technology future that’s evolving at breakneck speed and disrupting every business model, communication medium and biological process in its path? Maybe Ray Kurzweil’s next gig should have been in the House of Representatives rather than at Google .

Feature image courtesy of Shutterstock user Everett Collection.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Dissecting the data: 5 issues for our digital future
• Sector RoadMap: Content personalization in 2013

Chinese telecom gear makers Huwaei and ZTE are about to get blacklisted by the U.S. Congress in a report to be published Monday. Reuters reports that the U.S. House of Representatives’ Permanent Select Committee on Intelligence will recommend U.S. companies don’t use wares from the two companies over fears that their gear constitutes a security threat.

Sunday night the Intelligence Committee’s chairman, Mike Rogers, told 60 Minutes that Americans should “find another vendor” if they care about their own IP, their privacy and U.S. national security, which means that not only could telecommunications companies and data center gear buyers find themselves sans Huawei and ZTE gear, but also those interested in cheap handsets. Huawei has launched handsets in the U.S. market with rural and pre-paid carriers.

A Huawei handset for T-Mobile.

The report allegedly implies that Huawei and ZTE install backdoors and other mechanisms that allow them to spy on the packets traversing networks containing their gear. The reports also implies that these companies’ close ties to the Chinese government mean that they would share information gleaned from their snooping with the Chinese government. Thus, buying gear from these companies is akin to letting the Chinese spy on your network traffic. This same logic was used a few years back to stop Huawei from buying 3Com, U.S. networking company later bought by HP.

Huawei’s Bill Plummer emailed me the following in response to the alleged contents of the Congressional report:

Huawei is a globally trusted and respected company doing business in 150 markets with over 500 operator customers – the quality and security of our product is world proven. This investigation and report are nothing more than a politics exercise that has ignored technical, commercial and cultural realities – it achieves nothing in terms of securing networks in a world in which every major vender develops, codes and builds globally, including in China. Huawei looks forward to leaving this political distraction behind us so that we can work with rational industry and government stakeholders to develop real solutions to what are real and industry-wide cyber challenges.

Not exactly 50 shades of grey, but enough to confuse things.

This is a tough issue. Both Huawei and ZTE deny having close ties to the Chinese government and that they install such software on their gear. Yet, the Chinese government has supported both companies in their history and has a history of spying on U.S. companies. For example, Google came out in 2010, and said it had detected Chinese hacking on its network. Earlier this year Nortel, a former telecommunications gear vendor, disclosed that hackers originating from China had broken into its network.

So both Huawei and ZTE have benefited from Chinese governments (in the form of economic development loans at least), and the Chinese government is widely believed to have been a dedicated hacker. But are Huawei and ZTE guilty by association? There is also a strong hint of economic protectionism here as well. Both companies are a solid threat to Cisco and Juniper, two U.S. companies that stand to lose if their products are undercut by low-cost Chinese switches and routers. Cisco’s CEO John Chambers is a very active Republican who is vocal on this issue.

Plus, both Cisco and Juniper (as well as many U.S. companies) frequently make some of their hardware and even write some of their code in China and other places that the U.S. might consider a threat. Domestic companies point out that they don’t let engineers writing code overseas have full access to the source code, and that the foreign-produced code is reviewed, but there is an element of hypocrisy here.

Disclosure is the solution, but no one wants that

It’s cheaper to build things in China, be it software or hardware. Plus, executives at U.S. companies tell me that they never buy used networking gear from any vendor because it can have unexplained Chinese software on it. The Chinese don’t necessarily need a company in its pocket to install networking spyware, when it can sell gear on eBay to unsuspecting corporate buyers.

A source in the networking industry tells me that the solution here may be to demand a full source code review from Huawei to prove that Huawei is spying and sending what it discovers back to the Chinese. However, this person also notes that Huawei would be well within its rights to point out that the U.S. guys should do the same with code that they have written in China.

The problem standing in the way of the truth here is twofold. Problem one is that evaluating networking technology and espionage through hacking is a highly specialized and esoteric skillset, and problem two is that China’s opacity and ties to hackers, as well as the lack of transparency by both companies, make it difficult for the average person to believe ZTE and Huawei’s denials over the government’s influence and involvement in their corporate activities. So, if the U.S. House says don’t buy Huawei and ZTE gear, that will hurt those companies in this market — one where Huawei employs 1,700 people (it has 140,000 worldwide) and hopes to list on the public stock market.

Perhaps more will be revealed later today after the full version of the report is released (a classified version with more information was also prepared). The bottom line here is that when it comes to hacking allegations, China and national security, there’s a lot of self-interest and accusations based on some esoteric and difficult-to-prove allegations that can color the results of this report. However, the conclusions will undoubtedly cause economic harm to Huawei and ZTE in the U.S.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• For Operators Who Bet on WiMAX, There’s an LTE Plan B
• Where new opportunity lies in the mobile operating system space
• How the mega data center is changing the hardware and data center markets

In a post on Tuesday entitled “First they came for WikiLeaks, then the New York Times,” we wrote about how there is growing evidence that Congress and the Justice Department may be considering legal sanctions against traditional journalists who publish classified information — in other words, extending the kind of legal attacks they have been making on WikiLeaks to the traditional media such as the New York Times. In an emailed response to that post, former NYT executive editor Bill Keller said he strongly agrees that an attack on WikiLeaks’ right to publish such leaked documents is an implicit attack on the media as a whole, and that the mainstream media should protest any prosecution of the organization as a betrayal of the First Amendment.

In my post, I described how some members of a House Judiciary subcommittee seemed to be looking to experts for legal grounds under which they could charge journalists for publishing leaked classified information. The Department of Justice has also reportedly been warning reporters that if they publish such documents they could face prosecution — in the same way the DoJ is said to be pursuing a case against WikiLeaks and its controversial founder Julian Assange under the Espionage Act, (despite the fact that the government’s own researchers say using the act to go after journalists instead of leakers is a questionable strategy).

If WikiLeaks is under attack, journalism is under attack

My point was that if WikiLeaks, which I have argued before is a media entity — although one very different from the New York Times — is subject to that kind of prosecution for publishing classified information, then the NYT or any other traditional media outlet is in danger of being prosecuted as well. I also said that most mainstream media companies had been relatively silent on this point until now, but Keller noted in his email that he has repeatedly agreed that an attack on WikiLeaks is an implicit attack on media and journalism. As he put it:

I’ve said repeatedly, in print and in a variety of public forums, that I would regard an attempt to criminalize WikiLeaks’ publication of these documents as an attack on all of us, and I believe the mainstream media should come to his defense.

Keller went on to say that despite the rumblings from Congress that I referred to in my post, the government so far hasn’t made an official move against either Julian Assange or WikiLeaks. If a prosecution under the Espionage Act did in fact occur, Keller said he hoped to see news organizations of all kinds and press-freedom advocacy groups “filing briefs and otherwise objecting.” The NYT’s former executive editor also admitted that the newspaper’s relationship with Assange had been fractious, but said that personal feelings about the WikiLeaks founder shouldn’t prevent media organizations from coming to his defense:

You don’t have to embrace Julian Assange as a kindred spirit to believe that what he did in publishing those cables falls under the protection of the First Amendment.

Even if it isn’t journalism, it deserves protection

In a follow-up email, Keller also noted that he had made similar statements about the necessity of defending Assange and WikiLeaks’ publication of classified documents in a New York Times magazine piece excerpted from the introduction to “Open Secrets: WikiLeaks, War and American Diplomacy,” a book about the organization’s publication of thousands of diplomatic cables and the NYT’s role in that effort. In the piece, he said:

While I do not regard Assange as a partner, and I would hesitate to describe what WikiLeaks does as journalism, it is chilling to contemplate the possible government prosecution of WikiLeaks for making secrets public, let alone the passage of new laws to punish the dissemination of classified information, as some have advocated… criminalizing the publication of such secrets by someone who has no official obligation seems to me to run up against the First Amendment and the best traditions of this country.

As I wrote at the time Keller’s excerpt was published, it seemed as though the former NYT editor was grudgingly coming to admit that what WikiLeaks did was close enough to being journalism that — even if it wasn’t journalism with a capital J, or published by professional journalists — it deserved the full protection of the First Amendment. That’s a message it would be nice to hear from more journalists of Keller’s calibre.

Post and thumbnail images via Charlie Rose and Flickr user jphilipg

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q1: Content Farms and Niche Networks on the Rise
• What the New York Times Can Learn From Rupert Murdoch’s Paywall
• Content Farms: The Players, The Benefits, The Risks

Hey, now I can identify Sen. Al Franken!

My gut instinct is to call Senator Al Franken a well-meaning fool when it comes to his latest outcry — this time against the advent of facial-recognition software — but he actually has a point.

For the record, I have nothing against Sen. Franken (D-MN). In fact, I rather respect him. But any time I see a politician all up in arms about the problems some new-fangled technology is going to cause his simpleton, Luddite constituents, I get defensive.

By and large, I think Congress should leave new web technologies alone so they can grow organically and we, the people — who are actually perfectly able to adjust to them — can decide the limits with which we’re comfortable. If someone gets out of line, agencies such as the FTC are more than capable of bringing down the regulatory hammer on an individual basis. But facial recognition is a whole new beast.

I’m not even so concerned about government or law enforcement agencies using facial recognition to identify suspects under murky constitutional protection (although it is a legitimate concern). Rather I’m more troubled about the thought of any average Joe having this power at his own fingertips. If you’ve heard about Alessandro Acquisti’s work with the technology, you know why this possibility should be a little scary. Snap a photo of someone with a smartphone, analyze an image against a database of social media or Flickr pics and, voila, you have a name.

From there, it’s easy to get someone’s age, hometown, interests, news coverage, you name it. Given a name, an age and a city of birth, Acquisti has proven it’s relatively easy to predict someone’s Social Security number. It also wouldn’t be too difficult to start stalking a complete stranger, or to make a national laughing stock out of some innocent (but hilarious) face in the crowd. Or for a stranger to dredge up those pornographic shots of you that your ex-boyfriend posted online. Or to figure out the guy next to you on the subway is rich, follow him home and rob him.

Facial recognition is one of the reasons I believe we need an updated system of civil laws to account for the unwilling — and possibly global — publicity that the advent of social media makes possible. You don’t even need to have a Facebook or Twitter account to get thrown into the mix. All it takes is someone else snapping a casual photo, tagging you and you’re in, like it or not.

This is also why I don’t think Google went far enough with the face-blurring technology it announced for YouTube videos on Wednesday. Blocking out dissidents’ faces so they aren’t the targets of authoritarian governments is a noble goal, but it doesn’t address the issue of me being able to blur out my face in someone else’s video. I’m not sure how such a capability would work from a business perspective, but citizens need some level of control over how and where their images are used.

It’ll be slow-going, but I assume Congress or the Supreme Court will ultimately define the governmental uses of facial-recognition software just as they’ve done with wiretaps and GPS tracking, and as they should do very soon with other types of personal data stored with online services. But it’s the technology in the hands of everyday citizens that bothers me most. I don’t think I want a law saying “thou shalt not develop apps that utilize facial-recognition software,” but I also don’t want some complete stranger asking me how I liked my latest trip to San Diego.

You can watch video of Franken’s opening statement from Wednesday’s hearing here.

Feature image courtesy of Flickr user striatic.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• A near-term outlook for big data
• Connected world: the consumer technology revolution

If you’re concerned about privacy, this should be kind of scary. There’s some solace in the fact that most providers claim they deny requests they feel are overbroad or unauthorized, but they comply with most and the numbers are rising. A couple of carriers, including Verizon and T-Mobile, noted user-data request increases in the 15 percent per-year range. Further, as some companies explain in their responses to Rep. Markey, many requests ask for information on numerous parties or location data that might identify everyone who pinged off a particular tower.

Here’s a little chart I made comparing the estimated number of requests for user data from wireless providers in 2011 to that of Google. I also threw in Twitter’s recently released number from the first half of 2012, which, at a mere 679, doesn’t even show up on the chart.

Here’s a breakdown of requests in 2011 just by the carriers that provided information for last year, although all numbers are estimates because (1) some providers just provided estimates and (2) they weren’t entirely clear in differentiating between types of requests. Sprint, for example, claimed more than 300,000 court orders for wiretaps, pen register/trap and trace devices, and location information over the past five years, but also noted “approximately 500,000 subpoenas from law enforcement” last year alone.

Several providers were a little more vague than others. MetroPCS claimed “an average of fewer than 12,000 requests per month from January 2006 through May 2012.” Cricket said requests grew steadily “from a low of approximately 24,000 in 2007 to a high of approximately 42,500 in 2011.” C Spire claimed approximately 12,500 requests over the past five years, while T-Mobile declined to release any numbers. Here is AT&T’s rather detailed breakdown.

Aside from the numbers, the Times article also notes varying degrees of consistency with which law-enforcement agencies go about requesting user data in accordance with federal surveillance laws. Of course, this isn’t made any easier with ever-changing technologies and data sources, such as GPS, that begin to blur the lines between mere requests for user data and suspect surveillance that the Supreme Court recently declared unconstitutional without a warrant.

It’s yet another reason that Congress needs to get its Fourth Amendment act in order, redrafting outdated legislation to account for new technologies not even considered when many laws about data privacy were written decades ago. In the case of data such as that stored with cloud service providers or wireless carriers, that might mean focusing laws on the type of data authorities hope to access rather than where it’s located — a variable that changes with each new wave of technology.

Given the right case, maybe the Supreme Court will spur Congress to act and offer enhanced protection to everything we’re all but forced to share digitally. In United States v. Jones, the case mentioned above, Justices Alito and Sotomayor both wrote concurring opinions expressing their discomfort with how easily authorities can access digital data stored with service providers. As of 2010, many wireless providers were storing user data for years.

Sotomayor called the third-party doctrine “ill suited to the digital age, in which people reveal a great deal of information about themselves to third parties in the course of carrying out mundane tasks,” adding that she “would not assume that all information voluntarily disclosed to some member of the public for a limited purpose is, for that reason alone, disentitled to Fourth Amendment protection.”

Feature image courtesy of Shutterstock user Johan Swanepoel.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Carrier IQ and the continued erosion of operator trust
• Connected world: the consumer technology revolution
• A near-term outlook for big data

It’s tough to figure out whether that’s good news or bad news. Maybe he’s changed and it will be a lot of fun to work together. Maybe CISPA could actually be used primarily for the legitimate cause of fighting cyber attacks, and critics are just reading too much into its myriad vagaries.

Just to make sure, though, here’s a step-by-step guide Congress can follow in order to quell any concerns about how CISPA will be used.

Step 1: Figure out what you actually want

Stopping cyber attacks is a noble goal, but it’s a tad broad. This is especially true when it comes to information pertaining to said cyber attacks, which could mean anything from system-level data related to attempted attacks, to emails about hacking activity. Considering that CISPA covers the entirety of U.S. businesses and organizations, the types of information that could be shared are seemingly limitless.

That’s what has CISPA opponents up in arms, so narrow it down. They might not mind anyone sharing server logs illustrating suspected hacking incidents, but emails, phone records, download activity, you name it — well, that’s a little much.

Step 2: Let’s see some conditions

In the current version of CISPA, information sharing — and then government action upon that information — is conditionless. Everyone shares only they want to, and then they pretty much do what they want with it. The government can’t use data for regulatory purposes, but it can otherwise act upon it under the rather broad definitions of cybersecurity and national security.

Opponents might actually feel better about such broad sharing permissions if they were conditioned on the presence of specific characteristics that justify the need to share (e.g., a direct threat against a particular system, or more than passing similarities with previous attack methods). And if the government has authority to act upon any info it receives, there should be conditions on what actions it can take under what circumstances.

Step 3: Lose the black box

This is probably the bill’s single biggest flaw: there’s no accountability to anybody for anything. Not only is the information shared under this bill free from existing laws mandating the disclosure of agency records, but — and here’s the kicker — “[n]o civil or criminal cause of action shall lie or be maintained … against [anybody], acting in good faith for using cybersecurity systems or sharing information in accordance with this section.”

This goes hand-in-hand with Steps 1 and 2. Narrow what can be shared and how it can be acted upon, under what circumstances, and then have penalties in place for violations.  Otherwise, the bill reads “spy with impunity,” and very few people — not even the President — are getting on board with that.

Step 4: If all else fails, make it longer

As anyone who has read CISPA might have noted, it’s a pretty short bill. But when you’re talking about complex issues, such as privacy, you can’t simply do away with nuance and detailed explanations. Sorry, Justice Scalia.

If someone just sits down in front of CISPA — a whopping 15 pages of center-aligned text in large font — and starts writing literally anything, the bill has to get better. The current legislation is so vague as is that extra text that doesn’t somehow expand the scope of the rules couldn’t make it worse.

Step 5 (for citizens): Contact your senators

Sarcasm aside, this is serious stuff. As I’ve explained before, CISPA isn’t going anywhere because, unlike SOPA, it has actually has redeeming qualities. More importantly, it has powerful backers across Congress, industry and even the web.

What’s troubling now is that the House had a chance to amend the bill to address its shortcoming, but only did so with a few relatively toothless additions, and then passed CISPA in a surprise vote. Assuming the Senate won’t be willing to kill the bill entirely, concerned citizens must contact their senators and point out specific problems with the bill and how they might be improved.

If CISPA becomes an all-or-nothing issue, then the powers representing all will win. Let’s see if the web can learn to compromise.

Photo courtesy of Shutterstock user grynold.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Sector RoadMap: Content personalization in 2013
• Connected consumer first-quarter 2013: Analysis and outlook

The criticism that, by including a provision for the protection of intellectual property, CISPA is little more than a less-conspicuous form of the draconian SOPA bill seems misguided. CISPA is vague and unnecessarily broad, but it’s not SOPA. In fact, the very same Internet companies that were so adamantly opposed to SOPA might support CISPA. Facebook already does. So does outspoken SOPA critic Darrell Issa (R-CA). Here’s why.

1. CISPA is actually good, in theory. The idea of sharing cybersecurity information between private companies and the government has merit, especially in a world of increased cyberattacks against organizations in both sectors. If you’re trying to discover patterns in attacks, more data is always better, and web sites are attacked constantly. That they also could have access to classified government data is particularly beneficial.
2. CISPA doesn’t require service providers to do anything. SOPA all but forced service providers to monitor user behavior to the benefit of media companies (or to avoid being shut down by them), but CISPA only allows those providers to act in their own best interests. It’s unclear to me, at this point, why any company like Facebook, Google or Twitter would do anything other than obtain information on activity that directly affects the security of their platforms or their proprietary data.
3. I’m not certain the inclusion of intellectual property protection was driven by ulterior motives. For one, CISPA actually reads as if private parties can only gather information relating to their own rights and property, which would mean ISPs can’t go about monitoring for copyright infringement because they don’t own any copyright. There’s a strong argument that the bill primarily targets cyberattacks aimed at stealing data or files from a company’s servers (CISPA co-author Mike Rogers (R-MI) said as much in a press conference yesterday), although existing cybersecurity law certainly target some of that activity.

Probably the biggest problem is what a company is able to do to “protect” itself from such threats. As the EFF points out, CISPA allows companies to “use cybersecurity systems to identify and obtain cyber threat information to protect the rights and property of such protected entity.” It also grants companies immunity from lawsuits if they exercise their rights under the bill in good faith.

If the EFF is correct, companies could bypass existing laws regarding the monitoring of communications, claim good faith and — if they have a solid case — be free from liability. The EFF also talks a lot about CISPA allowing service providers to “block” sites, although it’s unclear what type of activity the bill actually allows in response to information gathered. Does it allow them to obtain information and take shutdown actions like those SOPA would allow, or just to react to information only within the bounds of what’s already legal?

It’s a little scary, then, that CISPA has such strong support in the House of Representatives. Whereas SOPA had only 23 co-sponsors, CISPA has 106, including Issa. That web companies such as Microsoft and Facebook have signed off on it isn’t too promising, either. It likely will take some powerful voices to at least clear up the vagaries of the bill, but it’s hard to see where they’ll come from this time around.

Feature image courtesy of Rob Allday.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• After the blackout: How the IT industry can stop SOPA in the long term
• Connected consumer first-quarter 2013: Analysis and outlook
• CES 2012: a recap and analysis

According to Decide.com Vice President of Product and Marketing Michael Paulson, members of the company were invited to attend the White House’s big data summit, and one of the big takeaways was that Congress might be willing to admit that it doesn’t know what it doesn’t know with regard to big data. While discussions about privacy get a lot of attention, lawmakers don’t always know how data is being used to improve the user experience. There are a lot of bills about to be written governing data use, Paulson said, and part of the summit was an effort to make sure those writing the bills don’t hinder benign innovation with short-sighted laws.

Paulson also noted that several agency members approached Decide Co-Founder and CTO Oren Etzioni a Decide software engineer and suggested they could provide datasets that might actually improve Decide’s buy-or-wait engine for electronics purchases. Decide works by analyzing large datasets on pricing and product-release information in order to give consumers an idea whether now is the best time to buy a particular product or whether they should wait because the price is about to drop, or a new model is about to come out. It also aggregates reviews, news and other info about a product so consumers can make informed decisions beyond Decide’s algorithmically generated predictions. According to Paulson, federal agencies have data on factors from energy efficiency to manufacturing location that they might be willing to share.

If you’re wondering what makes Decide such an innovative use case that it was invited to attend and speak at the summit, here are some highlights of how the service works:

• The founders and many engineers came from airfare-prediction service Farecast, which Microsoft bought in 2008. Co-Founder and CTO Oren Etzioni founded Farecast, as well as Clearforest, which Thomson Reuters acquired in 2007.
• Decide gives consumers a simple “buy” or “wait” decision on their planned purchases, as well as greater detail into how much the price might rise or fall in what timeframe, or when a new version is likely to be released.
• Paulson said Decide is 77 percent accurate on predictions and saves users an average of $54 per purchase. About 20 percent of the time, he said, there’s a good reason to wait.
• It has a 100TB-and-growing pricing database and more than 8 billion price observations to drive pricing models.
• It built the world’s large first “lineage database,” a massive store of information on what products and versions are related to each other. This — along with news and rumors — helps predict the likelihood of new models.
• Decide relies on Amazon Web Services for its operations, including Elastic MapReduce for Hadoop jobs. Other big data tools include Apache Solr/Lucene and Weka, an open source “collection of machine learning algorithms for data mining tasks.”

And we are only at the beginning when it comes to startups using data to build apps that will improve the consumer experience. Even for teams that aren’t stocked with Ph.Ds., analytic techniques such as natural-language processing and machine learning are slowly making their way into the mainstream. Just this morning, I had a call with the founders of Stazoo, a site that’s using sentiment analysis to help users determine what music and movies are hot among their friends and the greater world of Twitter users.

I have been adamant about the need for lawmakers to get a better grasp on where technology is headed so they don’t write laws that inadvertently stifle innovation down the line, and data is a particularly tricky area in that regard. If Congress and the administration are actually serious about getting educated on big data so they can ensure it flourishes, I say kudos to them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Here’s what you need to know about the proposed budget and energy:

• A total of $27.2 billion in discretionary funds for the DOE, which is a 3.2 percent increase above the 2012 level.
• $2.3 billion for the Office of Energy Efficiency and Renewable Energy (EERE), which is a rise of 21 percent.
• An elimination of $4 billion in fossil fuel subsidies for 2013, as well as $40 billion in fossil fuel subsidies over the next decade.
• Maintaining and expanding funding for the Advanced Research Projects Agency-Energy program, and asking for $350 million.
• $5 billion for basic energy research through the Office of Science.
• $310 million for the SunShot Initiative to help get solar to grid parity by the end of the decade.
• $95 million for wind energy, including off-shore wind technologies.
• $65 million for geothermal energy.
• $770 million for the Office of Nuclear Energy, which includes funding for advanced small modular reactors R&D.
• DOE Secretary Steven Chu said on a call with reporters that some funding was being pulled from energy projects, and 35 projects “did not reach research milestones.”
• Through the USDA, the budget calls for $6.1 billion in loans to rural electric cooperatives and utilities that will support the transition to a clean energy generation and job creation.
• Via the USDA, $200 million for support advanced biofuels.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Flash analysis: the Fisker debacle and its implications on investing, innovation, and government incentives
• The fourth quarter of 2012 in cleantech
• The race for cost-effective and efficient solar power