“I think today, we’re still in a world where we have brands telling us what we neeed to know about a brand’s individual products,” she said at the Glimpse social discovery conference in San Francisco on Wednesday. “But that’s not always the truth about the product, nor is it what’s interesting to you about that product. So the future will be reversed, where it’s just about the information that’s relevent to you.”

Wanelo is a small but immensely popular social shopping site where users post, like, collect and re-post images of items that are all for sale — the entire value proposition of the site is that you can click through to purchase any of the items. While the site only had 8 million users in May, Varshavskaya said 70 percent of them are coming back to the site every month, and they’re spending an average of 50 minutes a day on Wanelo. And because all of the items posted to Wanelo are selected by its users to be liked and purchased by others, it creates an interesting discovery engine for goods.

As I’ve written before, it’s actually quite a challenge to define what social shopping is. Does it mean tweeting out your purchase after you buy something? Does it mean adding Facebook Connect on a brand’s site to see which of your friends like it too? Does it mean browsing items on Pinterest posted by your friends, or reading reviews from strangers on a site? Or texting photos of a dress to your friends when you’re in a brick and mortar store?

The definition changes depending who you talk to and what kind of shopping experience you’re talking about. (And can be hard for big brands like Zappos to harness, when often people just want fast shipping and a wealth of products.)

“No one really wants to follow other Zappos users,” said Will Young, director of Zappos Labs.

AngelList founder Naval Ravikant, who is an investor in Wanelo and also spoke at Glimpse, pointed out that social validation around products comes in two broad categories: the power of the crowd to surface the best products (think Amazon reviews or topics upvoted on Reddit), versus the power of your friends to influence your tastes and opinions (think Facebook or Twitter friends).

“If I’m buying a stereo or digital camera, I don’t really care what my friends thought. What the qualitified group thinks is good enough,” he said. “But if I’m buying a piece of clothing, sometihng with taste, maybe I do care. The term social almost changes in meaning depending on the situation.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How social discovery is transforming entertainment
• Defining work in the digital age: an analysis by GigaOM Pro
• Frenemy mine: The pros and cons of social partnerships for online media companies

If you’re unfamiliar with Hive, it’s a project that Facebook developed in 2008 to make Hadoop function more like a traditional enterprise data warehouse. Hive stores data inside the Hadoop Distributed File System in structured format, and then allows users to query it using a language very similar to SQL. Until very recently, Hive has been the de facto method for querying (in a traditional sense) data stored in Hadoop, and it has proven immensely popular as more companies have begun tackling their big data woes with Hadoop.

Hive wasn’t built for speed

However, as more companies got used to Hadoop, they also began to notice its shortcomings. One of them is around MapReduce, a powerful but not-exactly-speedy method of processing data that requires running the job across every node in the cluster in order to find the right data. Although the Hive interface is that of a SQL query, it relies on on MapReduce as the processing engine.

(For more on how Hadoop and its flavor of MapReduce came to be, read this post on the history of Hadoop. To see me speak with Google Fellow and MapReduce creator Jeff Dean about how far Google has moved from a MapReduce-centric computing model, come to Structure next month.)

Users wanted faster, more-interactive query processing on top of Hadoop, similar to what they had grown accustomed to with data warehouse systems such as Teradata, Greenplum and Netezza. Hadoop vendors such as Cloudera (with Impala), MapR (with Drill), IBM (with Big SQL) — as well as a spate of startups — have obliged with their own new technologies that in various ways blend the familiarity of SQL with the scalability of Hadoop. EMC Greenplum, now Pivotal, has transplanted its existing database system inside of Hadoop.

Even Qubole, a cloud-based startup from Hive creators Ashish Thusoo and Joydeep Sen Sarma, is keeping an eye on how projects such as Impala and Shark (from the University of California, Berkeley’s AMPLab) might factor into its plans.

Giving Hive a better “Stinger”

Hortonworks, the Yahoo spinoff dedicated to driving the Apache Hadoop bus, is sticking with Hive. But is has a plan, and a point.

Essentially, VP of Products Bob Page told me during a recent briefing, “It just makes more sense from our view to have everything done in one place.” He means that Hive is already the method by which most people are already comfortable using SQL to access Hadoop data, so there’s no use rocking the boat by adding yet another technology into the mix. Hortonworks will just make Hive faster to the point (100x) where it’s at least in the ballpark of what these entirely new systems are capable of doing, but where users still use the same tools for interactive and batch queries.

It has in place a three-phase plan, under the “Stinger” codename, in order to make this happen. The first phase, now available as part of the Hive 0.11 release, is a new set of analytic functions and a columnar file format that Page says has resulted in a 50x performance increase over the previous version. The next phase is to move YARN Hive off of MapReduce and onto a still-under-development processing framework called Tez.

“You’ll see phase two come to bear later this year,” Page said, once YARN — a new resource manager that lets Hadoop clusters run multiple processing engines simultaneously — is ready for production.

The third phase is a whole new vector query engine for Hive and new tools for intelligent query planning. Page didn’t have a target date in mind for that phase, except to note that “we’re not talking about a five-year cycle.”

SQL isn’t the end game for Hadoop

It would be easy to dismiss Page’s and Hortonworks’ optimism about Stinger as a sweet lemons type of rationalization — the company was founded around Apache Hadoop and can’t really go about developing entirely new products outside that foundation — but they also appear to have their eyes focused on a future where SQL isn’t too big a differentiator.

SQL is the way folks used to data for the last 30 years can see how Hadoop fits in their environment, Page said, but the compelling thing about Hadoop “is it really unlocks a new way about how one thinks about storing and processing data.” Once YARN is ready to go, he added, there will be new avenues of innovation in areas like graph analysis and stream processing.

Page comes from a place of credibility when he talks about this evolution in thinking. Before coming to Hortonworks in March, he was vice president of analytics platform and delivery at eBay, a company that knows its way around big data. When people get all their data in one place, they want to do more things with it, he explained. The thinking becomes less about using Hadoop to lower cost and more about “How do I use Hadoop to increase my top line?”.

Besides, Page noted (echoing the sentiment of just about everybody else in the Hadoop space, including Cloudera CEO Mike Olson), even as companies turn Hadoop into their primary data store, it’s difficult to see Hadoop ever entirely replacing high-value relational data warehouse systems like Teradata. One could argue, then, that there’s no real purpose in trying too hard to match those systems in terms of capabilities.

At eBay, he said, they ran an in-depth analysis to see if it was economically or technologically feasible to collapse its big data workloads onto a single system. eBay has dozens of petabytes stored in Hadoop and possibly more within various Teradata appliances. The result: “We just couldn’t find a way in which we could justify collapsing everything we do into one system.”

Feature image courtesy of Shutterstock user vblinov.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• 2012: The Hadoop infrastructure market booms

The law calls for internet retailers with more than $1 million in annual revenue to collect sales taxes from out of state shoppers. State governments claim it will help them collect billions in unpaid revenue while brick-and-mortar retailers, who also support it, say it will level the playing field by forcing online competitors to collect tax.

Opponents of the law, which include libertarians and states like Oregon that have no sales tax, complain it will lead to regulatory burdens tied to collecting tax from numerous state and local governments. Supporters counter that the task will not be that onerous because the law would require states to provide merchants with free tax collection software.

For consumers, the law means paying more sales tax on online purchases. Right now, consumers typically pay only if the online merchant is located in their home state.

The bill will now go to the House where conservatives say they will oppose the bill; they may not succeed, however, as politicians from both parties have argued that the bill does not impose a new tax but instead helps collect taxes that are already owed. The Obama Administration supports the proposed law.

eBay, one of the law’s prime opponents, said in a statement that it will keep pushing for merchants who collect less than $10 million to be exempt.

To understand more about the law, see GigaOM’s primer on who’s for it and who’s against.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4

Dean Nelson, eBay’s vice president of Global Foundation Services, says the effort, called the Digital Service Efficiency report, “is the miles per gallon measure for technical infrastructure for eBay.” Essentially, the company has boiled its business down to a single currency — transactions (specifically URL requests) associated with users’ buying and selling on the site — and created a slew of metrics that measure how efficiently it delivers those transactions in terms of revenue, performance, cost and carbon footprint.

The project has been about 18 months in the making, Nelson told me during a recent phone call, and eBay was finally able to set a baseline measurement of its performance in 2012. Now that it knows what’s in place and how its infrastructure performs over the course of a year, the goal in 2013 is to cut its computing-related carbon usage and costs by 10 percent and increase performance in terms of transactions per kilowatt-hour by 10 percent.

In order to meet these goals, he said, every member of the technical team — from facilities managers to software engineers — has be striving toward them and also be cognizant of how turning their “knobs” will affect the other metrics eBay is measuring. “Think of it like a Rubik’s cube,” Nelson explained. “You can solve one side but screw up the rest of them.”

eBay plans to release quarterly updates on its progress along with its earnings reports, but employees will have access to down-to-the-second visibility into what’s going on. “It makes it personal for them,” Nelson said. “They can see what their efforts mean.”

52,075 servers doing a lot of work

Nelson offered some pretty compelling examples of how the Digital Service Efficiency project works in practice. If the goal is to decrease cost per transactions, data center engineers might try to minimize power usage at the facility level while server engineers might look to lower-power gear or better utilization on existing gear. They essentially reduce the denominator in that equation “and the net result is we should make more money from those transactions,” he said.

In one real-world instance, a software engineer tweaked some code that affected how much memory an application requires and the company was able to eliminate 400 servers. That cut energy usage by 1 megawatt and a $2 million savings in capital expense when the time would have come to refresh those servers.

eBay also has created a “list of fame” and a “list of shame” that highlight the 1,000 best- and worst-utilized servers within the company. “We have a hit list,” Nelson said, and it’s going to examine the bottom 20 percent to figure out why they’re as wasteful as they are.

However, he added, it’s important to remember on the server front that improving cost, performance and carbon usage doesn’t always mean buying lower-power gear. If eBay can improve the power density of its racks using technology such as liquid cooling — something its Project Mercury data center in Phoenix is pre-equipped for — it can handle more transactions on less gear. It already has some racks running at a sustained rate of 35 kilowatts and thinks it can push that up to 50 kilowatts, Nelson said.

Clean transactions with solar panels and Bloom boxes

On the carbon front, eBay has nothing but an open field in front of it thanks to some big clean-energy projects set to go live in 2013 in its new Salt Lake City, Utah, data center called Project Topaz. For starters, it’s using Bloom Energy boxes as the primary power source, which mean a slightly higher cost per transaction, but also a 13 percent reduction in carbon emissions and increased reliability (downtime costs eBay a lot of money).

Also, the company has finally cleared some regulatory hurdles to tie an on-site solar array back to the grid. Because of changes to a Utah law that eBay lobbied for, it’s about to start sourcing off-site clean energy for its data centers, as well.

“That is a corporate priority,” Nelson said. “We want to create the cleanest commerce engine on the freakin’ planet.”

Trying to change an industry

Of course, the Digital Service Efficiency methodology isn’t the only attempt by a major data center operator to show the world how efficient it is. Google publishes annual Power Utilization Efficiency (PUE) ratings for its data centers, and Facebook occasionally does as well. On Monday, Salesforce.com released a statement underscoring its commitment to sourcing renewable energy.

However, Nelson pointed out, what eBay is doing — and encouraging others to do — is more transparent in that it gives a lot more depth about operations, including the company’s server count. Even if companies don’t publish their results, tying operational efficiency to other business objectives should have a positive effect on the bottom line and the environment, regardless. Every company will have its own base currency, Nelson explained, and they’ll have to find their own metrics to measure and figure out what are the knobs that each part of the company can turn to meet goals.

“We all have the same challenges, the same things to solve for, but we have numerous ways to solve it,” Nelson said. …”[Their implementations] may change completely, but the point is the conversation is starting.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The economics of clean-data-center innovation
• How tomorrow’s mobile-centric data centers will look
• Migrating media applications to the private cloud: best practices for businesses

On Monday Max Schrems, the self-styled scourge of Facebook who has forced several changes in the way the social network treats people’s data in Europe, published a comparison table (PDF warning) that shows how entire passages have been copied, word-for-word, from lobbying documents into proposed amendments to the new regulation. The sources of these texts include Amazon and eBay, as well as the American Chamber of Commerce.

Amazon in particular seems to have had its proposed wording taken very seriously by some parliamentarians, who have copied wholesale the cloud provider’s proposals about reducing the responsibilities of non-EU cloud providers when hosting EU citizens’ data (the original proposals would, for example, force the cloud providers to remove data about certain individuals if those individuals want to be digitally “forgotten”).

But this is an unfinished project, and there’s still work to be done. What’s particularly entertaining is the way in which the privacy activists are tracking the similarities between the lobbying documents – which we know about largely through leaks – and the new amendments.

It’s being done through a website called LobbyPlag, in which the “Plag” is short for “plagiarism.” The idea was previously used for a site called GuttenPlag, which people used to collate examples of former German defence minister Karl-Theodor zu Guttenberg plagiarizing other people’s work in his doctoral thesis (zu Guttenberg subsequently resigned before being inexplicably made an EU digital freedom champion).

Here’s a screencast of how LobbyPlag works, courtesy of OpenDataCity, the German data journalism outfit that’s supplying the technology:

I think it’s fair to say that leaked documents plus crowdsourced analysis can make for a pretty potent combination. Whether that mix will have any effect against well-funded lobbying, though, is another matter. Don’t be fooled into thinking this is a simple U.S.-versus-Europe thing, by the way. As recent research has shown, actual consumers on both sides of the pond have similar expectations about data privacy — both sides want more of it.

“Certain companies will always be willing and able to throw millions of dollars behind lobbying efforts to ensure that new legislation doesn’t interfere with their business models — particularly if those models are dependent on invading people’s rights to privacy and data protection,” Anna Fielder, trustee of UK-based Privacy International, said in a statement.

“We would hope that MEPs are taking all sides of the argument into account when making law, not just the richest and most powerful corporate interests.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• Sector RoadMap: Content personalization in 2013

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The move is unusual in that there’s few power companies and utilities buying fuel cell power in the U.S. at this point. Fuel cell power tends to be more expensive than centralized power from power plants that use low cost fossil fuels. But the deal between Dominion and the utility is a power purchase agreement over 15 years, so Connecticut Light and Power will buy the power over a decade and a half at a fixed low rate. The fuel cells in the project will also use natural gas as the fuel, and natural gas prices are really low right now.

Perhaps the combo of new financing models and cheap natural gas will lead to an era of more and more fuel cell getting put into the ground. Fuel cell startup Bloom Energy also sells fuel cell energy using power purchase agreements over long periods of time.

The fuel cell farm in Connecticut will be built in the city of Bridgeport on 1.7 acres of land and the first fuel cells will start to be installed in the Summer of 2013. The project is supposed to be operational by the end of 2013. FuelCell Energy says it will generate $125 million with the deal, including $56 million from selling the hardware and $69 million in services, and the project will also contribute to the state’s renewable energy mandates, and provide 160 jobs.

FuelCell Energy also sold a 11.2 MW fuel cell project project to Korean power producer Korean Electric Power Corporation (KEPCO) in Daegu City, South Korea. Bloom Energy has a planned 30 MW fuel cell deal with Delmarva Power.

Internet companies have been as active as utilities, when it comes to using fuel cells to power their data centers. Apple is building a huge fuel cell farm in North Carolina (of which it just doubled the size), eBay is building one, too, and Microsoft is also experimenting with the technology.

Images courtesy of FuelCell Energy.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Q4 Wrap-up: SOPA and the future of digital content
• NewNet Q4: Platform mania and social commerce shakeout

Another change the company has made, despite being more significant, has received relatively little attention. It will add a clause that says it can now share your information with its affiliates. This underscores an important trend: Facebook is now one of several companies that shares your information among its various different services. As the range of services offered by consumer internet heavyweights like Facebook and Google continues to expand, this means that your personal data will end up being used by in ways you could never predict.

The trend towards internal data sharing

• Google’s unified terms of service (introduced on March 1) allows it to combine your data across products like Gmail, Google+ and Google Docs, as well as YouTube, Picasa and scores of other Google properties.
• Microsoft updated its services agreement effective Oct. 19 to allow it to use your content to “provide, protect and improve” all “Microsoft products and services.” The agreement had previously allowed the company to use your data only to provide the particular service in question.
• eBay and PayPal can share your data among all “members of the eBay Inc. corporate family,” which also includes Shopping.com.

Facebook’s ability to use customer data benefits users by allowing for innovation. Its existing Data Use Policy says that you allow the company develop “innovative features and services” by using “the information we receive about you in new ways.” But when its proposed revisions go into effect later this month, its ability to share users’ data will increase further. The following new paragraph will be added:

We may share information we receive with businesses that are legally part of the same group of companies that Facebook is part of, or that become part of that group (often these companies are called affiliates). Likewise, our affiliates may share information with us as well. We and our affiliates may use shared information to help provide, understand, and improve our services and their own services.

The Facebook affiliate that first comes to mind is Instagram. And one of the most immediate applications of this new provision could be that Facebook starts to serve ads on Instagram. But Instagram is just one of many Facebook affiliates. It has numerous subsidiaries, including one it created earlier this year to handle its payments business. And its publicly stated expansion plans mean that it will likely be acquiring other companies, forming new business units, and entering new markets in the near future.

How worried should we be about internal data sharing?

The ability of internet companies that operate in multiple verticals to share your data among their different services has potentially far-reaching consequences. Microsoft could use your Outlook.com email messages or your online Office documents to push targeted ads to you on Bing. eBay could use your shopping history to recommend financial services on PayPal. Facebook could use data you expose by taking Instagram snaps to customize your experience on its web or mobile apps.

Not all of these scenarios may be objectionable to everyone– and in fact, some people may find none of them objectionable. And increased internal sharing can be a good thing not only for the companies themselves but also for users. Being able to tightly integrate our experience across all Google products, for example, offers an efficiency as well as functionality boon. For the service providers, being able to increase the effectiveness of targeted advertising means that they can improve their bottom line (and ultimately invest in developing new services and improving existing ones).

But the point remains that we are increasingly allowing personal data to be used in ways that are not obviously related to the action we initially took in order to generate that data. To an ordinary Instagram user, it is not obvious that using the Instagram app will have any impact at all on their Facebook experience. When we think about data privacy, we normally think about a company giving or selling our information to someone else. But a single company using our information in ways that we did not anticipate raises privacy concerns that are just as significant.

What should we do about it?

Internal data sharing of some kind is for all practical purposes an inevitability. This need not be a bad thing if users are sufficiently well informed about how their data is being used and if they can opt-out of uses that they find objectionable. I propose three principles to guide us going forward:

• Better disclosure It should not be enough for a company to say that data can be shared across its services and among its affiliates. We should be given much more specific guidance. Whether disclosure is necessary should depend on how unexpected the sharing would be to an intended user. Some forms of internal sharing are obvious and need not be disclosed. We should assume that our data will be transferred to a company’s customer service department in order to handle our customer service inquiries. But if sharing is between unrelated products or services, users should be notified. This is especially the case if they are between separately branded services that do not reveal their common corporate affiliation (e.g. to an ordinary user, Instagram is not obviously owned by Facebook).
• More granular controls for users Users should be able to opt-out of internal sharing as a whole and also be given control to turn on and off particular kinds of sharing.
• Sensible defaults Despite the obvious benefit they offer, in practice many people simply won’t take the time to navigate through granular privacy controls. As a result, it is important for default settings to offer users a reasonable amount of protection.

As an increasing amount of the data we expose online is handled by a small number of companies that operate across multiple verticals, sharing between companies (external data sharing) could ultimately be less of a privacy concern than sharing within companies (internal data sharing). We should be concerned not only about who controls our data but also what they can do with it.

Andrew Nicol is an entrepreneur and attorney based in New York City. He runs Clickwrapped, a not-for-profit service that rates leading consumer technology companies according to how well they respect their users’ rights. Follow him on Twitter @aknicol.

Photo courtesy of Muellek Josef/Shutterstock.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Deepfield, which offers services to build, manage and optimize network infrastructure, studied online shopping infrastructure by sampling internet backbone traffic across a “large cross section of North America and multiple collaborating infrastructure and internet providers.” The goal: To estimate how many users hit these sites daily and determine market share of the sites based on those numbers.

According to Deepfield’s results:

“What is truly impressive is how much larger Amazon shopping is compared to any other online site. Amazon is almost double the next largest shopping competitor, eBay, which enjoys 8.8% of daily Internet users.”

With Amazon, Deepfield includes a raft of smaller Amazon-owned sub-sites like MyHabit, although Zappos, which Amazon bought two years ago, was counted separately.

Deepfield included e-commerce hosting sites — including Shopify — to show how vibrant this market is. Shopify, which offers a web storefront that is used by more than 30,000 sites, garnered 5.4 percent market share among daily users. Quidsi, the company behind diapers.com, soap.com and other sites, also got a healthy 3.1 percent of daily visitors, and was also bought by Amazon in 2010.

Update: (12:45 p.m. PST) The results are summarized in the updated Deepfield chart below:

Feature photo courtesy of Flickr user Shlomi Fish

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining work in the digital age: an analysis by GigaOM Pro
• Sector RoadMap: Content personalization in 2013
• How the mega data center is changing the hardware and data center markets