That’s why as Hadoop keeps thundering toward its destination as the de facto data platform for next-generation applications, companies such as Cloudera and Hortonworks that are making a killing off it might want to stop and thank the guys from Powerset for building HBase. Because the database — a columnar Google BigTable clone that runs on top of the Hadoop Distributed File System — is so fast and scalable, it’s helping Hadoop find a home in companies and with applications that HDFS and MapReduce alone might not have been able to penetrate so easily.

The latest HBase user I’ve come across is Gravity, the interest graph company that powers content recommendations for some of the biggest publishers on the web.

From big MySQL at MySpace to big data with HBase

Its co-founders were all senior executives at MySpace, including Gravity CTO Jim Benedetto, who was SVP of technology for the social networking pioneer. He was actually MySpace’s first architect and helped build platform’s MySQL database. Although MySpace never reached Facebook’s scale, it did have 150 millions users at its peak, all able to store unlimited numbers of wall posts, messages and photos. Benedetto eventually oversaw a 600-instance cluster that required about 30 database adminstrators to keep it up and running.

Benedetto (center) at Structure: Data 2012. (c) Pinar Ozger

So naturally, when it came time to build out the Gravity architecture, Benedetto opted for the MySQL he knew so well. Until about three years ago, he told me recently, that database held about 95 percent of the company’s data. At some point, though, Benedetto and his team realized they were spending way too much time keeping their MySQL environment up insteading of building new things, so it was time for a change.

It ultimately opted for HBase, but the decision wasn’t easy. “For us,” Benedetto said, “our data and algorithms are our company,” so making the move from a relational database to a column-based database that can serve MapReduce jobs was nerve-racking. After all, he explained, “You never want to migrate your data … and if you have to, you never want to migrate it more than once.” In fact, he added, “you’re not going back.”

But Benedetto says the move to HBase as Gravity’s primary data store has been “life-saving,” and it’s arguably a more important component of the company’s infrastructure than is Hadoop MapReduce. HBase handles the company’s real-time recommendation algorithms, and it does it across the entire Gravity platform rather than on a site-by-site basis. And although it’s not banking-grade when it comes to the consistency of transactions, Benedetto says it’s about 99.95 percent consistent in real time. Later on, batch MapReduce jobs swoop in and pick up whatever HBase dropped earlier, and process it all against the company’s graph algorithms.

An example of an interest graph from Gravity,

Scalable for sure, and getting easier to use

And although it took some serious engineering effort to get HBase operational when Gravity began working with it three years ago, Benedetto thinks HBase is getting to the point (as is rival NoSQL database Cassandra, he acknowledged) where one could safely call it “enterprise-ready.” Right now, he noted, “You’re not gonna to see HBase in a company that just buys Oracle because Oracle is the name and Oracle has been around for 20 years,” but for web startups that hope to reach a certain scale and even for existing companies that are running into the MySQL wall, he sees a shift occurring.

“The web farm is the easiest part of your infrastructure to scale because all it does is cost more money,” Benedetto explained. Databases, on the other hand, require a lot of thinking about how to migrate data, shard the database and otherwise make a piece of software likely designed for a handful of servers, max, spread across dozens or hundreds. HBase really eases the scaling process, as well as the subsequent management, he said. Now, Gravity’s 100-node HBase cluster has only two operations engineers dedicated to it.

Indeed, there are startups trying to capitalize on HBase by using it to power SQL and even MongoDB-compliant databases that can scale beyond what most relational databases can do.

Aside from scale HBase might soon start catching on because of the work companies like Gravity have been doing to make it more user-friendly. It might scale easily, but, as Benedetto noted, it’s not always easy to get started with — especially without some deep understanding of the intricacies of the underlying HDFS infrastructure. Last year, eBay VP of Experience, Search and Platforms Hugh Williams told me that although HBase is one of the big data tools the company is most excited about, it’s also the area where he’d like to see the most improvement.

To help alleviate some of the learning curve, Gravity has developed an open-source tool called HPaste that lets developers access data and run jobs on HBase data using Scala rather than the more-bloated Java programming language on which Hadoop and HBase are built. One of the biggest benefits of HPaste, Benedetto said, is that it lets new HBase developers see the data in a way that makes sense to them: HBase stores everything in byte arrays, he explained, and “when a human tries to read a byte array, it looks like ancient hieroglyphics.”

The Kiji architecture

Elsewhere, a startup called WibiData has created an open-source framework called Kiji that aims to provide a collection of high-level APIs that should make it easier to store different data types in and develop applications on HBase. The company envisions Kiji being to HBase what the Spring Framework has become to Java over the course of the past decade.

Hadoop’s weapon for the mainstream?

But user experience aside, a lot of companies already invested in Hadoop — aside from expert users such as Facebook — are starting to see the promise of HBase and are incorporating it into their architectures.

WibiData co-founder Christophe Bisciglia, who also co-founded Hadoop pioneer Cloudera in 2008, gave me his take on the state of HBase while discussing its role in the future of Hadoop earlier this year. ”If you talk to anyone from Cloudera or any of the platform vendors, I think they will tell you that a large percentage of their customers use HBase. It’s something that I only expect to see increasing,”  he explained. “… HBase is gonna be what takes Hadoop from an ETL and BI platform into a real-time application platform.”

The Cloudera Hadoop stack (Gravity uses Cloudera’s distro).

Benedetto appears to agree. He considers Hadoop as a whole incredibly important, almost on par with what Amazon Web Services did for computing resources, because it lets startups use commercial-grade open source software to do data storage and processing that previously was only available to massive web companies. “More and more … the shining star in that suite is HBase,” he said. “If I were Oracle, I’d be scared.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How to use big data to make better business decisions
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Social media in Q1: commerce and discovery dominated
• 12 tech leaders’ resolutions for 2012

Gravity’s recommendation system is based on its interest graph technology, which we detailed last year. Here’s how I described it then:

[T]he gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

As new content streams into Gravity’s system, it’s analyzed and categorized in real time, then presented to users accordingly based on their interests and behavioral history.

How Gravity’s platform works

Graph processing and graph databases — which store and analyze data based on their relationship to one another — are critical to our onlines lives, powering everything from online recommendations to social search to knowledge discovery. Graph technologies are also the focal point of some impressive life sciences work from companies such as Syapse and Ayasdi, which will be presenting at Structure: Data in New York next month.

But publishers struggling to stand out on a noisy web might have the most to gain from graphs and personalization, generally. At our PaidContent Live conference (April 17 in New York), executives from Prismatic, Zite and Bluefin Labs will take the stage to talk about the importance of personalization for helping consumers filter through the deluge of content online so they can find what they really want. It’s arguable that the trick to keeping readers happy is knowing what they want to read — possibly better than they do themselves.

According to Gravity, its platform currently “delivers more than 25 million personalized content recommendations per day to more than 200 million users. Beta partners have reported click through rates two to three times above previous levels, return visitation increases of 300 percent and session length increases up to 40 percent.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Sector RoadMap: Content personalization in 2013
• How big data analytics drives competitive advantage

As I explained in March when, Gravity has built a semantic-analysis engine that tries to gauge a site visitor’s interest by looking at more than the articles that person reads. Thanks to an expansive database of topics and a hybrid man-machine machine learning system that takes into account behavior as well as content, Gravity can determine other topics that might be of interest even if those connections aren’t visible to the naked eye. The result of this analysis is called an interest graph, which is like a social graph only that’s concerned with interests rather than people.

Currently, Gravity claims its total body of graph data exceeds 18 million megabytes, or 18 terabytes. The company says the new money will help it expand operations in the United States and even deploy its own content-marketing platform.

Of course, interest graphs are useful for more than just automatically presenting visitors with the news content they’re interested in. Gravity also has a product for advertisers to better target potential customers, and an analytics service so publishers can get in-depth visualizations of who’s reading their content and what content works better than other content.

However, the obvious elephant in the room when talking about interest graphs is privacy and how to collect and analyze user data without crossing any ethical guidelines. This will become even more of an issue as web platforms try to share data across services in order to create a more unified browsing experience in which interest graphs follow users around the web to inform personalization algorithms at every step. And as we’ll discuss later this month at our Structure: Europe conference in Amsterdam, some governments take user privacy much more seriously than others, which can make businesses based on that data a little trickier to operate.

Here’s Gravity Co-Founder and CTO Jim Benedetto, along with privacy attorney Ashlie Beringer, discussing the issue with me at our Structure: Data conference last March.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Content personalization in 2013
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

People may be very interested in Britney Spears pictures but they don’t necessarily want to share celeb posts on their Facebook news feed, Kapur said. To give users a truly personalized online experience, he says, “you need a personal identity attached to the user that’s not about sharing or social.”

Moderator Michael Wolf, VP of GigaOM Pro, mentioned “My Yahoo, maybe one of the last good things Yahoo did.” It allowed users to create personalized homepages, but Valla Vakili of Small Demons says users had to do too much “setup work” to personalize their pages. “The hardcore base did, but a lot of people didn’t,” he said. So Kapur says it’s key to “dynamically build that kind of experience.”

Jonah Peretti, founder and CEO of Buzzfeed, says “social bias” can be a good things sometimes, especially in the transition from a search-centric Web to a social-centric Web. “Search was ‘nude ScarJo pics,’” he said. “People aren’t going to say on Facebook, ‘I heard there were some nude photos of Scarlett Johansson and I have some time this weekend, can somebody share those?” Search “doesn’t appeal to our better selves.” And people search for topics that are “boring but useful.” But “on Facebook you’re much more likely to post something like ‘join me to help the people of Japan after the tsunami.’”

“I like social bias,” he said, “because I think it leads to content that you actually like to talk about around the dinner table, or with other people in your life. That’s the stuff that’s most interesting to talk about.”

Check out the rest of our coverage of paidContent 2012. Full archived video on livestream (registration required).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Content personalization in 2013
• A near-term outlook for the mobile app marketplace
• Connected world: the consumer technology revolution

Much as social graphs are maps of our social media connections that follow us across the web, interest graphs are maps of our interests. Some companies want them to follow us across the web, too, meaning that wherever we go, there we are. There’ll be no more need to search through news sites for the stories we want, or shopping sites for the products we want, because the site will know as soon as we hit its system who we are and what we like.

Whether you’re fascinated or appalled by the idea of interest graphs, here’s a taste of how they might work.

1. Figure out what you like

I recently discussed the idea of interest graphs with Gravity CTO Jim Benedetto, who described how his company determines visitors’ interests so its content-industry customers can deliver personalized experiences. Gravity does that by tracking a user’s activity across the Gravity network of sites to try and get a true sense of a user’s interests so it can feature that content automatically whenever that user visits a Gravity-powered site. An example on the Gravity Labs site explains the process of building a user graph in some detail.

Technologically, Gravity accomplishes this via a big-data engine that sits above expansive data sets such as Freebase and DBpedia and helps its system determine what that user meant when he clicked on a particular article or made a certain comment; what his actual interest is. Describing the system last week in a post about software patents, I called it “a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.”

Actually, though, Benedetto said that Twitter and Facebook aren’t as good for determining people’s actual interests (and Gravity uses them primarily to figure out what’s generally popular) as are clicks and comments on other sites. He said a high percentage of tweets are from people sharing information that’s important to them professionally, while Facebook is limited largely to sharing information among a collection of people with whom users are often quite close. In either case, he said, “There’s a lot of posturing [to impress others] … that results in incomplete data.”

At MySpace, where Benedetto was senior vice president of technology, he said his team often found people would join groups based more on their feeling toward the person inviting them than on an actual interest in the issue. We’ll sign up to save the whales not because we care so much about whales but because we don’t want to disappoint a friend.

2. Take it with you

“One day,” Benedetto told me, “people will be able to apply their interest graphs on every site across the Internet.”

It’s a compelling vision. Much as our social graphs influence our search results when we use Bing or Google, our interest graphs will influence what we see wherever we go on the web. If you like skiing, expect to see deals on skiing gear when you’re perusing ecommerce sites, or stories about winter sports on news sites. And expect to see this without having to log in, as it will be based on everything you’ve done across the entire web.

If you think about it, we all probably have several disparate interest graphs hanging around already, even if they’re not as advanced as what Gravity is building. The uproar around Google’s personalized search? Based on an interest graph. Amazon’s recommendation engine? Interest graph. Yahoo News? Interest graph.

The real trick will be in figuring how to connect these disparate graphs to build a single, portable account of who we are and what we’re interested in. Presumably, methods such as OAuth, which enable API-based data sharing among services, will have to play a role. Login once when you get online, and your interest graph is ready to go, ready to make the web a place that serves you rather than a place you must learn to navigate.

3. Rein it in

Of course, for every person in love with the idea of a personalized web-surfing experience, there’s probably one person who considers it a privacy nightmare. There likely are others who want an objective experience not tainted by any system’s assumption about what they want to see. For these reasons, the personalized web will almost certainly have to be an opt-in experience.

Privacy could be a particularly difficult problem to solve. Discussing the issue of the personalized web in a November report for GigaOM Pro (subscription required), I explained the necessity of empowering consumers to decide what data is a part of their digital fingerprint, or interest graph. Most people probably don’t want their affinity for pornography affecting the news they’re shown, or to be reminded of an impending foreclosure everywhere they go just because they’ve done a lot of research on the subject. Site permissions and “do not track” buttons might have to get increasingly granular, and increasingy easy to understand, to ensure the personalized web doesn’t become an enemy.

A true picture of our interests is one thing, but we’ll always need boundaries, especially as our lives get even more connected. Do you want ads for Valtrex popping up while watching Google TV with your new girlfriend? Do you want her popping on your computer to check the news and see wedding planning stories front and center, ruining your romantic proposal? Just as we can have our own rooms in our homes, or can drive across town and shop anonymously, we need pockets of privacy in the personalized web.

If you find the possibilities and pros and cons of a personalized web as fascinating as I do, be sure to get to our Structure:Data conference next week in New York, or at least watch the livestream. We’ll be talking all about the latest techniques for analyzing web and mobile data, and I’ll be speaking with Gravity’s Benedetto and Gibson Dunn partner Ashlie Berlinger about the line between what’s possible with analytics and what’s ethical, or even legal.

Interest graph image courtesy of Gravity; sherpa image courtesy of Ian S.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Sector RoadMap: Content personalization in 2013
• Social media in Q1: commerce and discovery dominated

Earlier this week, Gravity, a Los Angeles-based startup, launched a new service called Twinterest and outlined its intent to help personalize the web through what it calls an “interest graph.” This isn’t the first time the start-up — which has garnered $10 million in funding from the likes of Redpoint Capital and August Capital — has sought the limelight by making bold (and woolly) claims.

The company, which was started by Amit Kapur, Steve Pearman and Jim Benedetto — all former Myspace executives — came out of stealth in December 2009. Gravity initially wanted to reinvent the concept of conversations then data-mine these conversations to build interest analytics, and eventually build a business against these analytics. Suffice to say, I wasn’t a fan of the service, and took a dim view of its prospects. When it launched earlier this year in beta, the service was indeed a letdown.

Since then, the company has come up with a new strategy, and is now using Twitter feeds to build the Interest Graph, which it says it will then use to help personalize the web. Sometime in the future, Gravity CEO and co-founder Amit Kapur says, he wants to give publishers the ability to personalize content for each one of their readers.

One of the key building blocks for this “interest graph” is a new offering from Gravity called Twinterest. It’s a service that taps into the Twitter fire hose and hopes to map my interests and essentially connect them to some of the folks I might know. I tried it out and got some surprising results. For instance, the service says I have 476 interests including Country Music, Bernie Madoff, Appalachian State University, Debarge, Soviet Union and Anakin Skywalker.

Those “interests” are just straight up wrong. The rest of my “interests” remind me of a phrase we used to use often when growing up: Looking London, Going Tokyo.  What’s even funnier is that the service suggests that I should friend Rapleaf on Twitter (ironic, considering some of my writings about that San Francisco-based company). Gravity wants us to help fine-tune this “interest” data, so it can personalize the web in the future, but the Twinterest results so far make quite an un-interesting graph.

The inaccuracies in the interests displayed by Twinterest are symptomatic of some of the problems associated with natural-language processing services, which try to extrapolate my interest level in topics from proper nouns mentioned in tweets. I remember tweeting that I was listening to Debarge on the radio and feeling nostalgic. That doesn’t mean I’m interested in them.

Many services that propose to make sense of the Twitter feed and draw inferences have to deal with a whole can of worms. For starters, any service that proposes to build an interest graph or some sort of ranking, needs to do the following:

• Analyze the content shared by a user.
• Analyze the content of a user’s tweets.
• Analyze what others a user follows as a potential signal of interest.
• Analyze who follows a user to get a sense of authority of the user on that interest.

Gravity (and other services similar to them) have to figure our ways to do the aforementioned four things at scale, then build a taste graph for every single Twitter user. That’s not easy, nor cheap. One assumes Gravity has been able to do that — and have put some of their venture dollars to good use. But then as more data points are added to the mix, say Facebook, the complexity (and costs) of Gravity’s Interest Graph is only going to go up.

I think the challenge with services that use Twitter to draw inferences about me is that they only have an incomplete picture of me. My Twitter identity is very tech-centric. It ignores some of my real-world interests, and frankly, I don’t care to share a few things with the rest of the planet. In other words, Twitter can’t really help mirror the real me.

That said, it’s simple enough to build a content-discovery system based on the interest graph, and predictably, that’s why the company is building a news discovery service, The Orbit. I don’t think it’s really a very valuable service – i.e., it’s a terrible business idea. The idea of an interest graph is something that was initially championed by Hunch, which instead of doing natural-language processing, decided to start with a combination of machine learning and statistical learning.

Eventually someone will figure out a way to mine the web and build a personalized web experience. Gravity seems pretty far from it. This is something the guys who own the data – Facebook, Twitter, Google – will likely do.

Related content from GigaOM Pro (sub req’d):

• Could Privacy Be Facebook’s Waterloo?
• Facebook Tries to Navigate the Privacy Storm
• Lessons in Smart Grid Privacy From Facebook and Google

Post and thumbnail images courtesy of Flickr user Luis Markovic

First of all, Gravity is organized around the gimmick of “worlds” (broad topics) that you can “orbit” (follow), while being hosted by a cartoon dinosaur named Amir. There’s some organization into categories but most everything is a chronological thread. You can navigate using a live-updated thread of things you’ve subscribed to, or use an index or search to find new topics. Users, who don’t have to use their real names, are rewarded for participation with badges.

So far, I haven’t found many deep conversations or enough breadth of topics to fit the topics for which I’ve searched. Many threads consist of people posting personal stories or pictures; one entertaining one was “awkward celebrity encounters.” The real-time alerts about conversations and people you’re following seem to be a big booster of discussion. I posted in a thread about the Vietnamese noodle soup pho, and got three replies within a few minutes. Nothing profound, but at least people appreciated my contribution.

Gravity for me is an interesting contrast to Quora, the Q&A site from former Facebook employees, which is also set up around common interests and discussion threads. That site, in keeping with its pedigree, requires real-name participation (though it allows anonymity on a per-contribution basis) and seeks a high level of discourse, with users hastily correcting each other for contributions that aren’t seen as productive. Quora’s founders say their ultimate aim is to create “canonical consensus” on a wide variety of topics.

It’s all very serious and schoolmarmish, but I really like it, because users hold themselves to a high standard of participation. So far, I prefer Quora to Gravity, mostly because the small private-beta community consists of people with good knowledge of topics I’m really interested in — like tech startups — who put time and thought into crafting contributions. For many of them, it’s a new sort of blogging platform. The reason I go back to Quora almost every day is because the conversations are great. I’m not sure that’s what the company intended when it set out to build a Q&A platform, but it’s a great by-product.

Gravity, on the other hand, could be great, but only if it grows to the point that it features well-threaded, organized and searchable conversations on an extremely wide variety of topics. And it strikes me that even more useful would be the ability to mine the web’s existing treasure troves of forums, communities and groups — all of which have terrible interfaces but contain tons of great knowledge. I know it might be more appealing and manageable to create a new “interest graph” platform from scratch, but people have been interested in stuff online for a long time.

The company, which launched a beta version of its service, is essentially trying to re-invent the concept of conversations (aka message boards and e-groups) using a brand-new approach and then data mine these conversations for interest analytics and essentially sell advertising or market against these interest pockets. Gravity describes itself as a “conversation engine.”

And while I have not played around with the service, and looking at it from a very surface level, one can see that it has its work cut out for it. There are some questions about the potential and viability of this startup.

To be clear, I have no comments on Gravity’s technology — the company seem to have some kind of a secret sauce which it hasn’t really talked about. No point commenting about its user experience — the company is still hoarding their beta invite. My questions are simply confined to the obvious challenges it faces.

Now, no one disagrees with the logic behind the company: message boards and e-groups simply suck and it’s hard to monetize conversations trapped inside them. If one could solve this quandary, they could somehow make money. Gravity claims to have done this and has topped it off with some spiffy-looking graphical analytical tools. And that’s the end of the good news.

Typically I get very excited about companies (and startups) that are doing interesting things with large data sets, but in this case I’m skeptical. When I read the post, what I saw was a lot of spin garnished with a lot of fancy words — many of them, for example, Interest Graph. “Hate when a company coins a new term (Interest Graph) 2 describe existing concept – ah marketing” is how someone put it on Twitter. Gravity’s spiel is sentiment analysis and perhaps that’s why it was able to get a rumored $10 million, reportedly from the likes of Redpoint Ventures, an erstwhile investor in MySpace.

Now if you buy the spin, you still have to keep in mind that mining unstructured data for near real-time sentiment analysis — essentially what Gravity is claiming to do — is not an easy task. Let’s just assume that it does get the technology part right — and why not? Benedetto is totally legit and well known in big data circles. The question is, how are they going to get this service deployed and get massive traction?

Gravity executives tell Mike Arrington that the company is going to “allow people to create conversation around topics” and “the service will be available on the Gravity web site as well as via widgets and an API.” In other words, they are going to try and build a brand-new consumer brand around conversations.

At the same time, the company has to scale up its operations to a level where it has a large amount of data sets to mine and thus reveal enough “interest” spots for them to become useful and be used for diverse purposes such as advertising. It’s an arduous task. Ask Facebook, which is still struggling to build an “engagement-” and “sentiment-based” advertising engine despite having massive data sets, including personal information. Mining data is a complex computing problem that needs big computing resources, but that’s a topic for another day.

From the way I see it, building a rocket ship in one’s basement is perhaps easier. The conversations have moved away from message boards and groups to more real-time mediums such as Twitter and Facebook. It would be difficult for Gravtiy to change people’s behavior.

Sooner or later the company will send me a beta invite, but until then I’m relegating it to a pile labeled “hype,” for you will hear a lot about Gravity. But as philosopher (and humorist) Dave Barry once said, “Gravity is a contributing factor in nearly 73 percent of all accidents involving falling objects.”

Photo by SolidState via Flickr.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.