Our mobile devices are amazing at capturing real-world data — location, temperature, movement, sound — that just goes to waste if we don’t put it to use. It’s easy enough to get a personalized experience on the web, but these types of data might make it possible to get one in traditionally more-static places such as retail and radio as well. At the least, perhaps we can expect content, price tags and experiences that cater more to our actual tastes than those of station programmers and a fashion designer’s idea of what people should be willing to pay.

Location is the key to everything

Retailers already have a pretty good sense of what people are buying and even how they’re moving through stores, but they don’t really know where customers are going once they leave. This knowledge could be very useful, however: If you want to improve your store or figure out how to market your company, knowing what else your customers are up to could go a long way. This type of data is starting to become available thanks in part to a Seattle-based startup called Placed.

We’ve been covering Placed for about a year, since it launched its first product targeting developers interested in learning where users were accessing their applications and mobile sites. The company has since expanded its operations to include a Panels service that lets the company track around the clock, on behalf of paying businesses, the physical location of customers who have downloaded the app (usually in exchange for a small monetary reward). It also has its own Panels app, unaffiliated with commercial customers, that allows Placed to provide market data on the physical movements of some 70,000 consumers.

This week, the company released a report highlighting some national findings from the first quarter, including, for example, what departments stores are most popular with what demographics, what business categories experienced the most increases in traffic, and what businesses have the highest and lowest affinities (i.e., people who visit one also visit, or don’t visit, the other). If you’re willing to pay, Placed will tell you pretty much anything you want to know, founder and CEO David Shim told me, broken down by geographic region, business type, demographic, you name it.

Shim noted a couple of actual users and potential users that I think highlight why this type of data is so valuable. One is a high-end retail business that found out that while female millenials enter its stores a lot, they don’t buy a lot. Rather, the stores they visit next are usually discount retailers such as Burlington Coat Factory and Ross. The suggestion is clear: These shoppers want to see what’s hot and then buy a reasonable facsimile at a lower price.

He also noted that some Las Vegas casinos are interested in running their own Placed panels to figure out what restaurants their guests are eating at once they leave the casino grounds. Now, if casinos can figure out where else on the Strip people are spending their money, they can make better choices when it comes time to swap out their own restaurants and shops.

In both cases, it’s possible the answer to the question of how to get more of these customers’ money is to drop prices. If a 10 percent price reduction leads to a 14 percent increase in sales, that’s a win-win situation.

Rethinking radio

Location data becomes even more valuable when combined with other data, though, such as sound. Consider the implications of knowing not just what radio stations people are hearing — which is essentially how the Arbitron ratings work — but what songs they’re actually listening to. Just because you hear the Latino station for an hour at the taco shop during lunchtime or the top 40 station at the gym, that doesn’t mean you’re listening to them or like listening to them.

But the songs you choose to listen to in your car, for example, probably tell a lot about what you actually like. And the technology exists to determine that. Last month, I wrote about how Gracenote is able to use the internal microphones on tablets and smartphones to recognize the songs playing on people’s televisions or stereos. It can also detect reactions such as cheering or booing, and likely whether someone turns up the volume.

Arbitron’s Portable People Meter

Now, all of a sudden, one can envision a world in which programming managers at radio stations can figure out on a song-by-song (or artist-by-artist) basis what people are actually listening to, and when and where they’re listening. If all it involves is someone downloading an app, they can presumably do it at a larger scale than requiring people to wear special additional sensors or fill out a diary. Broadcast radio can never be as personalized as something like Pandora, but it could start sounding a lot more like what listeners would choose if left to their own devices.

Digital radio could get downright great, even better than what Pandora can currently offer. I might never add Disney theme songs or the Sesame Street favorites to my preferences list, but if that’s all I listen to when I’m in my car between 4:30 p.m. and 5 p.m. — and it is — maybe a service could hook me up with some new songs every day. If I’ve turned up the volume on a Taylor Swift song three times this week while I was at home, maybe I actually like it and want to hear more even if I won’t admit it.

Not just data, but good data

As great as all this might sound (it does to me), it’s the advent of big data that makes it all possible. Placed’s analytics are so accurate because it has special algorithms to determine where a person actually is — even if there are numerous options within a small area — and its models are constantly being trained. Shim said his company gets 15,000 responses a day to surveys asking Panels users whether it had them at the right location, and it has already validated 3.5 million of the the 13 billion locations in its database.

Gracenote, for its part, has audio and metadata for millions of songs that it keeps in memory so it can access them in a hurry for the sake of real-time recognition. It can group music into dozens of categories based on genre, artist, geography or even just how the songs sound. It wants to build an in-car system that can change songs based on driving conditions fed to the stereo from the car itself.

I acknowledge this all sounds a little creepy, but, ironically, it also sounds like the beginning of the end for some concerns over privacy. Heck, Shim said, about 500,000 people have already downloaded the Placed Panels app.

Really, it all comes down to value. If handing over a little bit of data actually provides value in return — in the form something better than just targeted ads — it appears people will be willing to do so. People tell Amazon about their purchases, let Google Now access their email and tell Placed which store they’re at out of five possibilities because they think they’re getting a worthwhile service in exchange.

The data-collection genie is already out of the bottle. Now it’s just a matter of making it work for us instead of at our expense.

Feature image courtesy of Shutterstock user Vadim Georgiev.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

At paidContent Live 2013 Wednesday, Prismatic co-founder and CTO Aria Haghighi said those in the business of helping readers discover new content can’t even agree on a definition yet. “We don’t all agree on what personalization means. I agree (that it means pointing you to) stuff you care about … some people think it’s only getting signal from noise,” he said. “I think we’re really early. I don’t think we’ve matured to point where we (the industry) have philosophical differences yet.”

Mark Johnson, who is CEO of competing product Zite, agreed that there’s so much more to do. “We’re good at finding stuff, but we can be so much better. We haven’t had that Google moment yet that’s, ‘Aha! That’s what personalization is.’” He compared it to when AltaVista users turned to early Google search and said, “So that’s what search is!”

That point of personalizing reading content for users is also the challenge: everyone’s feed will always be different because no one likes the same stuff. Improving that capability will depend on big factors that haven’t quite been figured out yet either: the social aspect and helping people understand why content is recommended to them.

Both Prismatic and Zite agree that helping users understand why you’re showing them content is key: “Even if you do world’s best job recommending an article or content to people, if they don’t have an explanation of why, people feel uneasy,” said Haghighi. “You have surface parts of the system to give information to people. That’s where user response intersects with AI.”

Part of recommendation is social — what your friends or people with your same interests liked — but no one has done a good job of that in their industry yet, Johnson thinks. Social “sharing is the tip of the iceberg,” he said. “There’s something deeper you can do in these applications that no one has nailed yet.”

Check out the rest of our paidContent Live 2013 coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• How social discovery is transforming entertainment
• Social media in Q1: commerce and discovery dominated

I ask because Facebook’s newest attempt to boost ad revenues — it’s “partner categories” program — seems too dunderheaded a move to be real. A company that’s regularly lambasted and has even been sanctioned by the government for privacy indiscretions is now going to let advertisers place ads based on what users have purchased offline, or at least off of Facebook? And, worse yet, based on public records such as the type of car someone drives?

This type of advertising happens all the time, of course, but I just can’t believe it’s for real from Facebook. Is it oblivious to public opinion about its privacy record (a recent study, for example, ranked it 42nd in overall reputation, far behind Google, which is 4th, and even behind Verizon and AT&T)? Or how ineffective this type af advertising might be, especially on a platform where people are trying to interact rather than look at ads for things they already buy or have bought?

But here’s the catch: If Facebook wants more people to click on ads and still doesn’t want to be called a creep, it probably isn’t doing enough data mining. I think some of the best ideas we’ve covered have to do with intent — that is, serving up ads that are in line with what users are actually expecting, or at least receptive to, from the data they’re already giving Facebook. Surely, Facebook’s highly talented data scientists are aware of these methods.

Intent-based targeting might be a bit creepy, but it’s not blunt-force creepy like a peeping tom staring in your bedroom window. It’s more like a cute co-worker whose pickup lines are so on-target you know he’s been researching you, but you’re in the mood to go on a date and he’s there and speaking your language, so …

Solariat Founder and CEO Jeffrey Davitz explained the concept to me last year as mining “big, sucky data” in order to put ads in front of users at the right times on the right topics. It’s a 180-degree difference from showing someone a sponsored story every time a friend “likes” something, placing sidebar ads based on someone’s stated — and static — interests, or even the new partner categories method of advertising for things people might have already purchased and might not want Facebook to know about.

Here’s how I described it then:

Davitz thinks there’s a way for social platforms to overcome this problem by using techniques such as natural-language processing and machine learning to identify those instances where users really are expressing “query-like intent.” It will never be as clear as entering “best hiking shoes” into a search engine, but, for example, someone certainly might note in a wall post or a tweet that he’s going hiking and needs new shoes. He might specifically ask friends which shoes they prefer. If you sell hiking shoes, there’s your signal. Rather than simply peppering someone’s page with ads about hiking because he listed it as an interest, now he’s actually in the market for gear and might pay attention.

This approach could help get Facebook ads heard above the noise that’s surrounding users and coming from their intended purpose for visiting Facebook — social interaction — as well as external sources like Twitter, email and text messages. And if timed right, Davitz noted, users might notice the utility rather than the creepiness. An ad for hiking boots that comes hours later, for example, might be more like a guy gracefully playing the rebound rather than asking a woman out the second he heard she broke up with her boyfriend.

It’s some extra work, yes, but a little nuance might make Facebook seem like it’s not just trying to push our privacy buttons as much as it can before we crack.

NYU Stern School of Business professor Arun Sundararajan nicely summed up the risks of ignoring user intent leading up to Facebook’s IPO last year. He compared annoying — and possibly offending — users with ads at every possible turn with playing the stock market and only thinking about making money. “[I]f you’re investing in the stock market and you’re only thinking about returns and not risk,” he said, “at some point you’re going to lose your shirt.”

Feature image courtesy of Shutterstock user Steven Frame.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• Connected world: the consumer technology revolution

The New Press is funded by publisher and CEO Jan-Jaap Heij and a partner, and arose from the ashes of a previous free newspaper called De Pers or The Press. After about five years of operation, the paper was forced to shut down last year as a result of losses incurred during the financial crisis in Europe, Heij told me in an interview from The Netherlands — but instead of closing its doors entirely, the editor decided to create a new entity with many of the same staff and focus solely on mobile news and content.

Readers can subscribe to one writer or a package

Launched on Monday, the New Press app is powered by a company called imgZine, and allows users to pay about $2.50 per month or $23 per year for a subscription to the entire output of a specific author or journalist, or about $6 a month and $50 per year for a package of all the writers who are currently being syndicated through the platform — a total of 11, according to Heij, with a goal of having more than 50 by the end of the year. In the future, he said, The New Press may also offer packages of writers focused around specific topics such as sports or crime.

Jan-Jaap Heij by Frank Groeliken

Although it is still early, Heij says there has been a substantial amount of interest in the app, and not just from users but from journalists and writers as well. As part of its deal with the authors that it distributes, The New Press shares 75 percent of the subscription revenue with them (after paying Apple its 30-percent cut), and if they get 500 or more subscribers to sign up, they get 85 percent. “We have had about 200 people approach us about being part of the service,” Heij says. “After the launch, my mailbox just exploded with requests from journalists.”

Why did The New Press decide to offer a subscription feature for its writers instead of going the traditional route of a blanket paywall or just a charge for the app? Heij says he believes that media is becoming much more about individual brands than institutional ones, and The New Press wanted to take advantage of that phenomenon:

“As we see it, people are now starting to follow journalists as brands and not the media they work for. A lot of journalists have way more followers on Twitter and Facebook than the brands the media company that publishes them, and I think it’s because people want to hear individual voices, not institutional voices. Media brands are losing power and personal brands are becoming more important.”

Heij said that while individual writers have always had strong personal brands, “it’s far more easy than it was even 5 years ago to publish yourself” and build your own following — to the point where some writers such as star blogger Andrew Sullivan have severed any ties to a traditional media outlet and set up their own publishing operation driven by subscriptions.

Maybe not the only answer, but one of many

The New Press publisher said that the current selection of writers is a mix of established or “star” names from different sectors of the journalism market, as well as newer writers who show promise and are willing to experiment. All sign freelance contracts with The New Press, and while they don’t require a certain number of articles per week or per month, Heij says that the expectation is that they will work hard for their channel or “one push of a button and they are gone.”

Some of the content related to its writers is free, the New Press publisher said — such as blog posts from other sites or their Twitter stream, for example — but the majority is behind the subscription wall. Some journalists have been using the service to publish their older material from other sites, he said, while others are writing daily or weekly pieces specifically for The New Press, and some are a mix of both approaches. The New Press also has staff writers who do a live-news blog that is free to users.

So is the personal subscription model the future of media? The New Press publisher says he isn’t prepared to go that far, but he thinks it will be a crucial part of the future of content:

“We see ourselves as a combination of a newspaper, a news app and a publishing platform. People have asked me whether this is the answer to the crisis in journalism and I have said I don’t think it’s *the* answer, but it is one possible answer.”

Heij, who made a substantial amount of money investing when he was younger, says that he and his partner have invested about 100,000 Euros in The New Press, and the venture also raised about 25,000 Euros through a crowdfunding effort to launch the app. “I’m quite certain we will lose money at the start, because everything you do in media tends to lose money at the start,” he said. “But I am optimistic that this could turn out to be a very significant business.”

This story was corrected at 5:06 pm to note that Heij made his money through investments, rather than by inheriting it as we originally stated.

Images courtesy of imgZine / Frank Groeliken and Shutterstock / artjazz and Flickr user George Kelly

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Building a better paywall: strategies for monetizing news content
• NewNet Q1: Content Farms and Niche Networks on the Rise
• Content Farms: The Players, The Benefits, The Risks

This might not be easy to do — especially since many media outlets seem to have their hearts (and wallets) set on paywalls as a solution — but the industry is in such dire straits at this point that almost any reasonable idea probably shouldn’t be ruled out. Some publications are betting on sponsored content, some are relying on real-world events and others are looking at affiliate links or “brand journalism.” Why not personal paywalls? (Note: We’re going to be talking about alternative monetization strategies at our paidContent Live conference in New York on April 17).

Why personal paywalls? Getting to know readers

I’ve tried to argue in the past that one of the biggest weaknesses of traditional paywalls or subscription plans is the undifferentiated quality they bring to a newspaper’s content: everyone hits the same wall and is asked to pay the same amount, regardless of their interests. This reinforces one of the overall weaknesses many traditional publishers have, which is that they know virtually nothing about their readers — or at least not enough to take advantage of that knowledge in any meaningful way. They are about as personalized as a street-corner newspaper box.

This is important because advertisers in particular are looking for personalized targeting, which is one of the reasons they are looking to new providers such as Facebook and Twitter for their business — those outlets can give them targeting based around an almost infinite number of variables, from income and geographic location to voting behavior. In other words, newspapers and other traditional outlets would benefit from getting to know their readers better in just about any way they possibly can.

One of those ways is to take advantage of the increasingly social nature of media in a digital age, and build monetization strategies around individuals rather than the artificial package of news and other content known as a newspaper. Many readers — particularly younger ones — consume media based not on corporate brands but on individual writers that they feel a connection to, and I would argue that is becoming the norm. We read the New York Times as much for Tom Friedman or Nick Kristof as we do because it is the NYT.

Five ways to create a personal paywall

Not all of these will apply to every writer at every publication, but many will. The overall idea is to take a lesson from the music industry in how to make money from content — the music business has spent a decade figuring out (painfully) that the songs themselves are not what people want to pay for. What they want to pay for is access to artists, both virtual and physical, and for ways of deepening that relationship. So here are some ways newspapers could take advantage of the same principle:

1) Allow readers to pay for an all-in-one package: If what readers identify with is Nick Kristof at the New York Times or Walt Mossberg at the Wall Street Journal or Felix Salmon at Reuters, then give them a way to get that writer’s content — in whatever form — in one easy package. Maybe they blog, write news stories, do video interviews, post on Twitter, etc. Provide all of that for a fee, and make it as appealing as possible and as easy as possible for readers to find and consume it.

2) Create new forms of specialized content: Maybe your wine correspondent is the star attraction for many readers — so why not provide early access to their reviews for readers who sign up for a membership in a personal paywall plan? This is also a model that many musicians have used to their advantage, by providing early access to music (or to better quality files) for members of a fan club.

3) Host live events featuring your writers: Plenty of publications, including The Atlantic and the Texas Tribune, are looking to monetize their content by putting on events that appeal to readers. But not everything has to be a 500-person conference — why not have smaller events that cater to a more exclusive reader group, where they can listen to an interview with a prominent figure in a particular area, and then mix and mingle with other readers who share their interests?

4) Create a virtual community worth paying for: Plenty of newspapers have topic pages or even author pages, but they do little to develop a real feeling of community for readers that justifies an extra fee. This is about more than just content — it’s about providing user forums, or wiki pages about a topic that readers (who pay a membership fee) can contribute to, or a chance for a one-on-one discussion with the writer. In other words, a real community that the writer in question is a part of.

5) Provide access to your writers’ expertise: If you have a writer who has some specialized expertise, whether it’s financial analysis or political savvy or technological knowledge, why not let them provide some of their professional advice to paying customers? This would be similar to a service like Gerson Lehrman or a startup called Clarity, where people buy a specific amount of time to ask an expert questions. Some might see this as a conflict for journalists, but it doesn’t have to be if it’s handled properly.

Offer your core readers more, not less

The bottom line with all of these suggestions is to look at membership or a subscription as a way of offering your readers more than just the regular news and content that you publish — an approach similar to the “reverse paywall” model that Wall Street Journal deputy managing editor Raju Narisetti and journalism professor Jeff Jarvis have both suggested in the past. This bases the monetization on a relationship with readers that is focused on rewards, not just putting up a paywall that everyone runs into after a certain number of pageviews.

Will this prevent some star writers from doing what Andrew Sullivan did and going solo? That’s not guaranteed, but if a writer sees themselves as being in partnership with the newspaper or magazine they write for — something that might even include a share of the extra revenue from the personalized-rewards model — they might be less likely to consider setting up shop on their own, especially if they saw a benefit from the marketing muscle that mainstream publications can provide.

Post and thumbnail images courtesy of Flickr users Mark Strozier and Christian Scholtz, and Shutterstock / Daniilantiq

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Building a better paywall: strategies for monetizing news content
• How Media Companies Can Compete Online
• NewNet Q1: Content Farms and Niche Networks on the Rise

If you’re not familiar with Causata, it’s a true big-data application dedicated solely to stitching together customer identities so marketers know what they want. It collects first-party data — cookies, email addresses, usernames, site activity, customer service phone calls and everything it can, really — and stuffs it into an event store, from where users can run predictive algorithms against the data. Because it takes in such a wide variety of data, Causata stores everything in HBase, the NoSQL database that sits atop the Hadoop Distributed File System and is designed with such unstructured or semi-structured data in mind.

Previously, though, as VP of Marketing Brian Stone explained to me, analytics and predictive modeling within Causata were solely an offline function. Analysts used R, Tableau, Qliktech, plain SQL or their data-analysis tool of choice in order to work through data, learn who’s who among customers and then ultimately build their models. With the new machine-learning capabilities, the system is always looking at how companies are targeting consumers and how those consumers are behaving, and then generating models to predict how certain actions might influence behavior one or even many steps down the line.

Once the data analysts figure out who’s who and how particular microsegments are likely to respond to particular actions, the marketing team can put these models to work in their existing platforms for placing advertising, surfacing offers or whatever other methods they might use to try and reach consumers.

About that personal data …

Any time we’re talking about personal data, though, a certain subset of consumers is likely to get creeped out — and rightfully so. It comes down to that now well-known tradeoff between how much we value personalization and how much we value privacy. Not surprisingly, Stone says he’s open to advertising when it’s “personalized, timely, relevant and intelligent.” If his bank didn’t “continually misfire” in trying to make him loan offers that don’t match his situation — something they should know based on his account information, online banking and site activity — he might actually be willing to take it up on an offer.

Besides, he noted, the only time a human being (at least using Causata’s software) would ever really have reason to look at personal-level data is during troubleshooting or when trying to figure out better methods for segmenting customers. Ideally, this is by activity-based data such as price-consciousness or loyalty rather than classical demographic data such as age, sex or race. But in terms of actual ads or offers served, the system clocks your activity, runs a predictive analysis against your identity profile and returns a result in well under a second.

This happens to be the same method, or at least a similar method, undertaken every time we see personalized ads online: No human being is sitting around, looking at our data and deciding we need hemorrhoid cream.

Given the amount of digital data we’re contractually giving away every time we use surf the web or use our smartphones, combined with the number of companies out there trying to help marketers make sense of it, the personalization genie probably isn’t going back into its bottle. Not that that’s necessarily a bad thing.

I’m reminded of a conversation I had with IBM Fellow and overall identity-data genius Jeff Jonas nearly three years ago. He explained his theory on how extensive data tracking will ultimately lead to a surveillance society but we’ll love it because we love optimization. “It’s seemingly irresistible to us,” he said.

When someone actually gets targeted advertising right, maybe it will be.

To learn more about machine learning, privacy, Hadoop and everything else driving the discussion around big data, come to our Structure: Data event March 20-21 in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

Gravity’s recommendation system is based on its interest graph technology, which we detailed last year. Here’s how I described it then:

[T]he gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

As new content streams into Gravity’s system, it’s analyzed and categorized in real time, then presented to users accordingly based on their interests and behavioral history.

How Gravity’s platform works

Graph processing and graph databases — which store and analyze data based on their relationship to one another — are critical to our onlines lives, powering everything from online recommendations to social search to knowledge discovery. Graph technologies are also the focal point of some impressive life sciences work from companies such as Syapse and Ayasdi, which will be presenting at Structure: Data in New York next month.

But publishers struggling to stand out on a noisy web might have the most to gain from graphs and personalization, generally. At our PaidContent Live conference (April 17 in New York), executives from Prismatic, Zite and Bluefin Labs will take the stage to talk about the importance of personalization for helping consumers filter through the deluge of content online so they can find what they really want. It’s arguable that the trick to keeping readers happy is knowing what they want to read — possibly better than they do themselves.

According to Gravity, its platform currently “delivers more than 25 million personalized content recommendations per day to more than 200 million users. Beta partners have reported click through rates two to three times above previous levels, return visitation increases of 300 percent and session length increases up to 40 percent.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization

Reed explained that the company is still testing ways to make the experience simple enough for consumers as well as resilient enough so Netflix won’t have to change it up much after launch. “Some time over the year, I am sure we will launch it,” Hastings said.

GigaOM was able to capture some exclusive first screenshots of the current personalized profile tests at CES this year. The profiles currently in testing could be offered as an added feature to consumers at no additional cost, allowing them to add a separate profile for each family member. Profiles would include a separate queue as well as viewing history and recommendations.

Netflix has in the past also floated the idea to give consumers access to family plans, which would allow them to stream to more than two devices at the same time for an additional fee. Hastings didn’t give any update on those plans during Wednesday’s call.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1
• Q4 Wrap-up: SOPA and the future of digital content

But first, it’s worth pointing out a few things: The test is ongoing, and could take as long as six months, after which the company will evaluate whether it resulted in increased viewing and engagement times. If things work out, this is going to be rolled out to Netflix’s entire user base as an added feature set.

Also, Netflix typically tests a number of iterations with separate groups, which means that there are other implementations out there. And the final result could look very different from what you get to see here:

The test version showed separate viewing histories and recommendations for each profile. Users are able to fine-tune their profile by telling Netflix about their movie preferences, and profiles for kids under 12 are automatically based on the service’s Just for Kids UI. In addition, parents are able to restrict the type of content displayed on a kids profile based on movie ratings.

Netflix is typically testing these kinds of things on PC-based browsers as well as Sony’s PS3 first. I got to see a browser-based version, and don’t know if there’s also a PS3 implementation currently being tested.

I didn’t have a chance to play with the test too long, but I know that I’d like to see this come to my Netflix account sooner than later: As a parent, I often find all my movie recommendations dominated by kids titles – and as much as I like my kids, there are days when I want to watch other things than My Little Pony and Dragon Tales.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1
• Q4 Wrap-up: SOPA and the future of digital content

Last week, New York Times reporter Steve Lohr wrote about the possibility of a big data bubble forming because people rely too much on data at the expense of experience and intuition. It got me thinking about all the technologies and algorithms I’ve covered, about all the discussions I’ve had about why a data scientist is more than just a statistician who can write MapReduce jobs. Nearly everywhere, it seems to me (save for, as Lohr cites, unique uses such as algorithmic trading), big data really is less about replacing human intuition than it is about augmenting the human experience by making it easier, faster and more efficient.

Like the purpose-built robots that have revolutionized manufacturing, today’s methods for processing and analyzing data are fast, scalable and precise, but they don’t yet (in most cases) make our decisions. Big data can make life and business a lot more efficient, but for the time being, human judgment and willpower are still very much in control.

Offloading grunt work to the machines

We’ve recently covered some obvious examples of this. Take, for example, recent university research demonstrating how media researchers could use machine learning and natural-language processing to save themselves the work of manually reading and coding every piece of text they wish to analyze as part of a study. Algorithms — like robots in manufacturing — are doing the mindless, repetitive tasks of discerning subject matter, keywords and sentiment, but researchers are still the ones poring over those results and telling us what it all means.

A couple months ago, I spoke with Recommind CEO Bob Tennant about how attorneys are using software to pore through terabytes worth of electronic documents during the discovery process. Predictive coding, as it’s called, frees them up to focus more on case strategy than on the tedium of analyzing every single PDF and email message to figure out if it’s relevant to a case. However, he noted, although the software typically does a better job than a person alone would do, most law firms still use a hybrid man-machine approach to leverage the strengths of both and ensure nothing gets missed. And the software certainly doesn’t assess a document’s relative legal relevance in light of a case’s facts and craft an argument around it.

A screenshot of the Analyst Overview

Even software products such as BeyondCore, which aim to minimize human involvement in the data analysis process as much as possible, are actually just about making business people more efficient. In this case, people are only integral to the first and final steps — selecting the metric with which they’re concerned and then interpreting the statistical correlations, respectively. The messy middle step of asking the right questions is (in theory) eliminated by software that analyzes all the possible correlations and scores and presents them accordingly.

In this sense, one of the better descriptions I’ve heard about actually using data in the corporate world came from ClickFox CEO Marco Pacelli, who compared it to figuring out which few of dozens of cockroaches to kill when the light comes on. Big data, like the flick of the light switch, can show people what’s really going on under the surface. But a smart executive still must figure out how to best solve the problem, capitalize on the opportunity or just put the situation into perspective.

Algorithms can only be so human

Of course, those examples are easy and largely ignore the world of really big data that exists on the web and presents its own its own challenges. Lohr, for example, citing Eli Pariser’s “The Filter Bubble: What the Internet Is Hiding From You,” noted a particular fear “that the algorithms that are shaping my digital world are too simple-minded, rather than too smart.” That’s an astute observation in a world of hyper-personalization, where one could easily find himself snowblind by the content, products, etc., he’s supposedly interested in, making it all the more difficult to gain visibility into the broader world.

But perhaps we’re just expecting the web to be smarter than it is and, really, smarter than any service built on the idea of scale probably should be. For example, web and mobile apps, ranging from Amazon Web Services to Instagram, are only able to automate processes for potentially billions of users because they offer fairly generic services (subscription req’d). Broadly applicable features and non-negotiable terms of service (however problematic) mean companies can focus on building great products rather than wasting time negotiating features and terms with every user.

You want data security or site reliability? Figure it out yourself or wait for your service provider to do it on its own time.

A sample interest graph from Gravity.

Why should personalization algorithms be any different? They can do a heck of a job automating the discovery of stuff we’re interested in, but creating a model intelligent enough to know when any given individual wants to — or needs to — view content outside their their typical interests could prove incredibly challenging for services that deliver personalization in part by identifying broad patterns in user behavior. It’s just not what they’re designed to do.

The web is an expansive place: If we as web users really don’t want to be slaves to algorithms and our usernames, maybe it’s up to us to log out, clear our caches and go do some anonymous digging.

Melding man and machine

That being said, the people tasked with creating the algorithms that power so many web services do seem to understand the need for human input in the model-building process, at least. Even machine learning — a term that conjures up images artificial intelligence and self-aware computer networks — is often just a tool to make data scientists’ lives easier through automation.

Smart data scientists knows know they can’t trust the machines alone, which is why companies doing everything from predicting the content you’ll like to predicting your credit risk have figured out how to make machines work for humans instead of replacing them. Yes, machine learning algorithms and big data technologies analyze a volume of data points that humans could never do, uncovering complex relationships the naked eye could never spot. But once the heavy lifting is done, humans come in and use their subject-matter expertise and logic to prune off bad connections, add context and maybe even inject a little serendipity into the final algorithms.

Whether it’s corporate business intelligence or the consumer web, though, all of this is about automation. Data-minded people have always used data to aid in decision-making without ignoring their instincts. Big data just lets them learn a lot more, a lot faster.

We’ll be talking a lot more about these issues and more at Structure: Data, from March 20-21 in New York, so feel free to mark your calendars. In the meantime, here’s a clip from last year’s event with lots of discussion about machine learning, including how humans will continue to play a role.

Feature image courtesy of Shutterstock user Nataliya Hora.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout