Why technology and content are inseparable at Netflix

4 Comments

Neil Hunt is the chief product officer at Netflix, and his job entails a lot more than it might sound like. The end product at Netflix is the video streaming through our iPads or smart televisions, but what we’re watching and we’re seeing it is result of a lot of work.

that come along with delivering

First things first: The recommendations

Believe it or not, Hunt thinks the importance of Netflix’s recommendation engine is actually underestimated. Customers watch about 7 billion hours of video every quarter, he explained, which breaks down into about 150 million events per day when someone actively chooses to watch something.

“If you could make just a portion of those, say 10 percent, 15 million viewing events a day, more productive … you could quickly influence the retention behavior of a large number of customers and generate hundreds of millions of dollars of increase in lifetime value from our existing members and not having to replace them at some cost with new members,” he said.

And Netflix is just scratching the surface on improving productivity. Hunt said the company carried out about 300 A/B tests in the past year, many of them involving recommendations, and nearly half of them resulted in users clearly preferring one option over the other.

Neil Hunt.

Neil Hunt.

“Think of it as, roughly, every second or third day there is a measurable improvement in the quality of what we deliver to our customers. That says to me that with that much progress being made right now we’re not near the end,” he said, noting that each improvement could potentially result in tens or even hundreds of millions of dollars for the company. “When we start getting one test out of ten as a success then I know that, well, we’ve tried all the interesting ideas and there is not much left to go and maybe we should not invest so much in it.”

In fact, Netflix is still learning things about how users’ star ratings work, including that ratings given immediately after a movie tend to be more accurate than ratings fired off 50 at a time. And that stars aren’t the be-all, end-all the company once thought they were, in part because people tend to conflate quality with enjoyment.

Hotel Rwanda or Schindler’s List is perhaps a super important movie that deserves a five-star rating, but it’s not necessarily something that you want to sit down and watch after a busy day at work,” Hunt explained.

So Netflix now gives a lot more weight to customers’ actual viewing behavior. It can learn a lot based on what people watch next, and even what they search for and end up watching when Netflix doesn’t have the target movie in its library. Rewinding suggests someone is engaged with a movie but sitting through a movie without ever touching the remote might suggest they’re not paying attention.

Think about that warning, which Netflix called the autoplay blocker, that comes up when you’re in the middle of binge-watching a series and haven’t done anything but stare at the screen for three hours.

“If you run into that and you don’t do anything, then it’s relatively safe to say you probably fell asleep or walked away and left the machine. Maybe we shouldn’t count that as a big part of the signal,” Hunt said. “On the other hand, if you run into that blocker and you say, ‘I’m still here’ and you watch another two episodes, then clearly it was a very motivating thing that kept you engaged.”

stillwatching

Building a data organization that works, and works with business

Hunt said Netflix separates the data experts it hires into three separate teams: the folks who build algorithms and do data science for the whole business; the folks who focus solely on algorithms for content discovery; and the engineers, who scale all those algorithms to run at Netflix’s scale. However, he added, the walls between the teams are fairly porous, so good ideas about any aspect make their way through regardless where they were thought up.

But there is a pretty clear distinction between the product teams in Los Gatos, California, and the content buyers down in Los Angeles. “The contract between us, roughly, is whatever they buy we figure out how to get the most value out of it by putting the right stuff in front of the right people,” Hunt said.

This being Netflix, though, Hunt’s teams will still deliver the people buying and creating content “tremendous” amounts of data about what shows are likely to be popular with which audiences, how many hours they might watch and how much bang Netflix can expect to gets for its buck.

Running it all in the cloud, for better or worse (but mostly better)

Netflix famously hosts its streaming infrastructure on the Amazon Web Services cloud (something we talked about more in depth on our podcast this week, with two senior members of its cloud tech team) and Hunt said that in some sense, that infrastructure is the product. It’s the part that’s responsible for delivering the features everyone else develops to all of Netflix’s users, whenever they press a button.

“It’s not that hard to write an recommendation algorithm that generates recommendations for one person overnight,” he said. “[It’s] much more difficult to be able to deliver 150 million choices a day, each based on a selection of how to prioritize a library of 10,000 titles.”

The company aims for 99.99 percent availability, which translates into a few minutes per month of downtime, but Hunt acknowledged it’s not there yet. Despite all the hard work Netflix’s cloud architecture team has done building resiliency and redundancy into its AWS environment — see, for example, its “Simian Army” set of tools that tests the infrastructure against various types of failures — there’s still more it can do.

A high-level diagram of a new Netflix failover system called Dynomite.

A high-level diagram of a new Netflix failover system called Dynomite.

And while Hunt said admitted he’s sometimes nervous about the AWS cloud, he’s been around enough to know that you can never do away with third-party tools. “It doesn’t matter whether you lease the equipment or buy the equipment, somebody else has a big piece of it,” he said.

In fact, Netflix’s biggest outage wasn’t the infamous Christmas Eve outage in 2012; it was in 2008, when Netflix was running its own servers and a disk upgrade corrupted the databases, Hunt said.

One key to embracing the cloud is to put it in context. At least with Amazon, Hunt added, there’s somebody on the other side trying just as hard to fix things. When there was a Xen hypervisor bug in October that required rebooting 60,000 of Netflix’s virtual servers within AWS, a few people stood back to monitor it but most people went to the Netflix 50-million-customer party and didn’t have to worry about it.

“I think the other key piece here is that we don’t depend on Amazon’s infrastructure being infallible,” Hunt said. Unless there’s some catastrophic failure, Netflix views most most problems as design failures on its end rather than as problems with Amazon’s cloud.

Of course, there’s more to cloud computing than just worrying about downtime — it also brings some very noticeable benefits. Hunt said Netflix can test whatever ideas it has essentially without worrying about cost or deployment time. It can, for example, test two new recommendation algorithms in full production, shut down the less-effective one, and not worry about too many sunk costs or extra servers.

“We pay about twice as much for our infrastructure through Amazon as we did back in 2008 when we ran all our own infrastructure, but we’re about 20 times bigger,” Hunt said. “That’s a tenfold efficiency increase.”

Amazon Web Services' "Cloud Heroes." Adrian Cockroft used to head up the Netflix cloud team, and Jeremy Edberg does now.

Amazon Web Services’ “Cloud Heroes.” Adrian Cockroft used to head up the Netflix cloud team, and Jeremy Edberg now works there.

The economics of 4K and beyond

If you think Netflix is a big company now, Hunt suggests we wait and see what’s coming next, especially as consumers start buying new televisions and want the content to go along with them. He thinks Netflix is a unique position to do what traditional purveyors — cable companies and big-box stores like Best Buy — cannot:

“But internet delivery bypasses some of that. We can, since we’re delivering one to one, deliver 4K for the 1 percent of people who have upgraded their televisions and have a 4K smart TV, without subtracting or taking away at all from the rest of the population.”[/blockquote]

And Netflix can commission a new show, film it in whatever formats it pleases — 4K (aka ultra-high definition) or, next year, maybe high dynamic range — and then deliver it over the internet onto an app it built for a specific smart TV.

“We have unique control over all pieces of pipeline to be able to put 4K in somebody’s living room screen,” Hunt said.

Netflix shot, and streams, "House of Cards" in 4K.

Netflix shot, and streams, “House of Cards” in 4K.

Looking further into the future, Hunt explained a pet project of his that involves working with television manufacturers on a way to let the Netflix video file automatically calibrate picture settings along with a film director’s artistic intent. Most sets are tuned way too bright, and it’s even worse if they’re set to “vivid” or “game” modes. On showroom floors, he noted, sets are often tuned not for the best picture quality but to stand out more.

“That’s a very interesting system problem in the sense that we’re dealing with not just our technology, but the technology of a dozen different television manufacturers,” Hunt said.

As for the extra costs Netflix has announced it will pass along to consumers of these higher-quality video formats, Hunt noted that the $3 a month increase recently put in place for ultra HD isn’t that much different than the cost of upgrading to high-definition from standard definition on some competitive services, and is significantly less when broken down by viewing hours. He also thinks it’s fair.

“It’s a pretty nominal charge for somebody who has got a 25-megabit connection and has just bought a thousand-dollar television set,” he said. “It’s more about making sure that we’re not loading the costs of and bandwidth and storage and all the R&D that goes into it on the student who is trying to watch on the laptop in their dorm room.”

4 Comments

nerdrage

Oooh that screenshot is too cruel. “Are you still watching Deep Space Nine”? My favorite series of all time. If only there was a new Star Trek series like that…or any kind of space series like that, that I haven’t already seen…

Which actually raises an important point: it doesn’t matter how superduperkeeno Netflix’s algorithm is if a) they aren’t licensing enough good content or b) the content we really want to see isn’t even being made! Netflix’s algorithm shows me one of two things: a) already seen it; b) don’t wanna see it. I also maintain a DVD queue, 10x as long as my streaming queue, and a constant reminder of the inadequacy of Netflix’s streaming library. Nerts.

TVwatcher

I hope for Netflix’s sake that CE OEMs start putting HEVC decoders in streaming devices soon. Netflix’s streaming already acccounts for more tha a thrid(!) of peak internet traffic. They need to get bandwidth utiliztion under control.

Sam

I made the point about Netflix not being beholden to many of the constraints of a traditional cable channel or even HBO in a comment on Gigaom last week. Netflix has the power and incentive to create content specifically for the latest generation of TVs. Traditional channels and even HBO are stuck producing for an HD cable box until such time as cable companies decide 4K should be a thing. In order to make 4K happen, cable companies have to take valuable spectrum away from other channels. Netflix can create a different encode for every new TV technology and not worry about making sacrifices for the older technologies. The consumer has the exact TV software they need to view the best stream for their TV. Netflix controls the content all the way from production to delivery. As more and more channels are consumed through streaming and a la carte, I think we’ll see a lot more competition over picture quality that was impossible in the cable/satellite era.

Comments are closed.