Despite the love people have for email, Twitter and even Facebook, the real star of the web in terms of sheer traffic is video. And not only is all this real-time video streaming possibly rotting our brains, congesting our broadband networks and threatening our pay TV businesses, it’s driving wholesale changes in how we pay for broadband and the future of television.

A great illustration of these changes comes from Sandvine’s Global Internet Phenomena Report: 1H 2013. Sandvine provides deep packet inspection and networking management tools to wireless and wireline ISPs, which is how it gets some of its data. While, many people already knew that Netflix traffic comprises about a third of the web traffic in the U.S., they might not know that YouTube is gaining rapidly with 17.11 percent of web traffic downloaded on wireline networks, up from 13.8 percent a year ago.

Video makes bandwidth hogs of us all

Few people are immune to the siren song of cat videos or Arrested Development. In fact, it’s changing the profile of what broadband usage looks like to the point where it’s normal to be a bandwidth hog. According to the Sandvine report in North America, the top 1 percent of subscribers who make the heaviest use of the network’s downstream resources account for 10.1 percent of downstream traffic.

However, those top 1 percent of users don’t look too much different from the top 30 percent. At the bottom, the network’s lightest 50 percent of users account for only 6.4 percent of total monthly traffic. In fact it’s those laggards at the bottom we should be worried about. Did they somehow miss Gangnam Style?

The average and median usage on both wireline and wireless networks in North America is on the rise. On wireline networks mean usage was 44.7 GB, a 39 percent year-over-year increase from 32.1 GB. Over the same period, median monthly usage increased at an even by 56.5 percent, jumping from 10.3 GB to 18.2 GB. On mobile networks mean monthly usage increased by 25 percent from 312.8 MB to 390.1 MB. Yet, median usage more than doubled from 25.5MB to 58.7 MB over the past year, driven in part by more people buying smartphones.

And mobile is even bigger than these numbers make it look like (or something like that). One out of every five bits — or 20 percent of the traffic on wireline network is generated by a smartphone or a tablet. And as Wi-Fi expands and is easier to connect too, that number should continue to increase.

Yes, video traffic will always be big, because videos are big

Before people accuse me of being unfair, let me note that sending video is one of the most data heavy options around. A two-hour HD movie file can contain 4GB of data or more, while a book that might also take two hours to read would top out at several megabytes.

The sheer volume of data is one reason video strikes fear into the hearts of both wireless and wireline network operators, while the loss of revenue from pay TV subscriptions keeps wireline providers up at night. Unfortunately for those implementing usage-based billing plans perhaps in hopes of influencing subscribers to keep their pay TV subscriptions, Sandvine shows that real-time entertainment usage goes up on networks with usage-based billing. In fact, the only thing reduced appears to be file-sharing traffic.

The rest of the report is chock full of great data such as this tidbit that confirms North America’s love of Apple products:

So what single home roaming device consumes the most Real-Time Entertainment traffic at over 10percent? It’s the iPad. In fact, Apple devices as a whole play a large role in the consumption of Real-Time Entertainment. If you add up all Apple manufactured devices (which includes iPads, iPhones, iPods, AppleTVs, and Mac computers), they consume over 35% of all streaming audio and video on North America fixed access networks.

There’s also some good data from Europe that shows that the lowered availability of over the top options like Netflix or the BBC’s video player cause the amount of real-time streaming traffic to drop. Additionally the report shows that in Europe file sharing is higher than in North America, something the report’s authors attribute to a lack of access to certain popular content because of geo-blocking.

But taken in its 40-page entirety, the data and case studies show how our love of video is causing both wireline and wireless ISPs to get creative to boost revenue and meet the challenges posed by the demand for video. Just like we said it would.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What Amazon’s new Kindle line means for Apple, Netflix and online media
• How consumer media will change in 2013
• Who and what to watch in the new era of the living room

“With the frequent, often last minute, changes in content flow the title expiration data available through our API has been inaccurate, so we have decided to no longer publish this information,” a Netflix spokesperson said via email. The company’s Director of Engineering – API Daniel Jacobson reiterated this point in a post on the company’s developer blog, adding that members will still be able to find the expiration date for each movie or TV show episode on the title’s web page.

The move will likely impact a number of third-party services, and comes two months after Netflix essentially closed its public API to all newcomers. Back in March, Netflix said that it was no longer issuing new API keys because the way the company was changing the API had changed: Initially meant to enable third-party apps, Netflix’s API has been playing a key component for the technology behind the company’s streaming service.

Restrictions to public APIs have been a common pattern for companies like Netflix and Twitter in recent months, but it looks like there may have been another reason for Monday’s changes: Netflix took a number of titles off its catalog in early May, leading some publications to write about “the great Netflix Instant vanishing of 2013” or even a “Streamageddon purge.”

Not all of those stories were completely accurate. Some reported a number of 2000 titles disappearing, but Deadline put the number close to 1000. And reports that Warner was pulling titles off of Netflix to power its own streaming service were quickly denied by the studio.

Netflix clearly wasn’t happy about all that streamageddon talk. Now it looks like it pulled the plug on another part of its API to prevent us from freaking out in the future — like at the end of the month, when a number of Viacom shows are set to disappear from the service.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1
• Q4 Wrap-up: SOPA and the future of digital content

Apparently, Hastings disagreed with his engineers about the best way to serve up recommendations. He believed that Netflix could just recommend new DVDs based on the star rating people gave movies. As in: Want to watch a new movie? Then check out these titles that others with similar interests have rated highly. His staff disagreed, and wanted to look at a whole range of other indicators, including the things people searched for on Netflix’s website.

From the story:

“Hastings spent two weeks over his Christmas vacation pounding away on an Excel spreadsheet with millions of customer ratings to build an algorithm that could beat the prediction system designed by his engineers. He failed.”

Of course, that lesson – more data is better – has been a key part of Netflix’s streaming business. The company is tracking all kinds of usage behavior, including every time a subscriber pauses or skips a movie, the order in which titles are consumed and more.

Businessweek reporter Ashlee Vance goes on to say that Hastings’ failed holiday hackathon led to the creation of the Netflix Prize. This $1 million competition pitted teams of researchers against each other with the goal of improving the Netflix algorithm by at least 10 percent. The irony is that Netflix never actually used the winning algorithm, because it had shifted most of its efforts towards streaming by the time the contest finally concluded.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 9 Companies that Pushed the Infrastructure Discussion in 2010
• How consumer media will change in 2013
• OTT technologies and strategies for broadcasters

Fox’s cable TV business made up the vast majority of News Corp’s profits for the quarter ending March 31, contributing $993 million of the $1.36 billion in operating income for the period. Total revenues were $9.54 billion, up 14 percent over the previous year.

The overall home video market is up five percent and “we’re up a bit more than that,” Carey said. “The driving force is digital…the overall marketplace continues to grow really well, and digital is becoming a growing part of what we do.” He also said that the DVD business has stabilized, “with Blu-Ray offsetting the decline in the older formats,” and that “really low-priced rentals” through services like Redbox  are “becoming less of a force.”

In response to an analyst’s question about the future of Hulu, Carey said that the service has “great momentum,” and “we’re particularly excited about subscriptions” through Hulu Plus. “There’s an important role for Hulu Classic in the marketplace,” he said, but “we need to develop the dual-revenue side of it.” In a few years, he said, “Hulu will look a bit different than it does today,” partly in response to changes in Netflix’s business: “Netflix talks about evolving their business to somewhat different business models” (he didn’t elaborate on what those are).

When asked to offer general advice to the broadcast networks, Carey said they are still the “viewership leaders,” but acknowledged the networks might need to “be a bit more targeted in the types of series [they] invest in…networks have been more about the volume game, stuck in historical practices…Do you need to break some of those rules? The answer is clearly yes.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected Consumer Winners and Losers of 2009
• As Q4 Begins, Online Video Is Now Mainstream
• Connected consumer third-quarter 2012

Building out the infrastructure for Snapchat was an act of faith, according to co-founder and CTO Bobby Murphy. The company, which apparently was so easy to build that a Facebook engineer took two weeks to mock up a similar service, operates on Google’s App Engine. That’s a notable choice in a field of startups that have chosen the more popular cloud services provided by Amazon Web Services.

But Murphy likes App Engine, he said in a recent phone conversation, and he believes Google is scaling out and willing to invest in this platform. He prefers some of the features for Snapchat’s purposes and believes when it comes to scale, Google could offer more than AWS for his application. The details behind his consideration will be the focus of Murphy’s chat onstage at the Structure conference occurring June 19 and 20 in San Francisco.

So if you caught Murphy’s appearance on The Colbert Report and want to learn more about the infrastructure and the economics of scaling out an app with 150 million photos uploaded daily, then register for Structure.

Murphy’s is one of several developer-focused talks we’ll have this year as we try to draw more attention to the fact that building out applications on massive cloud infrastructures requires a change in thinking. It’s not just about learning how to build an application in the cloud, but also mandates a strategic approach regarding how to architect your applications in a way that takes into consideration to the economics of hosting them on someone else’s infrastructure.

We’ll have conversations with Cory von Wallenstein, the CTO of Dyn, focusing on how to build a process for evaluating and changing your architecture without disrupting your existing users. There will be another with Gleb Budman, the co-founder and CEO of Backblaze, and Adrian Cockcroft, cloud architect at Netflix, about building hugely scalable infrastructures in the face of serious logistical obstacles.

Structure 2012: Aditya Agarwal – VP Engineering, Dropbox, Adrian Cockcroft – Director, Architecture, Netflix, Alexei Rodriguez – VP of Operations, Evernote Corporation, Jonathan Heiliger – General Partner, North Bridge Venture Partners

Six-and-a-half years ago when we started thinking about our first Structure event, it was a hard sell. People didn’t understand what cloud computing was, nor why a small technology blog would want to build a conference around web infrastructure. Our advertising team got questions like, “You want to hold a show on servers? Why?”

But we knew that just as the printing press changed the distribution of knowledge, the emergence of cloud computing, web-based services and even mobility would change how we disseminate information all over again. And in the process it would create new economic opportunities and change the way the world works.

However, that first Structure conference was about building that vision, not about the servers. If we were around back in the 1400s, we’d hold a gathering at a local tavern not about paper, but about the coming revolutions promised by that technology and and maybe even looking forward to the creation of the novel and widespread literacy.

So make sure you are in the audience at this event so you can predict how the future of the web is changing; not just how infrastructure has evolved, but how we’ll build businesses on top of it. Register here and we’ll see you in June.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

The spot poked fun at both politics and media, and Spacey’s Frank Underwood character didn’t hold back: “It must be so hard to write jokes about a town that already is one,” he said, just minutes after questioning whether NBC is a “real network.” Even Netflix was the butt of a joke, with congressman Steny Hoyer throwing an expletive-laden complaint about the service not working for him.

Of course, the whole clip is also a testament to how much Netflix has changed over the past few months, and how important it has become in Hollywood. Netflix Chief Content Officer Ted Sarandos, who was in the crowd Saturday night as well, recently told me that his phone has been ringing nonstop since the company first announced its plans for House of Cards.

It looks as if one of those calls may have been about Saturday’s dinner: A Netflix spokesperson confirmed Monday that the company worked together with the House of Cards team to produce the spoof, in a way giving back to the town that made its dark and twisted story lines possible. Or, as Spacey says in the clip:

“Well, you know my motto, Ed. You scratch my back, I won’t lacerate yours.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• OTT technologies and strategies for broadcasters
• What the shift to the cloud means for the future EPG

By supporting these tools that help deploy, run and monitor workloads on AWS, Eucalyptus is going a step beyond supporting the bare-bones AWS APIs, Eucalyptus CEO Marten Mickos said in a recent interview.

The new Eucalyptus 3.3 release, due in May, will support Chaos Monkey for testing the limits of a cloud deployment under stress; Asgard for automating deployment of large-scale applications; and Edda, a dynamic querying tool, for polling AWS resources.

“For Eucalyptus customers, this is real proof of AWS compatibility. Other folks who say they are AWS-compatible really aren’t — the real proof of the pudding is in supporting these Netflix tools,” he said. ”We’re not saying that everyone in the world will start using Asgard, although many will.”

That Eucalyptus would throw its lot in with Netflix is not shocking. Mickos and members of the Eucalyptus team attended the Netflix OSS open house in February. Netflix used that event to promote the use of its open-sourced cloud management, testing and monitoring tools by third parties, at least partly so that cloud alternatives to AWS will emerge.

Netflix is one of the biggest and most skillful AWS customers. Netflix tools fill gaps in AWS and help it run better. But Netflix is also acutely aware that Amazon has a streaming video service that is a direct competitor to its own core business and would very much like there to be another cloud out there that is as scalable and price efficient as AWS.

In the open-source cloud world, Eucalyptus contends with a slew of OpenStack players as well as OpenNebula and CloudStack.  But there is concern that the market, as young and potentially big as it may be, will not support all these options. Talk at the recent OpenStack summit and beyond is that there will be consolidation of the contending vendors.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud computing 2013: how to navigate without a map
• Cloud computing infrastructure: 2012 and beyond

So he and his CTO, Alvaro Everlet, spun out a new company to build a platform of software and infrastructure so others could connect devices without having to reinvent the same wheel they had invented so many times before. They called the startup Carriots, and it’s now in the middle of raising a first round of funding.

Like Electric Imp, Carriots, a Madrid-based company, wants to make it easy for people to build out services for connected devices. Electric Imp provides the cloud service as well as an SD card with built-in connectivity. Carriots provides the cloud service, essentially a platform as a service, or PaaS, for the internet of things.

Castillo explained that developers can build the physical product and then link it to a software-based service on the Carriots platform using a few lines of Groovy, a Java-based language. Carriots charges companies for the number of devices they connect to the platform, with the first 10 devices being free.

Carriots has 10 employees and plans to double in size this year, as well as open a U.S.-based office to support growing interest in connected devices and services here. As I see more and more startups eyeing the internet of things, providing some type of back-end platform helps open the field up to a variety of developers who might have an idea but less of a technical background. But I also wonder what devices will work with this type of model.

For example, a consumer-facing device that sells millions might rack up some huge bills, so the team might be better off building out its own infrastructure, especially if it wants to provide access to data via an API. For companies selling to businesses, or with fewer devices, such as Pantry, the startup I wrote about recently that’s building a connected, refrigerated vending machine, it may make more sense.

Of course, if Amazon Web Services has taught us anything, it’s that even large companies like Netflix can benefit from outsourcing their infrastructure if it’s not core to their value proposition. Perhaps the growth of platforms for the internet of things will produce similar examples even as it enables new businesses to come to the fore.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

The paper repeated some key points Hastings has made it the past, but also included a number of noteworthy new insights, including some data points on how Netflix spends its money.

Notably, Hastings said that Netflix is now spending over $2 billion a year on the licensing and creation of content. The company is spending another $350 million a year on improving its service and apps, including improvements to the streaming quality and customer service. And it is spending over $450 million per year on marketing in all of its markets around the world.

Here are some other key highlights of the paper:

On the future of TV

People love TV viewing, but they hate linear TV, including DVRs and cable VOD services, argued Hastings: “The linear TV channel model is ripe for replacement.” Stepping up to replace it are apps from companies like Netflix, HBO and ESPN, which deliver programming to multiple screens.

Technical advances, including 4k streaming and personalized advertising, will speed up the transition from linear TV to app-based on demand programming, and TV Everywhere will make it easier for cable networks to transition into this new world. And eventually, all of this will fundamentally change how TV is delivered, he said:

“Eventually, as linear TV is viewed less, the spectrum it now uses on cable and fiber will be reallocated to expanding data transmission. Satellite TV subscribers will be fewer, and mostly be in places where high-speed Internet (cable or fiber) is not available. The importance of highspeed Internet will increase.”

On Netflix’s focus

Hastings repeated in the paper that Netflix doesn’t want to compete with cable, but just wants to become one more channel — or app — for consumers to choose from. That also means that the company will focus on a few key areas, and for example not venture into ad-supported programming:

“We don’t and can’t compete on breadth with Comcast, Sky, Amazon, Apple, Microsoft, Sony, or Google. For us to be hugely successful we have to be a focused passion brand. Starbucks, not 7-Eleven. Southwest, not United. HBO, not Dish.”

For Hastings, this isn’t just about doing the things at which Netflix excels. It’s also about offering consumers a clear idea what they can expect from the service, which is key to get them to tune in to what he described as “moments of truth”:

“Those decision moments are, say, on Thursday 7:15 pm or Monday 2:40 am when our member wants to relax, enjoy a shared experience with friends and family, or is just bored. They could play a video game, surf the web, read a magazine, channel surf their MVPD/DVR system, buy a pay-per-view movie, put on a DVD, turn on Hulu or Amazon Prime Instant Video, or they could tap on Netflix. We want our members to choose Netflix in these moments of truth.”

On licensing

Hastings also shared some details about the company’s approach towards licensing, which has been shifting more and more towards exclusives. Data is guiding decisions on what to license, explained Hastings:

“We might pay, for example, $200,000 for a 4 year exclusive subscription video-on-demand (SVOD) license for a given title. At the time of renewal, we evaluate how much the title has been viewed as well as member rating feedback to determine how much we are willing to pay. How many similar titles we have is also a consideration.”

He didn’t mention it by name, but this reliance on data could be one of the reasons why Netflix decided to not renew a big licensing pact with Viacom. The company revealed earlier this week that its partnership with the cable programmer is about to expire in May. Netflix is now in discussions with Viacom to license individual shows instead.

The paper also pointed out that Netflix has fundamentally changed the licensing of TV shows in particular:

“It wasn’t easy for cable and broadcast networks to syndicate serialized storytelling to others, and we’ve pushed the price up considerably.”

On HBO

Hastings has long maintained that HBO is its biggest competitor. The company revealed Monday that it now has more domestic subscribers than the cable channel, and Hastings repeated in the paper that he wants to significantly outgrow HBO:

“We have more content, more viewing, a broader brand proposition, are on-demand, on all devices, and are less expensive, so we estimate that we can be 2 to 3 times larger than current linear-HBO, or 60-90 million domestic members.”

However, Hastings isn’t ready to count HBO out yet — and in fact argued that the competition from Netflix will actually help make HBO better as well:

“While we are passing HBO in domestic members in 2013, it will be several years before we are peers with them in terms of Original programming, Emmy awards, and international members. It wouldn’t be surprising to us if HBO does their best work and achieves their highest growth over the next decade, spurred on by the Netflix competition and the Internet TV opportunity.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• OTT technologies and strategies for broadcasters
• What the shift to the cloud means for the future EPG

Hastings was asked about password sharing after Wedbush Securities analyst Michael Pachter had estimated in a Bloomberg story from Monday that as many as 10 million people may be watching Netflix without paying, suggesting that the company may start to crack down on the practice.

Hastings said that sharing passwords with extended family members is “not what we would consider appropriate,” but he added that most of the account sharing would happen within the immediate family — something that Netflix wants to make easier with the introduction of both personalized profiles as well as a more expensive family plan.

Personalized profiles that will allow family members to maintain separate queues and get more personal recommendations will launch internationally within the coming months, the company announced Monday. Netflix has been testing these personalized profiles since the beginning of the year, and Hastings said Monday that the response has been positive. “The key use case is between kids and parents,” he explained, adding that parents have told the company in the past that their experience is suboptimal.

The company also announced Monday that it will launch a new $12 a month family plan that will allow users to stream up to four devices at a time, as opposed to the current two-device streaming limit. However, the company doesn’t expect a huge response to this offering, with Hastings saying Monday that he expect fewer than one percent of subscribers to jump on the offering.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1
• Q4 Wrap-up: SOPA and the future of digital content