Blog Post

5 ways startups are narrowing the content discovery gap

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

Consuming content on the Internet has long felt like drinking from a firehose. Yet the problem is getting even worse. Driven by a focus on page views and the growth of viral content from social sharing sites like Buzzfeed and Gawker, the volume of unfiltered and even unwanted content has exploded. Users continue to struggle to find the signal in the ever-louder noise.

In addition to wasting users’ time, this content discovery gap poses a significant opportunity cost for those who trade in information currency: publishers, analysts, investors, entrepreneurs, executives, journalists and other professionals. Like the adage “half of my advertising is wasted – I just don’t know which half,” the vast majority of content misses its target and goes unread, creating a vastly underperforming asset for producers and publishers. Fortunately, startups and innovative publishers are developing and employing strategies that promise to narrow the Content Discovery Gap.

Curation: A partial solution

Publishers have always “curated” content for audiences, but as the volume and variety of content have expanded, presenting only relevant content to readers has gotten more difficult. A number of startups are betting they can improve on curation – both and NewsWhip, for example, highlight stories by topic that are “spreading fastest” on Twitter. Inside is relying on brevity (approximately 300 characters per summary), clarity (conveying key facts), coverage (more than 1,000 stories a day), craft (individuals trained in the “art” of curation), and customization (readers can select from 175 topics, ranging from “Advertising” to “Yahoo”).

Despite numerous attempts over the years, curation platforms have not attracted a wide following, in part because topic categories (e.g., “advertising”) are too broad for users to effectively filter out unwanted content. In addition, even though the results are abbreviated, they are still a stream. The problem with any stream is that it’s difficult to match content to a reader’s interests at a particular point in time. While curation tools have value, their success depends on how well they learn and match users’ interests.

Chunking: Bits of content, details of engagement

Publishers and curators have always relied on headlines, captions, images and abstracts to convey details of a story to readers that will, ideally, entice them to read it. Mobile app Circa applies a variation of this approach to the content itself, splitting a story into its atomic elements (e.g., facts, quotes, images). With a Flipboard-inspired user interface, Circa allows the reader to swipe to view the next chunk of information in a story. Besides allowing readers to easily get the gist of a story, these smaller, more digestible bits allow Circa to track how much of a story someone has read, revealing the user’s level of interest in a story or topic.

Chunks fit well on smartphones and small-screen tablets, but because users must swipe or click to advance through a story, they are less efficient for users on a laptop or larger-screen tablet. From the publisher’s perspective, chunking provides more granular measures of users’ interests, which should, in turn, enable more relevant recommendations.

The Big Picture: Cloudy with some clearing

As noted media expert Jeff Jarvis has observed, not all news is new. Previous and related content often contain relevant information, especially for professionals and others interested in patterns, trends and “seeing the big picture.” Yet few publishers produce content that meets this need. Users are left to fend for themselves or subscribe to services such as Recorded Future, which shows related developments in an informative timeline.

Startups are attempting to bring the big picture into focus in several ways. For example, each day, a news concierge started by journalist Marie-Catherine Beuth, takes a story in the news and provides a brief overview, a more in-depth backgrounder and discussion of a specific issue within the larger story. “Wonk blogger” Ezra Klein recently left the Washington Post to create the world’s first “hybrid news site/encyclopedia” – his goal is to “explain the news,” not merely report it. Circa also allows users to follow stories and receive updates as the story develops. By revealing the big picture, these and other solutions can help users close part of the content discovery gap.

Social: Leveraging the signals in your social web

Social media play a dual role in the Content Discovery Gap. On the one hand, Twitter, Facebook, LinkedIn and other social media platforms introduce their audiences to new content and sources. However, these ever-expanding streams often overwhelm. Each platform recommends certain content, but the relevance varies widely and is often incredibly poor. Gawker, Buzzfeed, Outbrain and others add to the noise with tantalizing headlines and links to content that, like snack food, might satisfy a craving, but lack substance.

Though social media contributes to the volume and noise, it also holds the key to closing the gap. Who we follow on social media and the links we read reveal a great deal about our interests. Leveraging these signals, provides an email digest of “the most interesting news flowing through your Twitter stream.” Little Bird (founded by former tech journalist Marshall Kirkpatrick) uses social media to help users interested in a particular topic identify leading experts and influencers. On content discovery platform Pugmarks, links that are recommended by members of your social network are given higher priority, based on the premise that those individuals’ opinions are more useful to you.

Personalization: The Holy Grail

Companies such as Amazon, Spotify and Netflix use machine learning and other algorithmic tools to recommend selections that closely match a user’s preferences. Leveraged effectively, personalization takes much of the guesswork out of content discovery – on Netflix, for example, 75 percent of viewers’ activity is driven by recommendations. The equivalent for news – what Jeff Jarvis call his News Pal – has not yet been built for two reasons. First, building a recommendation system requires a lot of data about the user’s preferences. Unlike books, music and movies, readers gather news from multiple sources, so data on one’s interests are scattered across websites. Secondly, building and tweaking recommendation systems is a massive effort, requiring highly skilled data scientists. Few news organizations have made the investment.

Startups and publishers are addressing these problems in different ways. Zite (acquired by CNN in 2012, but recently spun out to join Flipboard) and Gravity (acquired by AOL in 2013) have signed up publishers and track users’ reading patterns across participating sites to compile a more robust “interest graph” for a reader. When an individual arrives at a publisher’s site, the interest graph is used to personalize the content presented. Taking a different approach, Pugmarks’ opt-in “trusted companion” accompanies subscribers as they browse and read stories – while reading a particular story, users are shown recommended content prioritized based in part on their own social networks.

Jeff Bezos’ acquisition of the Washington Post may also harken a new era, with Amazon-like technology deployed more widely by publishers. While it may seem like the Holy Grail, content tailored to users’ needs and interests is within reach.

Dr. Phil Hendrix is head of immr and an analyst for Gigaom Research. He is an advisor to Pugmarks.

16 Responses to “5 ways startups are narrowing the content discovery gap”

  1. Joe Watkins

    This area is clearly a big deal – so much going on. I read an article last week on techAU down here in Australia about another new tool called Cronycle. It looks like they’re tackling a similar problem but in a very different way – less about discovering new content and more about getting hold of the stuff you get already via Twitter and RSS using rules and filters and then sharing that. The article was pretty gushy, except for the fact that it’s a paid service (maybe they like ads more than I do :-). It’s due to come out of beta any time now I think and may be worth adding to the list above – it looked pretty sweet.

    • Phil Hendrix

      Thank you, Joe – “taming” Twitter and other streams represents a big opportunity. Getting users to set up and maintain filters is challenging, so will check out Cronycle to see how they’ve simplified the user experience.

  2. Phil, this is a great post, because content discovery is (has been) indeed the base of our society’s innovation and evolution.

    I agree 100% that the holy grail is personalization. The problem being, everybody is concentrated in what’s popular, viral or crowdsourced, assuming that it is what everybody wants to see. It’s like trying to have a girlfriend that a lot of people are in love with.

    That is why I like so much noosfeer’s approach, because I have discovered great content over there, even when the posts are not that popular, but they are exactly what I wan to read. This also gives a huge opportunity to “small’/beginners bloggers to be discovered.

    • Phil Hendrix

      Thank you, Jacob – I’ll check out @Noosfeer. I like their notion that “content finds the right people.” One of the challenges in personalization is the “cold start” – until observations are accumulated for a new user, it is difficult to determine their interests and begin to personalize recommendations. Provided the user is willing to share, their browsing history, who they follow on social media, topics they Tweet about, and other signals can help. Do you know how Noosfer gets past this challenge?

      • Phil Hendrix

        BTW, there are (at least two) significant negative consequences to the content discovery gap. First is the time users waste (e.g., trying to find relevant content; reading material only to find it isn’t what we were looking for; etc.), which is a huge drain on productivity. The other (and I suspect much larger) downside is the opportunity cost – not knowing what we don’t know, because we didn’t know to ask, inquire, investigate, etc. Your point that “content discovery is the base of innovation…” is spot on – unfortunately, the Content Discovery Gap is a tremendous source of friction and drag on innovation. Solving, even narrowing, this problem represents an enormously important opportunity.

        • David Gordillo

          Phil, we’re actually concentrated on making as few assumptions as possible about the users. For instance, someone that is tweeting a lot about a place, would need a lot of algorithmic effort in order to identify is s/he’s doing it because that person ants to visit that place, or because there is a natural phenomena going on over there, etc.

          This is why our algorithm (and even the way of thinking the UX) is to be as ephemeral as possible, given that the user’s interests as well as also the ‘internet’s mind’ can change from one day to another. For our ‘raking’ on the results we something that we call ‘instant personalized raking’. Which means that all content is teated equally, and its raking is exclusive to each user, thus, what is in top of your search today, would not be in top tomorrow or in two weeks, and certainly is completely different to what would be on top of Jacob’s search.

          Regarding the content discovery gap, we’re working on that, it is true that there is a lot of work to do. We believe that the best approach is to work directly with the content creators in order to “organize the internet”, as for now it is a library with all the books on the ground, we want to put them on their respective shelves.

    • David Gordillo

      Jacob, thanks a lot for your feedback. This kind of comments is what give us in noosfeer a real sense that we’re actually helping and improving somebody’s life.

  3. Marshall Kirkpatrick

    Phil, you sure do keep finding cool startups to profile! Newstapes looks awesome! Thanks for including us at Little Bird, too. One of the ways we’ve begun to aim for personalization is to point automatically to topics we think you’ll be interested in expertise on – based on your social connections online. Lots more to come though! Three cheers for the arms race to build the best engines of discovery – what more could a user ask for?!? Personally, having worked as a journalist in the past, I believe that starting with the right sources is key. That’s the approach we take – we emphasize content discovery through monitoring of peer validated sources of topical information. I’m glad you find it worthy of sharing with readers!

    • Phil Hendrix

      Thanks, Marshall – curating sources is at least as important as important as curating content and the “Source Discovery Gap” is at least as large as as the CDG. For a given topic, as one’s list of “valued sources” grows, a user still faces the perennial challenge of “keeping up” with the most relevant content from those sources. Solution(s) that solve both challenges (timely content from the best sources) really are the Holy Grail. At one time I thought Google’s Custom Search Engines could help… unfortunately, Google’s support for the platform was erratic. Your thoughts on how best to find and easily “stay abreast” of relevant content from valued sources?

      • Phil, Marshall, thank you so much for the mentions. I’m humbled and inspired to be in such great company. This is a great great post, that highlights the many routes there are to tackle one big problem. And I can’t wait to see the progress and discoveries that are still to come for all of these approaches.

        • Phil Hendrix

          Marie-Catherine – your metaphor of a “news concierge” is very useful. Conjures up a trusted assistant who intimately understands not just my interests, but what I’m apt to need next; is “contextually aware;” perhaps even knowing me better than I know myself (what UX experts call “deep empathy”). Curious which qualities of a concierge you feel readers need most and how close we are to delivering.

  4. thanks for the coverage.

    quick note: we are only a month or so old and we are getting what i could call a “significant” amount of organic (non-paid) traffic for our age. We are doing zero marketing and getting thousands of daily users.

    I think we’re starting to crack the code, especially if you look at two specific things:

    1. our /topnews section is half selected by us and half selected by our audience. this is driving some very interesting results since while our users can vote, they can’t add marketing/spam into the ecosystem (i.e. sort of like combining reddit and the new york times). see

    2. breaking news: we’ve been covering a couple of breaking news stories from their inception and it’s resulting in a really great history–something we weren’t anticipating. i.e.

    best, [email protected]

    • Phil Hendrix

      Thanks, Jason – Inside’s close attention to user feedback and responsive, “iterative approach” to innovating are impressive and bode well for the platform and UX (readers – see blog on It will be interesting to see how Inside continues to evolve. Any update on the release date for your Android app?

    • Phil Hendrix

      Dan – agree on the importance of Jeff Bezos’ purchase of the Washington Post. Promises to transform the iconic newspaper with Amazon’s lauded customer-centric focus (and presumably analytics). A number of other developments are worth noting. eBay founder Pierre Omidyar is investing $50 million and assembling a heralded team with deep expertise in journalism and tech to build First Look Media. Jessica Lessin left the Wall Street Journal to launch The Information, a $400/month subscription-based service for professionals who need “the inside scoop on technology news and trends.” Blending incumbents’ expertise with tech startups’ agility could bring much needed disruptive innovation.