Twitter announced last week that historical tweets would now be included in search archives, whereas in the past only tweets from the last few days would appear in results. Twitter has been unfolding its search ambitions over the past year, figuring out how to surface relevant information in a search product that for a long time, hasn’t been all that relevant.

“Now we’re able to look back at old tweets and think about how they should look and how people can relive past moments on Twitter,” said Sam Luckenbill, a senior engineering manager in charge of search at Twitter. “So you can find specific events in the past that weren’t previously available.”

Most of the company’s opportunities in search lie with real-time search, since companies like Oreo want to take advantage of current trends, like the Super Bowl, rather than old news, and Twitter’s clearly betting on the service’s value as a second screen to television with the Bluefin acquisition. But a more complete, historical search provides serious opportunities for the company around monetization (marketers can purchase promoted tweets for specific search terms) and intent (knowing that someone is looking certain information is more valuable than passively viewing something, even if you obviously intended to follow that brand or person.)

But even as the company is now including historical tweets in search results, it’s not all tweets ever that you can uncover — it’s really only a small percentage. The company has worked to develop an algorithm that will figure out which are the most interesting tweets that people would want to revisit, and then ranks those higher than potentially less interesting tweets.

While the company naturally wouldn’t disclose exactly how the algorithm was put together, they said Twitter is attempting to figure out both the most relevant tweets for a specific person and search term (so likely based on your interest and friend graph) and the most engaging tweets overall (which could be reasonably understood as those with lots of retweets, replies, etc.)

“You’ll see tweets from people you care about about and topics that you care about even if the word you chose was a bit vague,” Luckenbill said.

And the filtering doesn’t just apply to tweets that come up in your specific search results — it goes for the tweets Twitter chooses to make available to all searchers as well.

“We’ve started with a small percentage of tweets, looking at the ones that have been engaged with most. And then we plan to steadily increase the size of the index over time to capture more and more,” Luckenbill said. “We’re going to make more and more tweets available that are older, but it’s not necessarily that we want all the tweets ever to be in the index. Because we don’t want to put tweets that get no engagement ever in the index.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A modest proposal for the Google+ search integration problem
• How social search is changing the search industry
• Why Google Should Fear the Social Web

Twitter is currently being sued by PeopleBrowsr, a company that has had full access to Twitter’s firehose, after Twitter said it would restrict PeopleBrowsr’s access to the firehose following the expiration of a previous contract. From Twitter’s perspective, it makes sense to consolidate who is accessing and re-distributing that data due to the scope of the data and work involved in transmitting those tweets. But it hasn’t always been this way, and PeopleBrowsr is challenging Twitter’s earlier commitment to openness when it comes to that firehose of data.

In 2010, Twitter licensed its full firehose to both big guys like Bing, Microsoft, and Google, as well as several startups focused on realtime search, with then-Twitter spokesman Sean Garrett explaining that they wanted to build partnerships that were “sustainable and scalable.”

With access to the full Firehose of data, it is possible to move far beyond the Twitter experiences we know today. In fact, we’re pretty sure that some amazing innovation is possible. Today, we’re happily turning the Firehose on for some new partners focused mainly on exploring the incredibly rich field of real-time search and discovery. We are thrilled to announce that Ellerdale, Collecta, Kosmix, Scoopler, twazzup, CrowdEye, and Chainn Search join us as partners. These companies range from funded startups to part-time, one-person operations so we came up with a fair way to license access that scales with their business. If you think there may be a potential partnership involving access to the Firehose, let’s start a conversation.

Clearly, that conversation about access to the firehose has changed since 2010. A highly-public partnership with Google broke down, killing Google’s realtime search product that relied on the firehose of data. Twitter and Bing re-negotiated their terms in 2010, with AllThingsD reporting at the time that Twitter was asking about $30 million for the access. And most of the startups who recieved access in 2010 are no longer powering search engines: Ellerdale was acquired by Flipboard in July 2010, Collecta has removed their “customer-facing site,” Kosmix was acquired by WalMart in April 2011, Scoopler shut down after the team was acquired by Google to work on Google+ in 2011, and CrowdEye shut down after the founders said they were unable to build a profitable business.

So who’s still has access and distribution rights to the firehose? The full list of companies with commercial partnerships isn’t publicly available, but there are a few like Bing and Salesforce that have publicized this relationship, and several companies listed as part of Twitter’s certified program do have firehose access and can provide enterprise access to that data for other companies who are interested. Duncan Greatwood, CEO of Topsy, which is one of the companies licensed to re-distribute the data, said that including Topsy’s archives, there are now more than 250 billion tweets. That makes it unrealistic, in his opinion, that many companies would be prepared to access the full stream — most companies only access smaller percentages.

But PeopleBrowsr CEO Jodee Rich said his company relies on full access to the firehose to serve its customers, and an agreement with one of the other providers wouldn’t make sense.

“The nature of those agreements are so short-term that no one could possibly build a viable business on those agreements,” he said in an interview. In Rich’s legal statement, which is available online here, he notes that PeopleBrowsr was paying Twitter more than $1 million per year for firehose access.

Topsy, Gnip and DataSift are three providers who license the full firehose of data from Twitter and then re-sell portions of it to either developers building products with the data (showing your influence on Twitter, for instance), or to marketers and brands using sections of tweets to track conversations and trends on the network. PeopleBrowsr is arguing in the court documents that being forced to get data from one of these companies would compromise its ability to serve its clients, since it previously had full access. Twitter counters that it’s totally within its rights to change the terms given that its initial contract with PeopleBrowsr has run its course:

But as Twitter has grown, its contracting practices have matured. Where it once contracted directly with just a handful of data customers like PeopleBrowsr, it now has hundreds of data customers. In order to handle that broad commercial demand in a consistent and transparent manner Twitter has created channel resyndication partnerships with Gnip, DataSift, and Topsy. PeopleBrowsr is free to contract with any of them, just as its competitors do. What it is not free to do is insist that Twitter preserve forever its earlier business model, or continue to be bound by a contract that expired more than a year ago.

A Twitter spokesperson replied that, “We believe the case is without merit and will vigorously defend against it.”

PeopleBrowsr has won a restraining order that forces Twitter to continue firehose access for the time being. As we’ve learned this year, building your company on top of another company’s data is so inherently risky. But the lawsuit does demonstrate the value Twitter’s data has grown to hold, and the lengths companies will go to secure it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• NewNet Q1: Content Farms and Niche Networks on the Rise
• Finding the Value in Social Media Data

The deal gives Yandex access to Twitter’s firehose, and allows users to search it at twitter.yandex.ru, giving them almost instant access to vast troves of data streaming off the messaging service.

The move comes just hours after the Moscow-based service launched a program to incorporate more social networking data into its search results.

And it also comes after Google, which trails Yandex in the Russian market, famously ended its own deal with Twitter — a move that killed off its own real-time search product.

That decision was just one part of an ongoing spat between the two companies, but it appears that Yandex wants to try and take advantage of the situation to further extend its lead over Google in its home market. Currently Yandex is responsible for around 60 percent of Russian searches, but its rival recently stole a few points and moved up to control around a quarter of the market.

Although the terms of the deal aren’t known, it’s likely that Yandex is paying a substantial fee to access Twitter’s data: Microsoft is thought to pay around $30 million each year for the firehose.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet 2012: companies and technologies set to disrupt
• Google and the Ghost of Silicon Valley Past
• Google doesn’t like walled gardens — except its own

The Rice team has a detailed paper explaining the distributed image search solution, which can quickly search for useful images, such as a theft in progress (see above) or clues to the location of an abduction, for example. The researchers suggest that by using Theia software on a handset in combination with a Theia server, specific data from images can be found faster and cheaper than through traditional means:

Through user studies, measurement studies, and field studies, we show that Theia reduces the cost per relevant photo by an average of 59%. It reduces the energy consumption of search by up to 55% and 81% compared to alternative strategies of executing entirely locally or entirely in the cloud. Search results from smartphones are obtained in seconds.

The research paper details the process, but in a nutshell, Theia can remotely search both the metadata (time and location) and the content of photos on cellphones that are registered with the Theia service. To speed up the process and ensure smartphones don’t waste too many CPU cycles searching for content, Theia uses a partitioned search approach: Photos that match certain search parameters initially are uploaded to Theia servers, where the offloaded data can be further searched in real time using more powerful cloud computing servers.

I can see some interesting and practical uses for this, but of course, privacy is a concern. The research team suggests mitigating that by allowing users to specify which photo folder on a smartphone is searchable through Theia. Essentially, users would opt-in by registering their phones with the service, then designating which photo gallery could be scanned.

Although it doesn’t offer a distributed search feature, the photo upload feature of Theia reminds me of the Google Plus Android app, which already has an Instant Upload function. Every picture taken is immediately uploaded to a user’s Google Plus account, but isn’t shared by default. Instead, a user can decide to share it with the public or a specific group post-upload. Since the pics are already on Google’s servers, it could be easy to one day search through them, provided users allowed for that to happen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Bluetooth to Feel Blue as Personal Area Network Battles Loom
• Survey: How apps can solve photo management
• 5 Mobile Companies to Watch in 2011

Eye-tracking on this Google search for Etsy showed that users may not look low enough on the page to see real-time results.

Though you can’t find something until it exists, so real-time search is a bit of an oxymoron. An eye-tracking report from OneUpWeb posted by Kara Swisher today rightfully compares the challenge of finding information that’s both relevant and new to Heisenberg’s Uncertainty Principle. But the study did find that users are already responding to real-time results, especially when they’re seeking out news. Those seeking products, on the other hand, clicked less and found real-time results less useful.

OneUpWeb found Google searchers seeking current news and information clicked on real-time results 30 percent of the time, while users seeking products clicked 20 percent of the time. It also noticed a bias for Google to display real-time results (when it does, which isn’t for every search) lower for consumer goods than for information, which probably had an effect on whether people looked at them or not.

However, the study also found it took participants more than 10 seconds to fixate their gaze on real-time results — which means they may not even make use of them, given the previously reported average time on a search results page before clicking off is within 10 seconds.

News seekers were willing to spend more time on a page than consumers, allowing their eyes to rest on the real-time results page during OneUpWeb’s observation. I’d say that suggests they may actually be getting the information they’re seeking without clicking through.

OneUpWeb also found by surveying participants that news seekers said they liked real-time results more; 47 percent of that group gave real time the thumbs up, compared to just a quarter of those asked to search for products. But more participants from both categories said they were indifferent to real-time results.

None of these measures make real-time results a runaway success, but I’d say they’re actually pretty positive for a change to a product like Google search that’s so often used in our daily lives, and a feature released only this past December.

Related content from GigaOM Pro (sub req’d):

Report: The Real-Time Enterprise

search_gone_wild

BING: Following a hearty endorsement by Microsoft CEO Steve Ballmer in a keynote interview — “I’ve fallen in love with our real-time search; there’s nothing better than our Bing Twitter search” — Sean Suchter, general manager of Microsoft’s Search Technology Center, talked about the value of analyzing not only tweets but links shared on Twitter and Twitter user sentiment about trending topics.

Bing at this point uses only Twitter for real-time search, though it’s supposed to have a deal with Facebook to integrate public status updates. Suchter had no comment as to when that deal would be implemented, but said the Bing team is evaluating how to share the many different ways Facebook users communicate, including giving signals about other people’s relevant real-time updates by saying they like them.

Suchter showed a cool graph (embedded above) of the difference between a network of tweets on an organic topic — the conference we were attending — vs. a spam topic — teeth whitening. It’s pretty easy to see the difference.

Suchter didn’t get very specific about Bing’s real-time special sauce, but he said one of the most interesting ways his team improves Bing real-time search is to look at the past. It takes snapshots of the information available in the world at any one time, evaluates what the biggest thing was, and tries to figure out how Bing could have surfaced that.

YAHOO: Yahoo’s Ivan Davtchev, senior product manager of search, gave a bit more insight into how that company is building a model to determine what tweets are relevant. He said Yahoo emphasizes speed in real-time search but also allowed that freshness is deceptive as a measure of real-time success. Yahoo, which is rolling out real-time search on many of its properties, has built an internal tool called TimeSense (illustrated below) to determine what topics are spiking in real time. It uses language models to group words and then compares them to the body of time before and after. Since real-time updates don’t often include “anchor text” to directly tell search engines what they’re about, promotional factors become more important.

Davtchev spoke more specifically about what Yahoo considers real-time spam: content with “multiple buzzing terms,” overuse of URL-shortening services and overuse of hashtags.

Even though he focused more on the research side of things, Davtchev was the only panelist to get specific about where revenue specific to real-time search might come from. He said he anticipates that one of the most monetizable areas would be local promotions around events.

Davtchev also said to expect Yahoo to use what it learns from real-time relevance on non-search properties — which seems fitting, given Yahoo’s search share and also the fact that so many people use Yahoo as a portal to what’s new on the web.

GOOGLE: Google senior product manager Dylan Casey offered some insight as to how the search giant determines if a real-time update is relevant. “How old was the account, how often do they post, were they often outlinking or inlinking, are they often pointing to the same URL?” He said that Google is trying to emphasize comprehensiveness by including non-Twitter providers such as MySpace and Identi.ca (but Google currently has less access to Facebook updates than Microsoft does, so that might not be his best selling point). However, Google plans to soon publish a standard way to publish directly into the Google indext using pubsubhubbub, he said.

Casey said perhaps the most complex project in real time is to determine when to trigger the appearance of real-time results in search results. “We have huge internal debates on: Is this a good answer to this question, or are we just creating a tool for low-quality content?” he said.

Casey spent some effort justifying Google paying to include Twitter’s real-time firehose of tweets, saying it was an intensive technical integration on both sides, and that tweets are a fundamentally different form of communication due to the restrictions of their form. For example, Google has developed a “complex system” for removing users’ public tweets that are later deleted or marked private.

But even as the giants are sprinting to keep up with real time, they come off as fairly conservative about tweaking their core search experiences. After hearing from the major search players, a mostly emptied-out room was treated to a second edition on the real-time track from four leading startups in the space: OneRiot, CrowdEye, Topsy and Collecta. Seeing things like Topsy’s image search compared to Google’s makes clear the big guys have a lot to learn from the little ones.

Related content from GigaOM Pro (sub req’d):

Report: The Real-Time Enterprise

Twitvid.com is one of many services that makes it easy for users to share videos on Twitter. In fact, the market for video micro-blogging via Twitter has become so crowded that at one point there were two services using the Twitvid moniker. The other Twitvid has since re-branded as Vid.ly and competes with 12seconds, Twiddeo, yfrog, Bubbletweet and others. Twitter video search could help Twitvid.com separate itself from the competition — until others also jump on the real-time search bandwagon.

Real-time search has become especially hot ever since both Google and Microsoft via Bing said back in October that they would incorporate live tweets into their search results. Google and Bing both license Twitter’s so-called “Fire Hose,” meaning the companies get all tweets submitted to Twitter in real time.

Twitvid doesn’t license the Fire Hose, but it’s in a unique position because it has a lot of real-time data of its own. And Twidvid isn’t just trying to outsmart search bigwigs like Google and Microsoft; it’s also competing with similar offerings from Twitmatic and Oneriot, both of which we’ve covered before.

The company told us that it’s also launching advanced video analytics for its users today, as well as an odd feature called “virtual gifting” that sounds a lot like Facebook’s virtual gifts, but with video.

In related news, Imageshack’s yfrog Twitter video platform announced support for BlackBerry and Android smartphones late last week. Applications for BlackBerry OS 4.2 as well as Android 1.5 and 2.0 can be downloaded directly from yfrog’s web site.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• What the shift to the cloud means for the future EPG
• Connected consumer third-quarter 2012

Being that YouTube is such a popular site, it’s quite possible that a live-updated list of trending topics (a sampler: “flashing lights,” “danny gokey it’s only,” “kitten” and “lloyd doggett”) could give us a sense of the cultural zeitgeist. But YouTube comments can’t escape their reputation for being particularly silly and nonsensical (case in point: this excellent McSweeney’s feature: “YouTube comment or e.e. cummings?“).

So — not to be snarky, I swear! — on Friday I ran a little test, searching YouTube comments for real-time mentions of the terms “sucks” and “rocks.” I actually had to do this multiple times, as it turns out YouTube stops counting after it crosses 100 new comments. But in almost exactly an hour, I was able to get a very unscientific window into the sentiment of the general YouTube population.

1:44 p.m.: Started near-simultaneous searches for “sucks” and “rocks”

1:54 p.m.: 10 minutes in, “rocks” has 15 fresh entries and “sucks” has 21.

2:14 p.m.: The split is widening; 41 “rocks” to 59 “sucks.”

2:34 p.m.: 65 “rocks,” 87 “sucks”

2:44 p.m.: Sucks has crossed 100, and now says “More than 101 comments since you started searching.” Rocks” is dragging with just 74.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• What the shift to the cloud means for the future EPG
• Connected consumer third-quarter 2012

Google VP and search maven Marissa Mayer thinks real-time search is hugely important. She told the UK Guardian:

[R]eal-time search is incredibly important and the real-time data that’s coming online can be super-useful in terms of us finding out something like, you know, is this conference today any good? Is it warmer in San Francisco than it is in Silicon Valley? You can actually look at tweets and see those sorts of patterns, so there’s a lot of useful information about real time and your actions that we think ultimately will reinvent search.

OneRiot launch partners include Microsoft, Scour, Yoono, Nambu and Shareholic — and prospective partners can go to wiki.oneriot.com for more information. The API is, of course, free — but by approval only.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• The Case for Increased M&A in 2011: Actions and Outlooks
• Players and Strategies for Real-Time In-Stream Advertising