Comments on: How Network Statistics Can Make Search Better And More Relevant

By: The most important area of the internet today « Connected

The most important area of the internet today « Connected — Tue, 20 May 2008 16:51:41 +0000

[...] We are way past the critical mass of content and now the focus is managing and find this content in the most efficient way possible. 2007 has seen a ton of changes and new ideas brought to the search conversation. There was a recent article in Business Communications Review that I felt spins our current search environment on its head. The Cliff notes on this article can be found here on GigaOM. [...]

By: Insecure Borders - GigaOM

Insecure Borders - GigaOM — Thu, 07 Feb 2008 20:02:02 +0000

[...] Thursday, February 7, 2008 at 12:02 PM PT Comments (0) You might not be worried about your privacy being exploited by network operators, but how about if the Customs and Border Protection (CBP) decided to search your laptop, iPod and [...]

By: jeremyliew

jeremyliew — Sat, 27 Oct 2007 17:13:43 +0000

Allan,

I believe that this was Snap's original approach to building a new search engine.

By: sam

sam — Fri, 26 Oct 2007 17:23:36 +0000

By the way...

I believe that compete.com has entered into arrangements with various ISPs/Service Providers to take advantage of just this kind of information. If I recall, the CEO of Compete has publicly stated that the data corps will sell this information for the right price.

Of course, this appears to open up a Pandora's Box regarding privacy issues, but I'm also pretty sure that the agreement the user signs to get access allows the ISP to do whatever they choose with the data...

Peace.

By: pwb

pwb — Fri, 26 Oct 2007 01:50:03 +0000

Why don’t the search engines take into account what links searchers actually click on?

By: Allan Leinwand

Allan Leinwand — Thu, 25 Oct 2007 17:06:54 +0000

@John and Chris - Agreed that others are collecting this data (AKAM and others). The key difference here in my mind is the ubiquity of data that could be collected by a service provider. They could grab every bit off the wire for every user. That type of data collection is not an opt-in service like gmail, Google Desktop or browser cookies but would be done without your consent (or it might be put into a EULA). I'm truthfully not sure if this sparks privacy concerns - every bit you put on the Internet should be consider public domain anyway.

The fascinating part here to me is that this algorithm could make search more relevant and would make search engines customers of service providers (and not just for bandwidth).

By: John Gannon

John Gannon — Thu, 25 Oct 2007 07:59:52 +0000

Seems like it would be a good fit not only for Google, but the CDNs of the world (Akamai, etc). They are serving a ton of traffic and have a good sense of who is doing what and from which geographies. If you could securely repackage that data in a way that didn’t compromise anonymity, you’d have a pretty compelling offering to shop to the search providers.

By: Chris

Chris — Thu, 25 Oct 2007 05:55:06 +0000

It really is amazing the progress that has been made in search accuracy as far as getting you closer to what you really want to see. Yet, on the same hand, there are countless cases in which some topics are flooded by things you most certainly don’t wish to see.

In any case, it’s obvious that there are improvements to be made, but it’s nice to see a little delineation of the topic.

Also, I’m a big privacy buff myself, yet I don’t find any data collecting by Google or anyone else to be particularly bad. In using their service, I am consenting to provide my data to improve that service, and even beyond that, if giving up some personal data enables me to find what I want to find faster (or even find things I never would have otherwise found), I’m all for it.

By: Amit

Amit — Thu, 25 Oct 2007 05:44:50 +0000

The concept does put a new way to bring the improvements in searches. But, as Niraj pointed out, the concern lies in the fact that it talks more about popularity than relevancy. So if we say that popular items has to be more relevant to the search, it may lead to a problem when searches deal with complex queries.

One more thing that would need to be tackled is the fallacies in the data provided.

By: Scott Persinger

Scott Persinger — Thu, 25 Oct 2007 03:48:04 +0000

I think we're missing the obvious here. Guess who has detailed stats on visitors, time per page, click path and so on for thousands of sites already?

You got it - Google. Via Google Analytics, they're collecting tons of this data every day. It seems natural and obvious that they will at some point start incorporating that knowledge into search.

By: Chris Albinson

Chris Albinson — Thu, 25 Oct 2007 03:17:52 +0000

Allan,
Cool idea - I think US legislation prevents the carriers from using this data in the way you describe.

However, they maybe able to give this data to a third party on a “no names basis”. This third party could then use the data for targeting. Think - “people in your zip code go to Joe’s plumbing site more than any other plumber… inferred local link love at last”. This search engine optimization combined with a yellow pages local advertising/and a “AT&T” browser = a very interesting model for the carriers -

IF they could ever pull the threads together….
Chris

By: Niraj

Niraj — Thu, 25 Oct 2007 03:17:04 +0000

Sounds like an interesting idea, and I agree that there could be ways to avoid the privacy issue. But I'd be worried about how the information gets integrated because there's the risk that it would end up making popular sites appear higher in results (and thus make them more popular), while failing to let smaller sites rise to the top.

Basically I think it could potentially skew the search results from relevancy to popularity if it is given too much weight.

By: Coleman Foley

Coleman Foley — Thu, 25 Oct 2007 03:02:19 +0000

I wouldn’t care about them taking my information.

By: Allan Leinwand

Allan Leinwand — Wed, 24 Oct 2007 23:44:19 +0000

sam - True, you don’t absolutely need a service provider to gather this information, but if you can gather traffic statistics in the network the data would be complete and potentially more relevant.

By: sam

sam — Wed, 24 Oct 2007 23:32:18 +0000

hey alan....

why the need for the isp to give this information...

you can get most of what you'd need for this kind of analysis by creating a browser plugin that the user would manage for his/her own searching.

the plugin would be able to track/capture various data that the user then deems to make available as he/she sees fit. you'd need millions of users to use this kind of system to be able to have relevance, but the benefits of this kind of approach are many:

1) it's above board, no spammy/slimmy apps 2) user in complete control of the data and where it goes/who uses it 3) a solid core of users who've downloaded/installed the plugin.

the downside, you need to give/provide the user community with something of value in return for being able to access the information... might be better search results, might be coupons, might be who knows...

peace...