I would have been surprised had I not sat down with Microsoft Technical Fellow Dave Campbell the night before the event to talk big data. After all, I was in Redmond — home of Word, Excel and a, shall we say, misunderstood new operating system — not Silicon Valley, where “machine learning” now rolls off the tongue as easily and often as “startup” or “triathlon.”

However, a single rhetorical question from Campbell resonated pretty loudly and got me in the right frame of mind for what I was about to hear: Who else, he asked, has a top-tier web service business (complete with the hundreds of petabytes of data those services collect) as well as a top-tier enterprise software business?

He could have added to that list a consumer software business, 30 percent of the world’s long-distance calls, a mobile device business, one of the world’s most popular gaming platforms, a large-screen touch-display business, and a motion-sensing device that ties into — and can control — all of them. They all came into play at TechForum, as various company presidents, engineers and now-adviser-to-the-CEO Craig Mundie demonstrated a future where everything is connected and trying to learn what we like and what we’re doing.

Bing is the key to it all (even if it can’t touch Google)

Microsoft’s Bing search engine is at the core of everything the company is trying to do in the field of machine learning and cutting-edge big data. That fact makes it an important part of Microsoft’s future even if it never gets close to Google search in terms of revenue or users. “Its long-term value is just as much as a deep infrastructural element,” Mundie said during a Q&A session kicking off the event.

What he means is that Bing is valuable because the technology developed to power it ultimately stands to make Microsoft a lot more money in other areas. Qi Lu, Microsoft’s Online Services Division president (and an integral part of the maturation of Hadoop inside Yahoo earlier this century), describes Bing’s primary architecture as less of a traditional keyword index and more of an “information fabric.” We’re building a digital society, he explained, so there are digital entities — people, place and things — and Bing must be able to capture the rich spatial, temporal and other relationships among them.

A research project for analyzing viral web content.

Taking that vision company-wide, Microsoft can take in data from Bing, Skype, Xbox Live, Office 365 and other sources and actually be able to store, process and analyze it in a meaningful ways. Internally, this might be for business-intelligence or product-development purposes. Externally, Microsoft might use data to create experiences that span devices and services.

Bing also feeds the pipeline for future enterprise IT products, particularly when it comes to data management. Campbell tells the story of meeting a colleague years after he left the SQL database team and went to work on Bing’s infrastructure. At that point, their worlds were vastly different, but the advent of and hype around big data has converged them once again.

Satya Nadella at Structure 2012
(c) Pinar Ozger

During his presentation, Satya Nadella, Microsoft’s Server and Tools Business president, said the company now builds internal IT with a design-for-first-party-but-think-of-third-party mentality. As a result, the core of the Windows Azure cloud-computing platform is based on technologies developed to run Bing, as is the Windows Azure storage service. When Microsoft builds a new operating system, he added, it thinks about the project at webscale in terms of what it would take to run Bing using that platform.

And Campbell told me via email after the event that Microsoft is considering how to productize the various graph, NoSQL and other types of databases it uses to power the features within Bing. Ironically, though, its Cosmos and Dryad technologies that serve as the core of Bing are off the table: consumers demanded Hadoop, so that’s what Microsoft is currently pushing for mass storage and large-scale batch processing.

Google, of course, is doing something very similar, albeit with less of a focus on enterprise software as a final destination for its technologies (with the exception of its small suite of cloud services such as Compute Engine, App Engine and BigQuery). Rather, the types of advances in data storage, processing and analysis that Google has made thanks to products such as search and YouTube are finding their way into Project Glass and self-driving cars. Time will tell whose efforts prove wiser in the end.

A little history and prognostication on machine learning

Mundie said machine learning, especially, has been a core part of Microsoft Research’s focus for years. And although there were some initial struggles, including a dearth of good data and machines powerful enough to process it all, the company and the industry as a whole have come a long way. Among the big areas of improvement he cited were real-time speech recognition — Microsoft has done some impressive work in this area, actually — and natural user interaction.

“We’ve talked for a long time in the industry about IT meaning information technology,” Mundie said, “… you might redefine IT to be intelligent technology.”

Eric Rudder, Mundie’s protégé and chief technical strategy officer, elaborated. If you think about all the pictures and other info Microsoft’s devices and services capture, he said, you’ll see a lot of opportunity to learn and build better products. Stepping out of the consumer world, he questioned how one might begin working with a 40-billion-row Excel spreadsheet. Query it, talk to it or somehow use gestures to communicate with it?

Eric Rudder (foreground) and Craig Mundie (background). Source: Microsoft

Mundie thinks Microsoft can answer these and other questions — this despite a relative lack of attention compared with Google’s research efforts and a consumer community he says is “jaded” by the omnipresence of high technology. TV makers are copying Kinect, speech will be the most-prevalent user interaction and cameras as inputs are coming soon, he said. And Microsoft’s machine-learning research will let it capitalize or even lead the way on these movements, he added.

As I’ll highlight in a follow-up post, Microsoft showed off a lot of these capabilities to the handful of journalists invited to TechForum. Kinect, Office, Xbox Live — they’re all watching, listening, learning and working together.

It’s part of a greater transition away from “specialized gadgets” that process information and into a world full of generally intelligent devices and services that just let people get stuff done. “The vast majority of humankind,” Mundie said, “doesn’t really care about the computer, per se.”

Have research division, will persevere

In the end, Microsoft Chief Research Officer Rick Rashid expects Microsoft’s heavy investment into general research of the kind his team does will help it get the last laugh over some of its competitors. He wonders whether companies like Apple — which already saved itself once — will be ready to ride the next wave of innovation or the one after that without dedicated general research departments that aren’t necessarily tied to product development. His view is that you can only buy yourself into the next generation so many times.

A project (same as the feature image) called Adaptive Machine Learning for Real-Time Streaming.

It was Microsoft Research, for example, that developed a method for compressing 32-bit code in the early 1990s — something that would prove fortuitous when it came time to ship Windows ’95 and its associated applications despite the fact that most PCs lacked the proper hardware for the 32-bit OS. In terms of establishing the dominance of Office over its peers that had to wait until the hardware caught up, Rashid told a group of reporters during the event, “that was game over.”

“Our industry is littered with companies that aren’t here anymore,” he added.

Touché. Microsoft is the butt of a lot of jokes, but as the tech world shifts toward intelligent devices and alternative mode of human-computer interaction, the company’s research into areas such as big data and machine learning suggest it will still be very much around for some time to come.

To learn a lot more about machine learning and the latest trends in big data technologies, be sure to attend our Structure: Data conference March 20-21 in New York. Speakers will include some of the brightest minds in data from organizations such as EMC, Facebook, Cloudera, Quid and even the CIA.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• How search can unlock the power of big data
• The importance of putting the U and I in visualization

If you’ve ever done an online search for an unfamiliar ache, you can probably relate: That weird pain in your side could mean appendicitis, food poisoning or pregnancy.  That nasty rash on your arm could be poison ivy, a spider bite or cancer.

But despite “Dr. Google’s” shortcomings and concerns about so-called cyberchondria, the Web – and search engines in particular – remains a top destination for people seeking out  health information.  The Pew Internet & American Life Project this week reported that about a third of U.S. adults have gone online to look for health information. And, eight in 10 Internet users say their last health-related search began with a search engine – a figure that has not changed since Pew last asked that question in 2000, despite the rise of social media, health-specific content sites and startups. The report also found that those health searchers are reaching diagnoses that their doctors disagree with about one-fifth of the time.

Some physicians are making an effort to adapt to the new reality, but given the frequency with which people seem to page the doctor in the search box, more clinicians need to do the same. “Our relationships with patients was once entirely defined by our unique access to information,” said Bryan Vartabedian, a Texas physician who is active on Twitter. As patients access new information, their relationship with physicians is changing, he said. “The biggest challenge facing the health consumer in 2013 is online health literacy – understanding what’s reliable and what isn’t,” he added.

A few studies have attempted to evaluate the reliability of search engines but with mixed conclusions.  A 2001 study in the Journal of the American Medical Association concluded that using search engines to find medical information wasn’t efficient.  A study in 2006 found that when two medically trained doctors typed symptoms into a search box, Google was able to provide an accurate diagnosis 58 percent of the time. Other studies of varying quality have added other dimensions to the debate over time.

On one hand, the Web can help direct people to valuable information and studies that even their doctors may not be aware of. But search engines alone don’t give people enough ways of gauging a source’s reliability or providing the context they may need to make the most of sources that are actually good.

Dr. Vartabedian said he sees patients and first makes a diagnosis or discusses a condition, he warns them about the irrelevant information they’ll likely encounter online to stave off future concern and unnecessary questions.

Wendy Sue Swanson, a Seattle pediatrician, uses her hospital’s blog and Twitter to help her patients and others navigate online health information. She said that if Google can help patients feel more in control or nudge them into the doctor’s office, that’ s not a bad thing. “When you’re in a moment of fear, you don’t think about a specific site… they’re compulsive,” she said. “The benefit of being a clinician is that I know more about the sites and how realiable they are – but that’s where we’re not doing a great job.”

Tools that connect doctors with patients in HIPAA-compliant digital environments are growing – HealthTap, for example, helps patients directly ask doctors questions online, and Ringadoc lets people consult physicians via video conference. But they’re just beginning to appeal to doctors who are wiling to define their roles and organize their time differently.

Twitter is another way doctors are able to influence online health behavior en masse, but a recent study found that just 7 percent of doctors tweet (even though half use physician-only communities to learn and contribute information). If more patients could easily reach their doctors via email or other electronic messages, they might also be less inclined to search aimlessly for answers on the Web.

Meanwhile, insurance reimbursement and payment processes mostly don’t include electronic communication, and more systems and policies are needed to ensure patient privacy and assuage liability concerns.

As mobile adoption grows and digital natives age, a doctor willing to email you and curate online information isn’t just going to be a nice to have — for many, it will be a need to have.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Examining the rise of crowd labor platforms in 2012
• Social third-quarter 2012: analysis and outlook
• Listening platforms: finding the value in social media data

According to a new report from the Pew Internet & American Life Project, 8 in 10 Internet users say that their last health-related search began at a search engine, such as Google, Bing or Yahoo – a figure that has not changed since Pew last asked that question in 2000.  About 13 percent say that they started at a health-specific site like WebMD, 2 percent say they began at a general site like Wikipedia and 1 percent said they started at a social network like Facebook.

The findings are especially interesting given all the health content websites and startups that have launched in the last 13 years that are trying to position themselves as go-to online health destinations for consumers. Those companies include startups like Symcat and Meddik, which provide health-specific search engines that directly take aim at “Dr. Google,” as well as content sites like Everyday Health and WebMD.

“Health websites have not gained mind share among consumers over the last decade,” said Susannah Fox, Pew’s associate director of digital strategy.

Given internet users’ familiarity with search engines, the statistic makes sense. But the stakes for health searches are arguably higher than for most other searches, and Google’s ability to return reliable information is up for debate. The study has implications for how doctors consider online health searches in talking to patients and how health content companies optimize and position themselves online.

You can see more of Pew’s report on its site, but here are a few more key findings:

• Almost a third at 35 percent of U.S. adults, have gone online to figure out a medical condition. Of those, half went on to see a medical professional.
• Seventy percent of U.S. adults said that the last time they had a serious health issue they got information, care or support from a doctor or health professional.
• Half of health information searches are conducted on behalf of someone else.
• One in four people seeking health information have hit a paywall.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The state of cross-platform media measurement
• Social third-quarter 2012: analysis and outlook
• How social discovery is transforming entertainment

“Here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!” — The Red Queen, Alice in Wonderland

When it comes to the evolution of newspapers in a digital age, the New York Times is clearly something of a bellwether — and in particular, a sign of whether paywalls can (or can’t) make up for the ongoing dramatic decline in advertising revenue. Unfortunately for anyone in the industry who was hoping for a definitive answer, however, the paper’s latest financial results are a mixed bag. Subscription revenue rose, but both print and (perhaps most disturbingly) digital advertising continued to fall, raising the question of whether the Times will ever be able to close that gap, or whether it will have to become a much smaller and primarily reader-financed business.

The paper has already crossed the crucial point at which revenue from readers — whether through print subscriptions or digital subscriptions — has eclipsed the revenue the paper gets from advertising, a transition that other newspapers such as the Financial Times are also quickly approaching. But as more than one person has noted, there are two ways to reach that point: one is to have your subscription revenue increase, and the other is to have your advertising revenue decrease. Both of those are happening at the Times, and the only question is which one will slow down first.

Print ad revenue still falling, but digital down as well

As my colleague Jeff Roberts reported, overall advertising revenue for the company — which includes The Boston Globe — declined by 8.9 percent to $182 million, with print ads down by almost 11 percent and digital off by over 2 percent. For print, the story was the same as it has been for the past few years: continuing double-digit declines in real estate ads and other key classified categories that used to be bread-and-butter for newspapers. For digital, however, the picture was less clear. A spokesman for the paper said the drop in digital advertising was a result of:

“The challenging economic environment, ongoing secular trends and an increasingly complex and fragmented digital advertising marketplace.”

On the company’s conference call with stock analysts, the company tried to explain what one analyst called the “shockingly bad” advertising numbers, blaming “an abundance of inventory” and “efficient buying methods such as programmatic buying” offered by search engines and large advertising portals like Google and Yahoo. In other words, CPM rates — what advertisers are willing to pay for every thousand viewers of an ad — are continuing to fall, and the New York Times seems unable to shore that up at all (at least so far).

Subscription revenue was a much nicer story, since it rose by 7.4 percent to $235 million, but at least some of that has come from higher prices for the print product as well as an increase in subscribers to the digital product, where the paper now has 566,000 paying readers (although it’s not clear how many of these are subsidized in some way by discounts, etc.) And even though the boost in subscription revenue amounted to about $16 million for the quarter, that was still less than the $18-million or so drop in overall advertising revenue. So one of those factors is still falling faster than the other is rising.

The risk for the Times — and for a growing number of other newspapers that are making a similar bet on paywalls as a solution for their problems — is that advertising revenue could continue to fall, and even pick up speed, but the same probably can’t be said for the rise in subscription revenue. At some point, the number of people who want to subscribe to either print or digital will stop rising so quickly and begin to level off, the only real question is when.

A paywall can also reduce advertising revenue

There’s related risk with a paywall that the NYT and other newspapers face, but it’s one they don’t talk about much, and that is the effect that charging readers — which necessarily involves pushing some readers away — itself has on advertising. The Times has said that the decline in digital readership (and therefore the potential impact on advertising revenue) has been relatively modest, but results from comScore suggest otherwise: the numbers appear to show that pageviews have fallen by about 15 percent and unique visitors by almost 20 percent.

Newspapers like the Times could theoretically argue that each of their visitors or readers is much more valuable because a large proportion of them have paid to view the content, but it’s not clear whether advertisers will buy that argument, or be willing to pay higher rates for the privilege. It could be worse for the New York Times, of course — it could be reporting results like McClatchy, which saw both circulation and advertising revenues decline. But the question of what the NYT does as ad revenues fall and subscription revenues slow remains to be answered.

It’s true that no one is really doing a booming business in digital advertising right now, even Facebook, because the business is in the process of being disrupted. That’s why it would be nice to hear about how the Times is trying to change the way it does advertising as well as resting its hopes on a paywall — which is part of the reason why I think that if a paywall is your only strategy, you are probably doomed to be a much smaller business than you are now.

I’ve argued in the past that a membership model that provides added benefits for valued readers — whether it’s ebooks or live events or some combination of such features — is a better approach than a paywall, and the experience of some publishers such as The Atlantic seem to indicate that this is so. To me, newspapers like the Times would be better off trying to get to know their readers better so that they could offer them advertising or other features, rather than simply hitting them with an undifferentiated paywall.

In the end, newspapers watching the Times have to ask themselves not just whether they can duplicate the short-term revenue success of the paper’s paywall (which some or all of them may be unable to do), but also what happens when that still doesn’t make up for the decline in ad revenue.

Post and thumbnail images courtesy of Flickr users jphilipg and Flickr user Giuseppe Bognanni

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social media in Q1: commerce and discovery dominated
• Putting Big Data to Work: Opportunities for Enterprises
• The state of cross-platform media measurement

The Portland-based company, which is being demoed at PIE [Portland Incubator Experiment] Demo Day this morning, is also announcing a $1 million angel funding round today. Mark Cuban’s Radical Investments led the round, with participation from Howard Lindzon’s Social Leverage Group, Hubspot cofounder Dharmesh Shah and former Twitter engineer Blaine Cook. MetaFilter founder Matt Haughey, Accel’s Jonathan Siegel, Blogads’ Henry Copeland and social media expert Jay Baer also invested.

Little Bird determines which people are most influential on any given topic based on their personal connections, rather than on the content they create. “Unlike almost every other service out there, we are not doing content analysis for discovery of influencers,” Kirkpatrick told me. “We are looking at the specialists that other specialists are paying attention to…I think of it almost as a robot librarian. Whatever topic I’m interested in, I have the ability to snap my fingers and say, ‘Bring me the world’s most trusted neuroscientist.’”

Little Bird’s algorithm works by crawling the social graph of Twitter accounts and reducing them down to the “most trusted specialists” on a topic. (So you won’t necessarily be finding the world’s most trusted neuroscientist, but you might be finding the most trusted neuroscientist who’s also active on Twitter.) “The only way to climb up the ranks is to win the respect of your peers,” Kirkpatrick said. Though Little Bird is only crawling Twitter and blogs for now, the company plans to add LinkedIn and Google+ accounts soon. “We have reason to believe there are some serious professionals out there using Google+,” Kirkpatrick said, but “nobody knows who they are now.”

While Little Bird sounds somewhat similar to Klout, Kirkpatrick stressed that it’s “almost the opposite of a black box” in that it’s designed to be transparent and rational. He also says it’s a better discovery tool. “If you’ve already got someone in mind, Klout can tell you what their general popularity across the web is. But if you need to do discovery and what you’re really looking for is influence among other specialists, that’s something we provide better than Klout or anybody else.” Even so, he said, Klout data could eventually be added to Little Bird’s algorithm: “Sorting by Klout score would be an interesting way to display the data we’ve discovered.”

Along with person discovery, Little Bird offers various content discovery options. A “hot content” tab shows the links being shared the most around a given subject. “It’s like Techmeme for any topic,” Kirkpatrick said. A “top blogs” feature ranks blogs based on the number of inbound links, and a custom search engine lets users search inside “a whitelist of trusted domain experts” rather than across the web at large. Finally, if you’re in the mood for navel-gazing — and of course you are — “scorecard” lets you compare any Twitter account to other influencers on a topic.

So how well does it work?

I tested Little Bird and had pretty good results. I tried searching for two topics, “book publishing” and “ebooks,” figuring that I’d be most familiar with the results and would be able to gauge how good they were. My search for “book publishing” mostly turned up publisher accounts (for Simon & Schuster and Sterling, for instance) rather than individuals. Kirkpatrick admits that “in some sectors, companies do dominate,” but users can filter their results to show only individuals. When I did that, my results were much better.

My search for “ebooks” turned up a mixture of people I know and would actually consider influential, but also a number of marketing, company or promotional accounts that people probably primarily follow in order to get freebies. There were also many users speaking in foreign languages (though you can confine your search to a geographic area).

My search for “hot content” around ebooks revealed basically useless results: Two separate tweets about Catalonian independence (from the same user), two duplicate tweets from an editor and one tweet about Mitt Romney’s education policy.

These results may improve when Little Bird starts pulling in sources beyond Twitter. And the service also might be better for broader topics. A search for “broadband” pulled up our own Stacey Higginbotham, while “journalism” found people like Clay Shirky and Tim O’Reilly.

What it costs

Today you can view a few reports for free. Then Little Bird is rolling out free previews and subscription access in waves to individuals, small businesses and large businesses, with general availability expected in the next year. Individual accounts are $50 a month, and business accounts range from $250 for companies with three or fewer employees to $1,000 a month for companies with 26 to 500 employees (larger companies can get in touch for custom pricing). “It’s a little less expensive than [social media monitoring tool] Radian6 and more expensive than Meltwater,” Kirkpatrick said. “It’s a lot less expensive than hiring a consultant or agency to go out and do this research.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social media in Q1: commerce and discovery dominated
• Web startups: How to guard against security breaches
• Facebook’s IPO filing: ideas and implications

Well, actually that’s slightly inaccurate: the draft law would make search engines pay for reproducing newspapers’ headlines and first paragraphs. So, take those away and the links are fine. Even if nobody will have the faintest idea what they’re linking to.

Google’s North Europe communications chief, Kay Oberbeck, sounded off about the issue this morning in a guest post for a German press agency. That was in German, of course, so I got him to vent in English as well:

“Nobody sees a real reason why this should be implemented,” he said. “It’s really harmful, not just for users who wouldn’t find as much information as they find now, but such a law is also not justified for economic reasons or judicial reasons.”

Oberbeck also pointed out the obvious: that Google send readers to the publishers’ sites. And that anyone who doesn’t want their content to be indexed by Google can just throw a robots.txt file in there. And that publishers make money off Adsense.

But wait, let’s back up. To appreciate the full absurdity of the situation, we should take in a little history.

The German publishing houses, particularly Axel Springer, are very powerful in their country, with relatively strong influence in government circles. As Matthias Spielkamp of the copyright news site iRights put it to me:

“If you look at the U.S., if print houses there want something, they are up against American companies like Google and Yahoo. Here we have local publishers that are enormously powerful and are trying to target U.S. companies. I wouldn’t say it’s anti-American – it’s just that German politicians are much more inclined to protect German publishers’ interests when balancing that with a [foreign] company or industry.”

A couple of years ago, a leaked draft showed what plans the publishing houses were pitching to their friends in the coalition government. The first official draft legislation showed up in April. What it proposed was breathtaking.

The government was calling for a form of ‘ancillary copyright’ to be brought in, that would force companies to pay publishers license fees for using their work in a commercial setting. As in, employers would have to pay up for letting their employees read the news online at work.

German industry bodies were predictably apoplectic, as were opposition parties, and the government beat a hasty retreat. The second draft, which appeared in the last couple of months, drastically narrowed the scope of the legislation, so that it would only apply to search engines.

So now Google is furious for being picked on, when it actually drives traffic to the publishers.

And the publishers aren’t happy either – Anja Pasquay, a spokeswoman for the Federal Association of German Newspaper Publishers (BDZV), told me that the second draft “won’t help”, and her organization would rather see a revival of the first draft.

So, with nobody happy, and with the government looking increasingly isolated, a third draft is rumoured to be in the works. That’s why Google is piping up now.

“An ancillary copyright would mean a massive damage to the German economy. It’s a threat to the freedom of information. And it would leave Germany behind internationally as a place for business,” Oberbeck told me.

“Publishers should be innovate in order to be successful. A compulsory levy for commercial internet users means cross-subsidizing publishers through other industries. This is not a sustainable solution.”

On balance, it’s difficult not to take Google’s side on this one. The whole idea of this kind of ancillary copyright is ridiculous, and it puts the likes of Axel Springer in a very poor light indeed.

It’s not as though Axel Springer isn’t plunging headfirst into the web industry itself – only today, it announced the purchase of an online news and classified portal.

The German publishing giants are big enough to compete in the real world. Sure, it’s tough monetizing free web content. But cooking up hokey and self-defeating new copyright laws is a pretty shabby way to go about it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social media in Q1: commerce and discovery dominated
• The state of cross-platform media measurement
• Social third-quarter 2012: analysis and outlook

Google notes it won’t actually remove possibly pirated content from search results unless it gets a notice from the copyright owner, but the move is a step to push legal content higher in search results — so a query like, say, “Watch Mad Men online” should direct users to AMC’s site before an illegal streaming site.

The film and music industries have long pushed Google to tackle piracy more proactively. A case underway in France could force Google to censor terms like “torrent” from its autocomplete results. And the number of copyright-infringement takedown notices that Google receives has increased sharply in recent months. The week of January 2, 2012, Google received 239,189 such notices; last week, it got just over a million.

Michael O’Leary of the Motion Picture Association of America released a statement:

We are optimistic that Google’s actions will help steer consumers to the myriad legitimate ways for them to access movies and TV shows online, and away from the rogue cyberlockers, peer-to-peer sites, and other outlaw enterprises that steal the hard work of creators across the globe. We will be watching this development closely — the devil is always in the details — and look forward to Google taking further steps to ensure that its services favor legitimate businesses and creators, not thieves.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social media in Q1: commerce and discovery dominated
• Controversy, courtrooms and the cloud in Q1
• The 2013 task management tools market

For Google, it’s all about collecting and analyzing billions of data points to learn what each one really means. With Knowledge Graph, for example, Google uses a “database of more than 500 million real-world people, places and things with 3.5 billion attributes and connections among them.” It’s those connections that are the key, as they’re what make the system smart enough to know what you’re looking for that wouldn’t naturally show up in a standard keyword search.

Although Google hasn’t come out and said so, I’d imagine the Knowledge Graph utilizes Google’s Pregel graph processing engine. Graph processing and databases are catching on in social networks and other large-scale environments because they organize pieces of data by how they’re connected to one another. Those connections are called edges, and they’d keep Knowledge Graph results both informative and focused because the system knows how closely they’re related in any given circumstance.

This example of a personalized interest graph from Gravity Labs illustrates how one might visualize a graph, in this case the connections between a reader’s perceived interests:

Of course, Google has another tool at its disposal, which is the collective wisdom it’s able to glean from billions of searches every day. So, as Singhal wrote when first explaining Knowledge Graph in May, “[W]e can now sometimes help answer your next question before you’ve asked it, because the facts we show are informed by what other people have searched for. For example, the information we show for Tom Cruise answers 37 percent of next queries that people ask about him.”

Google’s other big announcement today is improved voice search on mobile phones, both Android and iOS. Here’s how Singhal describes the new capability:

You just need to tap the microphone icon and ask your question, the same way you’d ask a friend. For example, ask “What movies are playing this weekend?” and you’ll see your words streamed back to you quickly as you speak. Then Google will show you a list of the latest movies in theaters near you, with schedules and even trailers. … When Google can supply a direct answer to your question, you’ll get a spoken response too.

On Monday, a Google Research blog post noted how the company’s work on neural networks — which it famously used to train a system capable of detecting cats and human faces in video streams — is being used to power speech recognition in the Jelly Bean release of Android. Seventeen-year-old Brittany Wenger recently won the Google Science Fair by building an application atop Google App Engine that uses a neural network to help detect breast cancer.

As one might imagine, however, the big challenge for Google, Microsoft , Apple and everyone else trying to provide intelligent but intuitive user experiences is figuring out how to shape high computer science into easily digestible formats on ever-smaller devices. Search would certainly be a more effective tool if everyone could write complex queries directly against a company’s database, but the trick is making products good enough that we don’t have to. It’s boiling years of machine learning, natural-language processing and neural network research into “you ask a question and your phone spits back the right answer.”

Feature image courtesy of Shutterstock user Sebastian Kaulitzki.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Several of the entrepreneurs presenting their startups at Rock Health’s demo day on Wednesday held both medical and business degrees, which was reflected in their focus on using technology to improve efficiency and communication for medical doctors and their health-conscious patients.

Rock Health is a seed accelerator for digital health startups, and its second class of entrepreneurs presented their pitches on Wednesday. In exchange for admission into the program, the startups received $20,000, San Francisco office space, and five months to launch their ideas.

Three of the startups presented particularly interesting solutions to problems faced by medical professionals, and should be on your radar as they grow:

Docphin

Docphin has built technology that makes it easier for doctors to search for, download, and share medical journal articles without the hassle of multiple academic login screens. Doctors connect with their academic institution for access, and then search for articles that interest them, search by keyword, or check articles that were released since they last logged in. “It’s Kayak.com meets medical research, except you’re not trying to book a last-minute flight,” said Mitesh Patel, co-founder and CEO.

Docphin is also building an enterprise product that allows institutions to pay a licensing fee and then distribute training videos and documents to its doctors. Patel said Docphin is getting ready to close its second seed funding round, and has recieved investment from groups including the Mayo Clinic. It’s easy to understand how doctors used to readability apps like Instapaper or modern search engines would be attracted to Docphin’s product.

Cardiio

If you’re carrying an iPhone in your pocket, you’re carrying an incredibly sophisticated computer, and startup Cardiio looks to take advantage of that technology by using the iPhone camera as a biosensor to measure patient heart rates. How does it work? Users simply hold the iPhone camera up to their face, tap a button on the screen, and wait for the iPhone camera to measure light reflected off the person’s face. Increased blood flow reduces the amount of light reflecting off a person’s face, so the iPhone camera sensors can detect a person’s heart rate based on that information.

The app is incredibly fun to use, and would likely appeal more to consumers hoping to track their heart rate on the go than doctors in a medical setting. But the possibilities for use seem fairly widespread. The biggest question with Cardiio seems to be whether or not the technology will work consistently. Cardiio’s founders said they’ve tested the app against heart rate monitors, and it’s been very accurate. The Cardiio iPhone app will likely launch in the iTunes Store this week, and the company is working on an Android app coming up next.

Agile Diagnosis

Trying to remember the different steps to take in diagnosing a patient’s illness, but don’t want to carry around physical medical textbooks or copies of the the latest medical guidelines? Agile Diagnosis is an HTML5 web app that produces flow charts for doctors on the go, aggregating the latest medical recommendations from textbooks and professional society guidelines. The company began with information from commonly-used medical texts, and is expanding it to include guidelines from medical societies and widespread medical literature. The information is presented with hyperlinks in an interactive flowchart, letting doctors select for further information or skip steps they understand well. Experienced doctors can also use the charts and information to explain medical terms or conditions to patients.

Most recently, 30 percent of the app’s traffic came from iPhone users, closely followed by 20-30 percent from Windows desktop users. The founders said Agile Diagnosis is intended to provide doctors with the latest resources at their fingertips, and provide feedback to the societies writing and developing medical guidelines for doctors.

Agile Diagnosis already has 10,000 users since its beta version launched in March, representing more than 240 academic medical centers and 20 hospitals. The company has raised 2.5 million in funding, and is looking to expand its team.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Podcast: Mobile winners and losers in 2012 and what to expect in 2013
• Analyzing the wearable computing market
• Google doesn’t like walled gardens — except its own

Not quite.

While such accounts are a pain — and Twitter spends significant resources trying to hunt them down — a new study says that the most egregious promoters of Twitter spam could actually be the likes of Lady Gaga or President Obama … and even you.

The study, from researchers at the Max Planck Institute for Software Systems in Germany, examined more than 40,000 identified spam accounts, and looked into how they tried to achieve success.

“Pretty much everyone on Twitter is targeted by spammers,” says Krishna Gummadi, head of the networked systems research group at MPI-SWS. “But the key question for us was ‘why would someone follow a spammer?’. So rather than focus on spammers themselves, we looked at their support network — the people who helped them.”

And what they discovered was that instead of being supported by other spam accounts — as is common with web-based spam — the egg people were reliant on real Twitter users for their success.

How it works

At the heart of Twitter’s spam problem are two things: the way the social messaging service works, and the way search engines think about the site.

First, Twitter prominently displays follower counts as a crucial measurement of popularity — something Google and others use that number to help determine somebody’s authority. Then, there’s a secondary effect: if you are followed by somebody who is followed by lots of people (what the researchers term somebody’s “indegree”) then your influence increases even more.

This leads to a kind of Twitter-based link farming, where spammers don’t simply target small-time users with just a few friends, but larger accounts with thousands — or sometimes millions — of followers. And unlike the web, where people with high reputation tend not to link to nefarious, spammy sites, many Twitter users automatically follow back anyone who comes their way.

“At the high end, politicians like Barack Obama’s campaign staff, or the UK Prime Minister’s office, these guys tend to follow back because they want to increase social engagement. But they’re following spammers back,” says Gummadi.

That top tier of users is topped by Yoko Ono, who boasts more than 775,000 people on her following list, but includes Gaga (currently following 139,000 people), Britney Spears (416,331).

But perhaps more dangerously, the study found that it was those in the middle of the scale who were causing the most damage by handing spammers credibility.

“The people with very low indegrees of just a few people were not responsible for spammers,” he says. “It was really people with 1,000 to 5,000 followers who were worst — that’s the range where there are a lot of people who follow back anybody. These are the social marketers.”

It’s a well-known grey hat tactic that following more people will increase your own follower count, and many people who try to profit will inflate their figures by following as many people as they can. But those individuals who are just trying to bump up their numbers are not just hurting their clients — they’re ruining search results for the rest of us.

“They want to increase their social capital and they have very similar incentives to spammers,” says Gummadi. “Spammers want to collect links, social marketers want influence. So they end up colluding with each other, even if not intentionally.”

You ain’t no follow back girl

Having identified the most dangerous Twitter users, Gummadi’s team — which included researchers from IIT Kharagpur, India, and UFOP in Brazil — have started exploring ways to reduce the impact of indiscriminate follow-backs.

That’s complicated, because it involves trying to change people’s behavior. But the researchers suggest it may be possible to implement some sort of “collusion rank” to sift out those people who are giving spammers oxygen and demote them. Just like Google’s pagerank can change depending on a site’s behavior, so too could their Twitter ranking if they link to too many spam accounts.

The team recognizes, however, that this may be tricky — because the line between helping a spammer and trying to be a good social network user can be very, very fine. After all, imagine how much time it would take Barack Obama’s team to sift each potential follower for spamminess — and the outcry if there was some sort of incorrect line drawn.

Whatever the case, it is clear to the research team that fighting Twitter spam needs to be about more than just shutting down accounts.

“We’re only beginning to understand spam in Twitter, and we’re sketching a solution,” says Gummadi. “But it does not lie in just banning spammers, because it doesn’t stop the social marketers.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Web startups: How to guard against security breaches
• The 2013 task management tools market
• How consumer media will change in 2013