Parakweet is a company that’s working to use natural language processing to cull through your tweets and make smart, targeted suggestions based on the data. On Friday, the company plans to announce the launch of two products. One is Bookvi.be, a consumer-oriented book recommendation engine, and TrendFinder For Movies, which is a social media dashboard primarily for entertainment companies to monitor conversations around movies. The latter is a paid product that provides the company with revenue, and the former is free for consumers.

“It’s a very hard problem we’ve tackled, which is accurately identifying sentiments,” CEO Ramesh Haridas said. “With 400 million tweets a day, there are 700,000 a day discussing movies, and if you tried text-matching techniques you’d come back with 40 million results. Many movies and books have very common titles, so you’d just drown in data.”

Both products use natural language processing to figure out how common a title is on Twitter, but also how a consumer is tweeting about a particular product, and they make recommendations based on those tweets. For instance, if I tweeted that a particular book is terrible and no one should ever read it, it would look ridiculous for a book recommendation engine to suggest that book to people. So Bookvi.be is structured to recognize the words I’m using in my tweet and know not to recommend that book. Users can choose to have a weekly email send to them with book suggestions, and they can type in their Twitter username to get book suggestions based on the people they follow.

“The bar on accuracy is very high,” Haridas said. “Especially if it’s sent via email, the precision needs to be intact.”

I’ve looked at a good number of social recommendation tools, and this one definitely stood out. For one, it was incredibly accurate — all the books it suggested were books I would actually read. But most importantly, it didn’t require me to create a new social network, or depend on friends for reviews, so you could get a lot of value from it right away. This is the obvious benefit of using someone else’s social graph, but Twitter seems perfectly suited to making content recommendations for things like books. Because unlike my Facebook friends, the people I follow on Twitter tend to accurately reflect my intellectual interests.

Of course, there are the obvious potential pitfalls of building a product around someone else’s platform, although Haridas said they support Facebook and are adding other platforms. But there’s a good deal of money to be made in accurately processing and understanding the words people are tweeting, as evidenced by Twitter’s acquisition of Lucky Sort this week, a similar company that also tries to figure out what people are talking about on social media.  As I’ve written before, as Twitter ramps up its advertising products it’s more important than ever for the company to be able to provide brands with more accurate ad targeting which hinges on the words people are tweeting and searching.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Examining the rise of crowd labor platforms in 2012
• The state of cross-platform media measurement

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Word that Lucky Sort is shutting down and that several of its team are joining Twitter’s revenue engineering department suggests this is exactly what the acquisition aims to accomplish.

As it stands, companies use Twitter as a way to track how people are talking about them and maybe, if they’re really advanced, do some sentiment analysis. If they’re willing to pay a third party, Datasift and Gnip are more than happy to broaden marketers’ views to encompass the entirety of Twitter’s data, both real-time and historical. What companies really can’t do, though, is run their own advanced analytics about topics straight from the Twitter platform.

One view of the Lucky Sort dashboard

The value proposition from such a product should be obvious at this point. Facebook, Google and Yahoo all collect a lot of data about how people are using their platforms and what topics are trending, and they all offer it up via a variety of products targeting marketing types and the public at large. If Twitter wants to be taken seriously as a venue for advertising budgets and a platform for measuring the pulse of the nation, people need to be able to ask questions of its data without relying on an intermediary or the occasional Twitter blog post.

As a journalist, I’d love to have access to this type of tool to track trending topics in real time and spot possible stories as they’re happening. The appeal to marketers should be obvious. As IBM’s Erick Brethenoux told me recently, “[Marketers] talk a good game about social data. Very few actually leverage it effectively today.”

At Twitter, though, data is a slightly different beast than at other web companies. Twitter’s value lies largely in real-time data — topics can be peak, crest and all but vanish within a 48-hour window. This situation has hampered some of Twitter’s efforts to surface optimal search results, and it has spurred the decision to buy companies such as Backtype (for its streaming-processing Storm technology) and parallel-processing startup Ubalo.

The latter move, which happened last week, should help Twitter’s development team create new features without worrying about the intricacies of making them run — and run fast — across a cluster of machines. (You can learn a lot more about how companies such as Google, Facebook and Box are rethinking infrastructure to handle their unique data needs at our Structure conference next month in San Francisco.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Finding the Value in Social Media Data
• Listening platforms: finding the value in social media data

When you’re in Atlanta, as I was during a recent trip, the obvious place to start talking about orange juice and data is with Coca-Cola. Foreman can tell you all about how the beverage giant — whose headquarters tower over the city just a just a mile away from MailChimp’s office — uses advanced algorithms and giant vats of different juices to ensure the proper flavor of its Simply Orange line of orange juice. However, it’s something else Coca-Cola is doing that inspired the way Foreman thinks about data and that’s helping MailChimp re-imagine what it means to engage with fans, readers and customer through their inboxes.

Anyone familiar with how large web companies came to pioneer the practice of what we now call “big data” should appreciate the analogy. Coca-Cola, which also owns Minute Maid, produces a lot of excess pulp when it makes orange juice. For decades, presumably, it had just been throwing that pulp away, but in 2006 it decided to make use of it by launching a new product called Minute Maid Pulpy. Sold primarily in Asian countries, Pulpy has become a billion-dollar business for Coca-Cola.

Once MailChimp is done with its primary business of sending emails, it has a lot of pulp of its own in the form of data. And rather than just ignoring it or writing up some cute blog posts (which he also does), Foreman and his bosses want to turn that data into revenue.

First things first: Making better orange juice

Neil Bainton

Actually, though, MailChimp first brought in Foreman in 2011 to help the company improve its core business of letting users build and send their emails. MailChimp’s culture was built around many things, COO Neil Bainton told me, but data wasn’t one of them. It had “various fits and starts” through the years trying to work data into its business model, and each step just added more complexity.

The challenges were technological as well as cultural, but Foreman had a plan, of which focus was a key aspect. Keeping a tight focus meant Foreman and his lone-developer sidekick could build what they needed to in a short timeframe. It also meant the company didn’t have to worry about some massive overnight transformation into a data-obsessed company like Google.

John Foreman

“[They] don’t need to be afraid the entire culture is gonna fall down if we bring in this weird math guy,” he joked.

Foreman’s first project — deploying artificial intelligence models that would automatically detect spammy email lists from MailChimp’s users – is actually critical to the way MailChimp operates, though. It was up and running in production within a year, after a technologically challenging effort of merging separate database instances for each customer into a single environment that would let MailChimp run complex analyses across its customer base.

It’s such an important project, Foreman explained, because internet service and email providers keep reputation scores on the IP addresses that send email through their systems. Because MailChimp serves as the email engine for its millions of users, sending too many messages that get flagged as spam and lower MailChimp’s reputation will have a negative impact on everyone. The company used to deal with spam manually, and only after recipients began complaining about the messages they received.

“It used to be before we had that AI model in place that everyone had a crappier experience,” Foreman said.

Say goodbye to those ’90s fans, Pearl Jam

Source: MailChimp

Now, however, MailChimp knows some of the telltale signs of spam for which it should be on the lookout. If too high a percentage of email addresses on a given list are also available via publicly available lists or those you can buy on sketchy corners of the internet, it’s probably spam. Too many old and far-more-likely-to-be-dead Earthlink or Compuserve addresses, or letters within one keystroke of each other as if someone just mashed the keyboard? Probably spam.

Thankfully, though, about 98 percent of the spam that MailChimp identifies is what Foreman calls “ignorant” — that is, people or companies that just don’t know the laws or best practices around sending emails. But ignorance doesn’t mean MailChimp relaxes its rules. Recently, it even flagged Pearl Jam for spammy practices because the band was trying to reconnect with old fans whose email addresses read like a who’s who list of 1990s email providers.

Having such a high percentage of ignorant spam actually has a positive effect on the company’s overall goal of monetizing its vast data repositories. Because the AI model automates what used to be a manual process, and because most innocent spammers will fall in line quickly once they’re notified (as opposed to nefarious spammers who constantly try to outsmart the system), MailChimp can pretty much set the model loose, forget about it and get to work on new efforts, Foreman said.

Now, about that pulp

Spam under control, MailChimp can focus its efforts on actually building new products with data, just like Coca-Cola did with that extra pulp. One of its first orders of business is figuring out how to help customers get to know better the people to whom they’re sending their newsletters.

With this in mind, the company built a service called Wavelength that shows customers other newsletters that are similar to theirs. But the system that powers Wavelength also stores pretty much every interaction that every email address in the company’s database has with the newsletters they’re sent. That means what emails they open and when they open them, what links they click and when they click them, and what other newsletters they’re subscribed to. MailChimp also has a feature called Ecommerce360 that lets customers track clicks right through to conversions (marketing speak for someone actually buying something).

The company has been playing around with this data to identify clusters of users based on their behaviors and their interests — some of which Foreman has detailed on the company’s blog — and now it wants to roll it out to customers via a product MailChimp is calling ChimpQuery. Built atop Google’s BigQuery analytics service, ChimpQuery will let customers start doing this type of clustering and segmentation on their own, while saving MailChimp the troubles of hosting that infrastructure itself. (You can play with a monstrous, interactive graph of the entire MailChimp subscriber list here.)

If you sell knitting supplies and you find out there’s a big cluster of people on your mailing list who also are interested in wedding planning and custom jewelry, there might be an opportunity to create your content with these interests in mind or even to partner with companies in those spaces.

A sample cluster of subscribers.

Another topic that has been on Foreman’s mind lately is what he calls “frequency elasticity of engagement.” He’s done research suggesting that blasting the heck out of your email list might actually have detrimental effects in the long term (regardless of how the Obama campaign successfully exploited this strategy) but noted that engagement also has a lot to do with content and a particular company’s given user list. MailChimp’s data could help customers figure out the ideal schedule for emailing their subscribers.

For example, Birchbox has really high engagement because people love the service and have to open their emails to find out what goodies they’re receiving. Emails from a company like Papa John’s, on the other hand, might sit in someone’s inbox essentially as spam until they want to order a pizza and go searching for a coupon. Everyone has to figure out what pace and engagement metrics work for them.

Reining expectations back in

However, now that management is fully sold on the power of data, Foreman sometimes finds himself managing expectations rather than just pitching his ideas. COO Bainton, for example, is adamant that MailChimp start aiding its publishing-industry customers by using techniques such as natural-language processing and semantic analysis to help them personalize emails based on readers stated and unstated interests (that is, what boxes they check when they sign up and what stuff they actually click on).

Foreman, well, he’s pretty sure that’s too big a challenge for MailChimp to tackle considering how many publishing customers it has. MailChimp would have to understand all those customers’ industries to some degree (open source tools tend to highlight technically but not situationally relevant relationships, he said, and don’t always understand things like sarcasm) and probably the different languages they publish in, as well. Rather than understand content, he’d rather focus personalization efforts around how users are connected.

The company also needs to balance its ambitions with what’s legally and socially acceptable. The creep factor might be more important than what’s legal when it comes to email marketing. MailChimp determines the legality of everything it does before rolling it out, Foreman explained, but in era of “post-modern spam” where legitimacy is in the eye of the recipient and where some people use their “spam” button as a proxy for unsubscribing, companies must be careful not to offend.

“The more we can tell you about that list without getting creepy is really useful,” Bainton said. However, he added, ”I think expectation is more important than law.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

In an article posted to Facebook’s engineering blog on Monday, research scientist Maxime Boucher and Xiao Li, engineering manager on the natural-language team in Graph Search, provide detailed information on the ways in which Graph Search calls on natural-language processing to guess what users want.

• Graph Search breaks down search strings into multiple components that serve as commands with which the system can query the database. For instance, “my friends who live in San Francisco” would be run like this: pulling up the user, grabbing that person’s list of friends, calling on the filter for people who currently live in a place, and filtering out only those friends who have San Francisco in that field. Graph Search considers that search query “intersect(friends(me), residents(12345)).” And if that’s exactly what the user had in mind, that query gets converted into language for the Unicorn search engine to chew on.Search terms sometimes include words Graph Search has no use for. At other times, words for guiding queries are missing. And users might plug in terms in the wrong order. Say a user types in “friends San Francisco.” Graph Search might offer “my friends who live in San Francisco” as a good option. If it sees “San Francisco friends,” it could respond with “my friends who live in San Francisco,” which is more in accord with the correct sequence for a query.
• Graph Search analyzes words users enter to look for possible entities that users are referring to in the database, across more than 20 entity categories, such as cities, employers and schools. Using statistics for the entity categories, the tool identifies sequences of words that could be more applicable for certain entities than others. If “san” precedes “francisco,” the user likely is referring to a city, not a person.
• The system recognizes slang, nicknames for places, misspellings, the many ways of expressing particular types of data and other peculiarities that users type into the search box and swaps out each of those for terms that actually exist in the database. That means, for example, that subject-verb agreement isn’t necessary for the system to serve up query options that might lead to what users want to see. And words such as “besties” get interpreted as “friends.”

  Graph Search is visible on one of the most popular social networks in the world and therefore needs to be satisfying for its users. As Boucher and Li write, “The challenge for the team was to make sure that any reasonable user input produces plausible suggestions using Graph Search. To achieve that goal, the team leveraged a number of linguistic resources for conducting lexical analysis on an input query before matching it against terminal rules in the grammar.”

Graph Search still has a long to-do list for engineers to address. One of the biggest challenges is to construct and deploy a language-agnostic Graph Search system, so Facebook users all over the world will be able to do what English speakers can do with the tool. It will be difficult to produce a tool that can adjust for unusual spellings, handle incorrect grammar and otherwise optimize search strings entered in any language. “In Russian, there’s so many inflections around words and a lot of language-specific things we haven’t encountered in English,” Li told me in an interview on Friday. Engineers are now looking at different ways to make the tool available for other languages, Li said. One option? A whole lot of drop-down menus.

While there is still work to do in letting more people try Graph Search, it’s clear that the simple interface for navigating hundreds of millions of objects required engineers to produce a bunch of systems and models. It’s no Google, Siri or DataPop, but, because it contains elements tailored to the data set at hand and common use cases, and because it’s getting better over time, Graph Search is worth keeping an eye on.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Social 2013: The enterprise strikes back
• Examining the rise of crowd labor platforms in 2012

Most virtual assistants powered by natural language processing are taught to do specific tasks very well but tend to come up short when given unfamiliar assignments. For instance, Siri excels at jobs like making calendar appointments and dictating text messages but can be confounded by more general requests for information, usually resorting to simple web searches.

Sherpa CEO Xabi Uribe-Etxebarria said he set out to create a natural language platform that had a much greater scope of understanding, which could easily be applied to new tasks without “training” the app to perform them. He also wanted to create a language-independent platform, one that understood meaning and intent independent of a language’s vocabulary or syntax.

To that end, Uribe-Etxebarria and his machine-learning team developed a sort of meta-language, encompassing 250,000 semantic concepts accompanied by 5,000 rules used to order those concepts. Sherpa uses off-the-shelf speech recognition services (right now it uses Google’s speech API) to translate commands into its meta-language, and then it parses meaning and intent from the resulting string of concepts.

The result is a flexible virtual assistant that can easily be applied to new tasks, Uribe-Etxebarria said. Sherpa’s repertoire is constantly growing as it hooks into new apps and information sources. For instance, Sherpa has struck a deal with PayPal, allowing the app to make payments via voice command. It taps into Twitter’s API, letting users navigate their twitter feeds — toggling between mentions, direct messages and home stream views — through voice prompts. For general information requests, Sherpa has developed a nifty information card format, which aggregates information from a variety sources ranging from LinkedIn profiles to Wikipedia entries.

“We’ve gone beyond Siri in many cases,” Uribe-Etxebarria said. And given the flexibility of its technology, he added, Sherpa can continue to add new services and functions at a much quicker space than its competitors.

Still, Sherpa is entering an increasingly crowded space. New virtual assistants are popping up left and right, some very focused on specific tasks like Incredible Labs’ Donna, while some like Nuance’s Dragon technologies are spanning devices, trying to create a single virtual assistant for all things. And of course, Google and Apple are building their speech technologies directly into their phone operating systems – it’s hard to argue with the convenience of that big fat Siri button.

Sherpa got off the ground in Bilbao, Spain, but it now has offices in Redwood City, Calif. It has raised $1.6 million in angel funding.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• NewNet Q4: Platform mania and social commerce shakeout
• Analyzing the wearable computing market

SkyPhrase is a very early-phase company that, according to its web site, has “made breakthroughs in algorithms that enable computers to understand more complex language with greater precision than has ever been possible.” The goal is to improve search functionality but also to give developers a new, easy way to incorporate natural language processing into their apps. The company was founded by Rensselaer Polytechnic Institute Professor Nick Cassimatis.

In January, MIT Technology Review reporter Rachel Metz covered the company and actually reviewed an early version of the technology as applied to searching through tweets and emails. It wasn’t yet trained to do what she wanted with tweets but, she wrote, did a “decent” job searching through emails. Part of what makes it work appears to be its ability to understand conjunctions, even if it doesn’t yet have semantic capabilities: “I could search for, say, ‘e-mails from Bob Loblaw in December and January about recipes with a PDF,’ or ‘e-mails from Bob Loblaw or Tobias Funke about cookies in December,’” Metz explained.

Nanostraws in a cell

Breakout Labs’ other new investment, Stealth Biosciences, is a team of Stanford professors, executives and entrepreneurs that has invented a way to get materials into and out of individual cells and to monitor their activity via electric probe. Called Nanostraws and Stealth Electrodes, respectively, the companies two techniques do just what they sound like they do: NanoStraws let doctors inject or extract material from cells in the aims of advancing research and delivering personalized medicine, while the electrodes “automate long-term intracellular electrical recordings of neurons and heart cells.”

Stealth Biosciences, in particular, seems like a heady endeavor, but that’s exactly what Breakout Labs is all about. Launched in 2011, the organization aims to fund projects too early in their lives to attract traditional venture capital. Those funded aren’t giving up large equity stakes in their companies, but are expected to provide a “modest portion” of their revenues back into the program to fund the next generation of Breakout Labs investments. Other investments thus far include Modern meadow — a company trying to create artificial meat using 3-D printers — and AVEtec, a Canadian startup trying to harness the power of tornadoes for good.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How emerging technologies will influence collaboration
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

The San Francisco company doesn’t want to reveal how everything works yet. But previous work from Idibon CEO Rob Munro provides hints of what’s possible. In his 2012 Stanford Ph.D. dissertation, entitled “Processing Short Message Communications in Low-Resource Languages,” Munro explained that it was possible to build natural-language-processing systems that could handle many variations in word spelling in text messages and tweets in Chichewa, Haitian Krèyol and Urdu when classifying, even when the systems had little time to train and get better and no previous familiarity with the languages. In the case of the texts in Haitian Krèyol that were sent following the January 2010 earthquake in Haiti, prioritizing helped quickly sift out the genuine emergencies. The question is whether a tool could be developed to pick up patterns in text in any language. Such a system, if combined with a powerful translation tool, could be deployed for a wide variety of applications, from sentiment analysis to intelligence gathering.

Rather than leave machines to bear the burden of figuring out what people mean when they communicate in obscure languages, Idibon wants humans to play a role, such as verifying that data is correct. That sort of work could be crowdsourced. “Machines are never going to be 100 percent accurate,” Munro said. The idea of bringing together humans and algorithms to solve problems has come up in other applications, and several came up in on-stage conversation at GigaOM’s Structure:Data 2013 conference in New York last month.

How could enterprises use Idibon? Half a dozen customers are using the beta version of the software in different ways. One is relying on Idibon to run a medical question-and-answer system that can spit out an answer or possible answers. And “a sales organization” is using Idibon to rifle through news articles, blogs and other documents to document relationships among people and organizations and point to past acquisitions, Munro said. It’s also possible for Idibon to process information from multiple languages to serve up data for business-intelligence applications.

For now, Idibon is “just a simple API service,” Munro said. Some direct integration of the Idibon data is happening, too. The software takes in unstructured data — from tweets, instant messages, emails and so on — processes it and responds with structured data, he said. Ultimately, though, “we want to become the leading organization for scalable cloud-based natural-language processing,” Munro said.

English comprises a small fraction of all communication — roughly 375 million people call English their first language, out of more than 7 billion people in the world — and that’s why a tool with more universal linguistic powers sounds so appealing. While not many enterprises might be looking to capture data in little-known languages now, it could become essential in the coming years. If Idibon can come out with a product soon, it could be the beneficiary of a sort of international arms race for truly global understanding.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• What’s driving the next phase of the e-commerce evolution
• Siri: Say hello to the coming “invisible interface”

As individuals, doctors probably don’t see enough of any given adverse reaction to notice patterns emerging. But as a collection, their notes on patients’ medical records can provide valuable insights, as a group of Stanford researchers recently discovered. Using “18 years of patient data from 1.8 million patients [consisting of] 19 million encounters, 35 million coded ICD-9 diagnoses, and >11 million unstructured clinical notes,” the team was able to accurately identify interactions by analyzing the free-form text that doctors had entered about patients’ symptoms, conditions and prescription regimens.

A key aspect to being able to predict adverse interactions is understanding the relationships among the different sets of terminologies used in different medical fields. It’s a lot easier to spot patterns across hospitals or even an individual patients’ records when you know that a radiologist writing X is the same, or related to, an oncologist writing Y. We covered an earlier collaboration between the study’s leader, Nigam Shah, and medical-data startup Apixio around this very topic in 2011.

How Shah’s team developed its patient-feature matrix

Shah and his team hope their work can complement the current process for tracking drug reactions, the FDA’s Adverse Event Reporting System. Whereas that system requires doctors and patients to manually alert the FDA of potential adverse side effects, their method could highlight potential problems that no one noticed or took the time to report. I’d consider this similar to some early research by social medical sites such as PatientsLikeMe, whose users are producing lots of data about their conditions, drugs, dosages and side effects that could produce correlations ripe for controlled experiments.

A press release announcing the study’s publication highlights some of its future promise and current limitations:

“[T]he research team is working on refinements that will cull even more useful information from clinical notes, such as reports of reactions caused by drug combinations, the use of medications typically prescribed for one condition but found effective for treatment of a different health problem, or finding medical profiles of patients that fit a certain scenario. …

One downside is that most electronic health record systems are set up for patient care, not patient research, Goodman noted. In this study, the researchers mined a data system created for this kind of research, which isn’t widely available. The researchers used the Stanford Translational Research Integrated Database Environment, known as STRIDE.”

This is just one of many ways in which researchers are experimenting with big data concepts to help medical professionals make sense of more data than they could possibly analyze on their own. Other examples we’ve covered recently include an artificial intelligence model for prescribing safe, cost-effective treatments, the application of Google PageRank-like algorithms to map the spread of cancer cells throughout the body, and the use of graph data structures to organize highly complex sequencing data.

Feature image courtesy of Shutterstock user Maksym Dykha.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Why the next front in big data might be psychological
• Connected world: the consumer technology revolution

Fairfax, Va.-based DataRPM on Tuesday announced that it has raised $250,000 from angel investors and rolled out a new feature for its business-intelligence Software as a Service (SaaS) called Instant Answers. The feature uses natural-language processing to figure out what users want to see, based on typed or spoken queries, and displays the visualization that the software thinks is the best fit. Users can filter and comment on the results.

While methods and purposes vary, the idea of making software or a site respond to limited user input isn’t new. The approach reminds me of Facebook’s Graph Search, which rapidly delivers several options for search results based on likes, friends and other user information. Software from BeyondCore also comes to mind, as it quickly displays graphs and audibly speaks out its findings to show the biggest drivers of, say, revenue. BeyondCore CEO Arijit Sengupta took a few minutes of stage time at GigaOM’s Structure:Data conference in New York last month to show off the software.

More natural-language processing and machine-learning technology could make DataRPM’s Instant Answers tool a better choice in a crowded market. Perhaps the SaaS could keep tabs on which data users call up and how users might modify their searches if they don’t get the data or visualizations they want the first time around. Later, it could predict what users want. The original Instant Answers is nevertheless a good start.

Feature image courtesy of Shutterstock user anaken2012.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How HR can make the case for workforce analytics
• The fourth quarter of 2012 in cloud
• Big data 2013: key trends and companies to watch