A new startup called CrowdControl is launching Monday, and it aims to bring order to the world of crowdsourcing by using artificial intelligence to judge workers’ accuracy. It works with existing services such as Amazon Mechanical Turk, only it adds a quality control mechanism to help ensure jobs get done right. And although it’s suited to a variety of tasks, CrowdControl’s greatest promise might be in sentiment analysis.
Humans: Smart, but lazy?
Sentiment analysis already is becoming big business for companies such as IBM and SAS that are turning their predictive analytics engines on social media streams. But CrowdControl Founder and CEO Max Yankelevich says there are two big problems in the space right now. One is that current natural-language-processing technologies are better suited to identifying keywords than they are to deciphering true sentiment. The other is that humans, whose brains are inherently better at looking at text in context and working around abbreviations and poor grammar, have a tendency to underperform.
“The throughput and the price look good,” he said — you can have a thousand people complete a job in an hour for a penny a piece — “but the quality [can be] really atrocious.”
To cure that problem, CrowdControl contains more than 15,000 rules to determine how accurate workers are in completing their tasks. Those rules comprise much of the company’s secret sauce, but Yankelevich explained the methods for “adjudication,” the process of judging accuracy, at a high level. A big one is called “plurality,” which entails either assessing a worker’s answer in relation to everyone else’s answer on the same question, or giving the same question multiple times and looking for the same response. Another is “gold answers”: The tester continuously inserts questions to which it knows the answer and calculates how often the worker gets it right.
A final method Yankelevich noted is simply looking at statistics. If someone is routinely spending 30 seconds on a job that should take an hour, chances are he’s not working very hard to find the right answer.
Ensuring accuracy doesn’t just help CrowdControl customers, but it benefits workers as well. Yankelevich said CrowdControl has a complex reward system in place to give bonuses to workers who do quality work on a regular basis, and only pays at all if work is done accurately. A reward system is necessary, he added, because studies have shown that just offering a higher price per task has negligible effects on worker accuracy.
A platform years in the making, with Amazon’s help
Figuring out the ideal algorithms, adjudication methods and even the payment process took three years, but CrowdControl was lucky to have a partner in Amazon. Yankelevich said Amazon gave CrowdControl access to Mechanical Turk to perform its studies, through which the company learned a great deal. The workers who participate in crowdsourcing ventures are like a mini economy, he explained, and different people react differently to different practices, and can vary greatly in the work they take on and the time they spend on tasks.
The most interesting difference might be geographical. Yankelevich said that while American workers tend to carry out tasks during downtime at work or commercial breaks while watching television, Indian workers tend to organize into small communities and do these jobs full-time.
Amazon was willing to work with CrowdControl, because it means more business for Mechanical Turk, Yankelevich said. It presents the CrowdControl platform to its customers, who presumably will return to Mechanical Turk time and time again if they have quality experiences.
Aside from sentiment analysis, Yankelevich says CrowdControl has promise in all sorts of areas where data can be difficult to characterize. One of those is determining the accuracy of data on small businesses, which open, close and change addresses with relative frequency. When paired with a database for finding the source data on the web, such as Datafiniti, this use case actually could be quite valuable.
Most talk around big data focuses on the increasing intelligence of systems, but the continued rise of crowdsourcing startups such as CrowdControl and CrowdFlower suggests human know-how, fallible as it is, will still be important for a long time to come. Even if, like all things in the cloud computing era, brainpower is delivered as an on-demand service.
Feature image courtesy of Flickr user Spec-ta-cles.