Blog Post

Online job boards don’t work — how big data can fix the problem

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

In September, there were 3.5 million unfilled job openings, according to the Bureau of Labor Statistics. Yet only roughly three percent of those positions ended up being filled during that month. While that may surprise some, it’s sadly business as usual for those of us in the recruiting space. Some have argued that there is just not enough qualified talent out there to fill these positions, but in the course of my career I’ve come across hundreds of smart, capable candidates unable to even get their feet in an interviewer’s office.

It would be convenient to fault the candidate for not conducting a proper, modern job search. But not me – I blame it on technology.

Resume spam

Human resource experts will tell you that for any one job posting, they receive hundreds, if not thousands, of resumes. As a result, they’re able to spend just six seconds evaluating each resume, typically scanning only the candidate’s education and their last job. Such a brief perusal means candidates who are a good fit for a company’s culture or who can bring a different and much-needed fresh outlook can easily fall through the cracks. There is simply not enough time in the day to wade through all the resumes that flood the inbox.

The resume deluge started when businesses began to rely on online job boards to find candidates. With a few clicks of the mouse, job seekers were suddenly able to upload their resumes and cover letters and then apply to dozens of jobs at a time – sometimes even more than once. This “spray and pray” strategy has completely clogged up the system. Just last month, Facebook announced a new jobs app that, at launch, boasted it already offered users access to 1.7 million job opportunities. You can imagine how that will only compound the problem even further.

The adoption of digital databases to conduct the initial winnowing has done little to stem the tide. Job seekers know now to litter their resumes with the right key words to game the automated systems and grab recruiting eyeballs. It’s no surprise then that, on average, 66 percent of applicants for a given job meet the minimum qualifications. The current system is set up to deliver the wrong people to prospective employers. This bottleneck has meant a huge financial drain on businesses. Millions of man-hours are wasted sifting through inappropriate or unwanted resumes, costing a company on average $5,504 and up to six months per hire. Conversely, job seekers now spend a median of 19 months looking for the next position.

A technological solution

So how can we make labor markets move more efficiently and effectively, for the benefit of all? The solution is a technological one – big data. (Full disclosure: My company specializes in software that uses big data to connect job seekers with opportunities. We’re joined by companies like and TalentBin, both of which are trying to make job searching more simple and intuitive with the help of big data.)

Big data can help recruiters find the right candidates to interview by cutting through the noise created by the chaos of the current job search process. Big data tools such as modern distributed file systems and map/reduce/clustering techniques make large data sets accessible and more easily analyzed. Five years ago this simply wasn’t economically possible. Back then, it was cost prohibitive to purchase enough computer servers to make these calculations, and further, vendors were constrained by the physical size limits of data centers.

Now, vendors can process billions of transactions in the cloud at a fraction of the cost of local servers. Thus, employment-related data, regardless of size, can be leveraged to find subtle patterns reflecting a current candidate’s qualifications.

Another added benefit is the reduction of human bias. All human recruiters, regardless of background, bring a bias to the resume evaluation – it’s human nature. Big data algorithms, though, are blind to names on resumes that may surface a job applicant’s race, ethnicity, religion or gender. Machine-learning algorithms, utilizing large implicit- and explicit-feedback datasets, can be trained to simulate decisions made by professional recruiters and thus reduce or eliminate evaluator bias.

As a result, big data allows for a multi-faceted statistical approach to the filtering process at the first level, and thus helps identify better candidates from a deeper pool.

To be clear, this isn’t to say big data can replace a job interview. The interview is about culture fit, body language, eye contact, voice intonation, and the discussion of a general fit between the job seeker and the position. Technology can’t solve that on its own. Technology can, however, make sure that candidates you bring in for an interview are the best qualified, right from the first screening.

The job search is not rocket science. But the application of data science and big data can streamline the process so interviews are filled with the best fit candidates more quickly, efficiently, and at a lower cost. Big data can revolutionize the labor space.

Steve Goodman is CEO of  

Photo courtesy of Everett Collection/

7 Responses to “Online job boards don’t work — how big data can fix the problem”

  1. Jojari Jobs

    Moving away from keyword searching is the key to the future. Boolean searching just does not work. There are so many businesses claiming “semantic search”, but most of these technologies are far from it. I have a lot of respect for what Bright are doing, they’re applying real science, not slapping “semantic” on the same old boolean search with a few tweaks. This field will be radically different in 5 years, no doubt about that…

  2. One problem with automated filtering of resumes is the lack of “intelligence” in the algorithms. For instance many job descriptions state something like “Bachelor degree or equivalent experience” but how to you write a program to evaluate that equivalency? I ran into this after being laid off in 2009 and couldn’t get past the filters. I don’t have a BS degree but do have 12 plus years of computer software testing and programming experience. According to many job postings I was qualified but was rejected by the filters. I skirted the issue by signing up for classes at a local collage and putting on my resume “Bachelor of Science (in progress)” on my resume and immediately started getting more responses. Sometimes a job seeker needs human eyes to see the potential they hold for a company. Using even bigger data makes that even tougher to make happen.

    • Filip De Geijter

      Wtih the curent state of technology that is possible obviously the application should be capable of allowing you to get through mandatory fields in the UI. We at Actonomy have created an intelligent layer based on deep semantics that can handle this. Applications to come! But yes if applications are not designed to handle big data or semantic analyses, it will not work. Filip De Geijter

  3. Jeff Putz

    The problem I have, is the reverse problem. I get spammed daily from outsourced call centers hired by staffing firms to hit anything with a matching keyword, or even a match on the industry. I get calls for COBOL when I’m a .NET guy. I would argue that a significant part of the problem is the third-party agencies. They don’t add any value. I’ve never had a gig through one, and yet, they make the most noise.

  4. Steve,

    Do away with the HR departments, replace them with data scientists who serve the company. Operate as the big corporation’s candidate locator. Stop looking for all your revenues from corporate HR buyers (and their tiny budgets) and design a system that works as efficient as Amazon for job seekers. Then you’ll have something. You’ll never ever get through to the present generation HR. So go around them by getting buy-in from the job seeker and not the head of HR. When you have a system that truly services both parties, you’ll be as powerful as Google is with local ad word advertising.

  5. Kent Ong

    You are correct, Steve. Big Data can solve the problem so effectively and fast. I am looking for one to come to Malaysia to work together as well. Are you interested in Malaysian market?