Here's why an AI expert says job recruiting sites promote employment discrimination (MSFT, FB)



  • Data science consultant Cathy O’Neil said discrimination in hiring starts with job advertising sites like LinkedIn,, Facebook, and ZipRecruiter.
  • In an interview with Business Insider, O’Neil said that by focusing on demographic data, algorithms used by job sites often prevent qualified people from ever seeing job openings.
  • O’Neil said algorithms are biased by definition, so rather than making them “colorblind,” companies need to continually assess whether they’re disproportionately filtering out candidates from marginalized groups.
  • That starts by “defining what it means to be qualified, then ignoring other things,” O’Neil said.
  • Visit Business Insider’s homepage for more stories.

Data science consultant Cathy O’Neil helps companies audit their algorithms for a living. And when it comes to how algorithms and artificial intelligence can enable bias in the job hiring process, she said the biggest issue isn’t even with the employers themselves.

A new Illinois law that aims to help job seekers understand how AI tools are used to evaluate them in video interviews recently resurfaced the debate over AI’s role in recruiting. But O’Neil believes the law tries to tackle bias too late in the process.

“The problem actually lies before the application comes in. The problem lies in the pipeline to match job seekers with jobs,” said O’Neil, founder and CEO of O’Neil Risk Consulting & Algorithmic Auditing.

That pipeline starts with sites like LinkedIn,, Facebook, and ZipRecruiter, where algorithms can play a significant role in determining which candidates see which job postings, filtering out those deemed unqualified.

“[Algorithms] are intended to discriminate, they’re trying to discriminate between someone who’s going to be good at this job versus someone who’s not going to be good at this job,” O’Neil said, adding that “the question is whether it’s legal or illegal discrimination.”

O’Neil has written extensively about the role algorithms play in fueling inequality both in her book, Weapons of Math Destruction, and on her blog In an interview with Business Insider, she talked about how bias shows up in the hiring process and what employers — as well as platforms like LinkedIn — should do to weed it out.

AI hiring tools are far from perfect

Federal laws, such as the Civil Rights Act of 1964 and the Americans with Disabilities Act, prohibit employment discrimination on the basis of categories like race, religion, gender, national origin, disability status, genetic information, and other categories.

While algorithms may speed up the process of narrowing the pool of job candidates, they’re often not great at finding the most qualified ones, and instead end up disproportionately filtering out people in those exact categories.

“I actually don’t think that most hiring algorithms are that meaningful,” O’Neil said, arguing that in many cases, they’re no better than “random number generators” at identifying qualified candidates.

In 2018, Amazon shut down a tool it had built to automate its hiring using artificial intelligence because it was biased against women. Researchers have also shown how AI tools that analyze video interviews are often biased against people with disabilities.

But it’s not just employers who have run into issues with biased AI hiring tools, it’s also a problem for the companies that help them find candidates.

A study in 2019 found that ads placed on Facebook for jobs with taxi companies — even when targeted at a broad audience — were seen by an audience that was 75% Black. And an investigation by ProPublica and The New York Times found that Facebook allowed employers to exclude users explicitly by age.

Bad data in, bad data out

There are several reasons why algorithms can end up discriminating against certain groups. One is the problem of “bad data in, bad data out.”

Programmers “train” an algorithm by showing it a massive set of historical data. In the case of a job site, they show it information about past candidates, telling it to look for patterns among people who ultimately got jobs, which it then uses to identify potential candidates with those same qualities. That can lead to problems, however, if the dataset is already skewed.

“If they’re trained on historical data, which they all are, then they’re going to be as racist and as sexist and as classist as human society is,” O’Neil said.

That’s exactly what happened in Amazon’s case. Since men had predominantly applied (and gotten) jobs in the past, the tool determined that men were preferable and penalized women’s résumés as a result.

Big data means biased noise

A second issue gets at why O’Neil believes biased job sites are particularly problematic: they factor in information that may have no bearing on a candidate’s ability to do a job, rather than focusing only on relevant details.

Sites like Facebook, LinkedIn, ZipRecruiter, and use a wide range of demographic information to train their algorithms. Those algorithms then help determine which job ads are shown to which candidates as well as which candidates appear in recruiters’ search results.

Companies collect as much data as possible because they think it will give them “a larger picture of the person,” O’Neil said, “but what it’s really doing is picking up all kinds of distracting and biased information.”

Even if that information isn’t explicitly about a candidate’s race or gender, it can still lead to racist or sexist results.

How companies — and job sites — can reduce bias

Some job sites have tried to combat this problem by not collecting or considering information that could introduce bias into their algorithm.

ZipRecruiter told Business Insider in a statement that its algorithms aren’t allowed to take into account “explicit markers (e.g., age, race, gender, etc.) or implicit markers (e.g., surname, specific residential address, etc.) of status within any protected class.” ZipRecruiter also prevents those models from differentiating between gender in titles or job postings.

LinkedIn, in a similar statement, said “we proactively detect potentially discriminatory language and review/block these jobs from being posted.” It also requires advertisers placing job ads to “certify that they will not use the ad to discriminate on the basis of gender or other protected characteristics.”

O’Neil said those steps don’t necessarily address the issue, however.

“It’s not going to be convincing for you to say ‘well, we don’t collect that information, so we’re colorblind,'” she said. “There is no way to get rid of proxies — everything is a proxy for race because race affects everything in our country.”

Instead of companies trying to make AI hiring tools “colorblind” by blocking explicitly or implicitly biased data points, O’Neil said they need to be more intentional about the information they do consider when filtering out job applicants.

“We should be defining what it means to be qualified and then ignoring other things,” she said.

One example she cited is the use of “blind auditions” by major orchestras, where they reduced gender bias by having auditioners play from behind a curtain. In deciding that “being qualified” really meant “sounding good,” they were able to structure the hiring process so it highlighted candidates’ qualifications. Just as importantly, they made it blind to other factors, like appearance, surname, or hometown.

“This is something that none of these AI hiring algorithms do,” O’Neil said.

But her ultimate concern isn’t even how these tools are designed, though that’s still important. O’Neil’s main point is that companies be more transparent about how they do things and what the end result is. That means continually testing their algorithms to see which candidates end up seeing jobs, and then correcting for any undesired bias.

LinkedIn should be “forced to prove that what they’re doing isn’t exacerbating inequality,’ O’Neil said.

LinkedIn has taken small steps in this direction, telling Business Insider “we ensure our Recruiter search delivers balanced gender representation, and we offer gender insights in reporting, so employers can understand dynamics in their Jobs and Sourcing funnels.” The company also prohibits advertisers from targeting job ads by age.

Gender and age are just two of the many dimensions along which people face discrimination, however. And while LinkedIn and ZipRecruiter both said they don’t tolerate discrimination on their platforms against any protected class, neither provided information about how they test the results of their algorithms to make sure that’s actually the case.

Facebook and did not immediately respond to questions about bias and the use of algorithms on their platforms.

SEE ALSO: Here are all the companies Airbnb has acquired to help it grow into a $31 billion business

Join the conversation about this story »

NOW WATCH: Apple just revealed its AirPods Pro for $249, which feature noise cancellation. Here’s everything that was wrong with the $159 pair of the wireless headphones.