The algorithms of Chat-GPT may exhibit signs of racial bias when evaluating resumes for job applications, according to an analysis conducted by Bloomberg.

As artificial intelligence (AI)  technologies like Chat-GPT have become increasingly integrated into various industries, concerns have been raised about biases in their decision-making processes, particularly in the area of recruitment.

According to the report by Davey Alba and Leon Yin of Bloomberg, before the adoption of AI in hiring processes, many companies relied on automated screening systems to assist hiring managers in assessing candidates. However, this approach was not without its flaws, as candidates often resorted to strategic keyword placements to bypass these systems. This led to the development of AI-powered recruitment tools such as Talenteria, Emi, Sapia AI, and LinkedIn’s Recruiter.

Chat-GPT shows the pros and cons of AI

While these AI services promise increased efficiency and streamlined workflows, concerns regarding bias persist. Bias in generative AI refers to the tendency of the AI model to produce outputs that reflect or perpetuate unfair stereotypes, prejudices, or imbalances present in the training data. This can result in generated content that discriminates against certain groups or reinforces societal inequalities.

Rodger Werkhoven, creative director at OpenAI, noted that biases in AI reflect biases in society, during an AI conference in Lagos.

To investigate the potential bias in Chat-GPT’s recruitment capabilities, Bloomberg conducted an experiment focusing on the impact of names on resume evaluation. Using a dataset of 800 demographically distinct names associated with various racial and gender identities, Bloomberg tested Chat-GPT’s ranking of fictitious resumes against real job listings from Fortune 500 companies.

“We replicated a simplified version of a hiring workflow by feeding a set of eight fictitious resumes into GPT – keeping all the qualifications equal, and only changing the fictitious names topping the resumes. Then we asked the tool to tell us who the best candidate was for a particular job listing. We asked GPT to evaluate these would-be candidates against four real job listings from Fortune 500 companies: an HR business partner, a senior software engineer, a retail manager and a financial analyst.”

In terms of methodology, Bloomberg crafted a dataset of 800 demographically distinct names, ensuring a balanced representation across gender and racial categories. This involved selecting 100 names each for males and females associated with Black, White, Hispanic, and Asian identities.

By identifying the 100 most popular first names and 20 most distinct last names unique to each demographic group, Bloomberg ensured a diverse pool of names. Each name was strategically paired with a last name to create distinct identities within each racial and gender category. 

The results revealed clear indications of name-based discrimination, “In our own experiment, we found clear signs of name-based discrimination. When asked to rank the resumes 1,000 times, GPT 3.5 favoured names from some demographics more often than others, to an extent that would fail benchmarks used to assess job discrimination against protected groups. We found at least one adversely impacted demographic group for every job listing we tested, except for retail workers ranked by GPT-4.”

In response to these findings, OpenAI emphasised that the results may not accurately reflect how customers utilise Chat-GPT, as businesses have the option to further fine-tune the AI model’s responses.