Fortune Editors raceAhead: Can A.I. eliminate bias in banking?

Powering the People Team: HR Insights
Human capital is the prime source of value and competitive advantage for companies, putting the Chief People Officer and HR leaders at the center of the new world of business. Download Fortune’s latest collection of insights for HR Executives:

There's fresh evidence of racial bias in the financial system.

Researchers from the New York University Stern School of Business recently published a paper examining the racial disparities in small business lending that occurred during the COVID-19 pandemic. Through their examination of loans dispersed through the $806 billion Paycheck Protection Program, the researchers discovered that Black-owned businesses were more likely to receive loans from fintech companies than traditional banks, specifically the smaller firms.

One of the paper’s most noteworthy findings was that the analysis suggests “that automation might help reduce racial discrimination in the loan origination process,” the authors wrote. Furthermore, when small banks incorporated automation into divvying loans, the researchers found that the technology “more than doubled small banks’ propensity to lend to Black-owned businesses.”

Score one for artificial intelligence?

Considering there’s been much research indicating bias as a major issue plaguing A.I.-powered software like facial-recognition technology, the NYU study suggests that automation, when carefully used, doesn’t have to amplify societal biases. Of course, the devil is in the details, or in this case, the data.

Sabrina Howell, the paper’s lead author and assistant professor of finance at the NYU Stern School of Business, tells Fortune that the researchers’ analysis doesn’t attempt to explain why the small banks did not disperse loans to as many Black-owned businesses as their larger counterparts. And it should be noted that The Independent Community Bankers of America, a trade group that represents small banks, criticized the NYU study’s methodology and said that its members “outperformed the rest of the banking industry in serving minority-owned, women-owned and veteran-owned businesses.”

Methodology aside, Howell says she could only speculate as to why the data from the study showed the discrepancy between small banks and fintech companies and larger financial firms. One possible reason for the discrepancy was that the fintech lending systems could have relied on “simple automation” in which computers merely verified if a loan recipient’s driver’s license was fraudulent or not. Small banks, on the other hand, may have relied on humans to verify each recipient’s driver’s license, potentially polluting the ID-verifying process with those people’s personal prejudices against people of color.

“In the fintech setting, you have a computer looking at the picture of the driver's license and gauging whether that driver's license is fraudulent based on millions of driver's licenses that it has seen in the past,” Howell says. “And we can easily imagine how the computer could be less biased than the person who might see a Black face, and then impose a higher bar on the license in terms of ensuring that it's not fraudulent.”

This “simple automation” is not exactly cutting-edge deep learning, the kind of technology that powers today’s facial-recognition systems and excites businesses. In fact, its simplicity might give it an advantage in being less biased than deep learning software that attempts to deduce whether a person’s face belongs to a white male or a black female, among other tasks.

As Howell explains, the PPP loan distribution process is mainly determined through basic background checks like verifying people’s driver’s licenses and corporate payroll accounts. The process doesn’t take in account “credit underwriting” or other calculations that involve credit risk, which could contain hidden proxies for race.

Theoretically, an automation system that was trained on a database of driver’s licenses would excel at the task of verifying each new driver’s license it analyzes. It’s a simple enough task, and it was trained on a relatively simple type of data.

But if financial firms were to make that automated loan dispersion task a bit more complicated, let’s say by incorporating people’s credit risks or other data like zip codes into their calculus, there’s more opportunities for societal bias to creep into the process. The more data sources an automation system relies on to make its decisions, the more hidden societal biases may be lurking in the data.

So I wouldn’t say that this study proves that A.I. can be less biased than humans at making decisions. It depends on the complexity of the decision an A.I. system is trying to make, and the data sources that inform how the A.I. makes its decisions. And after all, it was human academics at NYU who ultimately discovered this potential racial bias issue in the PPP loan dispersion process.

For a more equitable society, humans need to be monitoring and studying every important decision, automated or not, that touches people’s lives.

 

Jonathan Vanian 
@JonathanVanian
jonathan.vanian@fortune.com

What you need to know about Latina Equal Pay Day On average, Latina workers earn 57 cents for every dollar earned by non-Hispanic white men. This year, October 21 was the day when Latina pay would catch up to that of their white male peers — which meant that it took Latinas 21 months to earn what white men earned in 12. As gender equity pay days go, it historically falls latest in the year. This year, the observance is particularly grim. Latina workers were more likely to become unemployed during Covid-19 than any other demographic, and more Latina adults slipped into poverty this year. According to the American Association of University Women, it would take at least 176 years for Latinas to achieve equal pay at this rate. And the individual wealth gap is profound Latina stands to lose about $1.1 million over a 40-year career.  More stats and resources below.
NBC News

The Dave Chappelle effect: Netflix employees stage walkout This recap does a great job explaining the ongoing uproar around both Chappelle’s final Netflix show—specifically, his remarks mocking the transgender community—and how it relates to ongoing tensions inside the company. It’s a problem that’s not going away. A Netflix employee group, Trans* Employee Resource Group and its supporters, has published a list of demands aimed at Netflix. Among the asks are for a hiring plan to recruit trans, nonbinary, and ethnically diverse leadership; a fund to support transgender creators; and explicit disclaimers on shows that feature transphobic or hateful content.
Fortune

What’s keeping managers up at night? New survey data from online leadership development company ExecOnline finds that leaders are reporting facing a new challenge to their own effectiveness going forward. The Great Resignation is taking a toll; managers say their top challenge is managing a workload with smaller, and increasingly burned-out teams. Some 52% report that employee fatigue and burn-out is their number one concern, followed closely by employee retention, and post-pandemic organizational culture. Bright spot alert: across the board, most self-reported perceived leadership capabilities took a dip this year, with one exception. Diversity leadership is the only capability that’s going up. Good news! But employees want more: Some “51 percent said their leaders will need to be able to demonstrate empathy to support workers’ needs; 44 percent want leaders to have skills that foster the concept of inclusivity; and 32 percent feel leaders will need skills that show an understanding of diversity and equity issues.”
ExecOnline

 

This edition of raceAhead was edited by Ashley Sylla.