Fighting Racism in Health Care Outreach & Algorithms

Webinar

Featuring

Chicago Booth logo

Description

Bias abounds in health care, whether it’s intended or not. Even in health care marketing and consumer engagement, common targeting methods unintentionally exclude race, gender, and sexual minorities. For example, methods that segment by zip codes and income, and even methods using EMR data, common algorithms, or machine learning frequently perpetuate bias.

Dr. Emily Bembeneck, Sr. Associate Director for UC Booth Applied AI, will share the story on bias in widespread approaches in health care and their playbook for health care leaders to fight bias in their own organizations.

Fighting bias doesn’t stop with algorithms. Rebecca Nissan from ideas42 will discuss the behavioral sciences involved, and Chris Hemphill and Ling Hong from Actium Health will discuss coordinating data science with patient engagement leaders to make these efforts real.

Learn about issues impacting health equity today:

  • Bias in health care, health care algorithms, and its impact
  • You can’t manage what you can’t measure – measuring bias in outreach and populations
  • The crossroads – will algorithms and AI approaches perpetuate or reduce bias?
  • Regulatory efforts around how we’ll use algorithms and AI
Dr. Emily Bembeneck

Dr. Emily Bembeneck

Sr. Associate Director
UC Booth Applied AI

Chicago Booth
Becca Nissan

Becca Nissan

Sr. Associate
ideas42

ideas42
Chris Hemphill

Chris Hemphill

VP, Applied AI & Growth
Actium Health

Actium Health
1

Transcript


Chris Hemphill:
Hello, healthcare. Hello, everybody watching us on Zoom, or if you’re watching this later on YouTube, hello. Just wanted to say that we appreciate you coming together for this topic. It’s a very important, it’s a very heavy topic because there’s this proliferation of algorithms, not just with the artificial intelligence, but with algorithms in general. We’re going to explore deeper into algorithms, what they are and how they might be perpetuating bias in a lot of instances in healthcare and what we can actually do about that bias. There is an all-star cast to help us with this discussion today. We’re excited to bring these folks together. They’re folks that we’ve worked with at SymphonyRM or they’re part of SymphonyRM, to help address this problem that we’re passionate about.

Chris Hemphill:
So I hope you’ve come here with questions or with comments, with stories to tell. The reason that we’ve set it up this way is so that … this is a rare opportunity to talk to people who are involved in behavioral science, involved in applied artificial intelligence, involved in really tackling this problem. So we’re wanting to do this to open up this conversation and open up thinking around the challenges and issues that algorithms might be bringing into healthcare. So kicking that off, Dr. Emily Bembeneck, she’s senior associate director at the Chicago School of Business booth. So Chicago Booth Center for Applied Artificial Intelligence. And that work actually has her at the forefront, having discussions, maintaining industry relations, as we’re moving towards understanding more about AI, the biases that it can bring in and solving challenges around that. So she heads up the algorithm bias initiative. So with that quick intro, Emily.

Dr. Emily Bembeneck:
Hi, I’m very happy to be here. Thank you, Chris. Like Chris said, I lead the Center for Applied AI at Chicago Booth. We’ll tell you a little bit more about that later, but for now, thank you so much for having us and we’re really excited to share in this conversation and help make a difference.

Chris Hemphill:
Excellent. Appreciate that, Emily, and the work that you’re doing to get this out into the industry. Along with that, Rebecca Nissan, who works along with Emily at Chicago Booth and also works for a behavioral science nonprofit called ideas42. So she brings a lot of machine learning expertise to this conversation, as well as behavioral science expertise on various steps and behaviors and things that we should be launching and focusing on to addressing these issues. So quick intro, Becca.

Rebecca Nissan:
Hi, everyone. Like Chris was saying, my work is at the intersection of behavioral science and data science. I work with partners in healthcare, both through my work at ideas42 and the Center for Applied AI, specifically on this topic and reducing bias in algorithms. So really excited to be here today.

Chris Hemphill:
Appreciate that, and thank you for coming on and sharing in some of this work that you’ve done. Finally, Ling Hong, at our organization plays a key leadership role in the data ethics efforts that we have now. She’s built this data science pipeline that addresses ethical issues and bias, and it’s extremely exciting to have her here because there’s a lot of talk about the problems, about the challenges around AI and algorithms, and Ling is doing a lot of work on the response and the solution. So Ling, quick intro from you.

Ling Hong:
Hi, everyone. I’m Ling, I’m a data scientist at SymphonyRM. So a huge part of my job is to exam biasing algorithms and develop solutions to alleviate the bias. So I’m very excited to be part of the conversation today.

Chris Hemphill:
Excellent. Fantastic. And with that, we’ll just go and get going. The way I see it is a lot of the conversation about algorithms and data science and all that has been thoughts, the data never lies or people completely relying and thinking that these algorithmic approaches are completely fair and unbiased. We want to dig into what algorithmic bias actually is and what causes it, and then talk about how to combat that bias within healthcare approaches. We’ll conclude the presentation portion with a case study on responsible AI. That’s probably a name that you’ll start seeing a lot more of. The Gartner group has done an assessment on where responsible AI is in their Gartner Hype Cycle. There’s going to be a lot of conversation as the market learns more and as publications start putting more stuff out about this responsible AI, is going to be a big part of our agendas and thoughts moving forward.

Chris Hemphill:
Then we want to open it up because I’m aware that this is a new subject for a lot of folks, and I want everybody to feel comfortable asking questions, sharing thoughts and things like that, drop them down in the chat or in the Q&A section. We’re going to have a big chunk at the end, about 20 minutes devoted to getting to those questions live. But just digging in, we’re looking at racism in a little bit of a different way. So if we look at the definition of racism in the dictionary, it defines it as the belief that races are different. Some of those differences lead to some racist being inferior or superior. But the challenge with using this belief-oriented way to define racism is that it can’t really be proven. If somebody commits an act that disadvantages races or disadvantages a group in a certain way, whether or not they believed it was racist, whether or not they believed it was an issue, that really can’t be proven.

Chris Hemphill:
So when we’re looking at this in terms of what our business impact is or what the impact of particular algorithms is, rather at the beliefs, we’re looking at the outputs. So actions that disproportionately harm or neglect people based on their race. We’ll start exploring those in a couple of minutes. But the basic idea here is that with the concept of racism, sexism and any sort of this bias or discrimination, we can perpetuate those things, even if we disagree with those tenants. I believe that the majority of people who are releasing algorithms and doing this work, I don’t believe that the Ku Klux Klan has a lot of data scientists out there. So the focus here is how do we get away from practices that might be disadvantaging racist without us even knowing it?

Chris Hemphill:
And to start on what some of those practices are, I’m going to jump a little bit outside of healthcare and just talk about these processes that affect different industries. One example, in criminal justice and sentencing, in this example, well, algorithms are in place in a lot of different municipalities to determine who’s at high risk for committing crimes in the future. That determines how sentencing might occur. So we see Vernon here rated as low risk, Brisha related as high risk. But when we look at their past offenses, Vernon’s look, just intuitively to us are humanized. They look potentially more egregious than what a Brisha has done in the past. And lo and behold, the algorithm’s job is to predict who’s likely for repeated crimes, repeated offenses and our high-risk in this example had none.

Chris Hemphill:
Another example, output of the algorithm. Again, a reminder, this particular algorithm used a bunch of different characteristics, but race was not one of them, yet we’re seeing these major differences in the types of crimes that were committed, one petty theft versus all these offenses, but this person was rated low risk while Robert was rated as medium risk. ProPublica did a study on machine bias. In this algorithm it’s called the COMPAS recidivism algorithm. But the main point is that for people that were more or less equally weighted, even controlling for race, it was 77% more likely to identify black people as high-risk individuals.

Chris Hemphill:
Another example it’s, this is in criminal justice, but even a bias, even plagues the upper echelons. Big technology companies are not free of this stuff. In this example, Amazon was using an algorithm to identify who based on their resume is likely to extremely successful candidate. That makes sense, right? Look at a bunch of resumes, look at thousands of resumes and use that data to determine who another successful candidate might be. Whoever’s most similar to that resume is likely to be a successful candidate. But the result when they started coming through the data and looking at who this algorithmic approach was recommending, it was heavily biased towards men. Even if gender wasn’t an explicit term used, it would even pick up on the presence of names of women’s universities and give those candidates lower scores. They eventually decommissioned that algorithm and took it out of use.

Chris Hemphill:
And of course, the industry that we’re focused on today, the industry that we’re in is in healthcare. Actually in this example, and we’ll go a little bit deeper into it when we talk about label choice bias. But in this example, another algorithm that wasn’t using race as an explicit predictive factor, still disadvantaged black people at a much higher rate. Basically white patients that had the same number of comorbidities as black patients were flagged as higher priority to reach out than black patients. So, to kind of deconstruct that question, this is just an example of artificial intelligence algorithm using a bunch of different characteristics to determine who should receive cardiology communications. Reasonable things such as how they score on health risk assessments, biomarkers, such as systolic blood pressure and BMI, these are all very reasonable inputs to the model, but even if we’re using a set of a pretty reasonable inputs, are we guaranteed that we didn’t use race, we didn’t use sex, we didn’t use gender, but does that mean that the algorithm is unbiased?

Chris Hemphill:
I would argue that that’s not necessarily the case, and that there’s action that needs to happen afterwards to verify that. So, just some examples of the type of bias that can sneak in without explicitly using those variables are systemic bias, especially if your study is informed by data from people who have visited health systems, where people who have had care, then if there’s been systemic issues where one population has been significantly disadvantaged, well, we missed their data from the system and that’s a systemic bias, then creeps into the recommendations that one might make. Financial bias plays into that as well because when we’re using data modeling approaches, we’re looking at who was able to afford those services and who is able to deal with the financial complexities of interacting with healthcare.

Chris Hemphill:
Another type of bias is based on the idea, label bias. The really simple concept is that an algorithm is no better than what you’re optimizing it for. So hold that thought. This is going to be a big key to the part that Emily and Rebecca go over, but label bias is another insidious form of bias that often isn’t considered in these examples, but by looking at it, you can start avoiding it. So verdict is if there are algorithms in place, consider them guilty, consider them biased until there’s been some sort of check or review that reveals and defines, or modulates the output of the algorithm so that it removes whatever bias might exist. So what we’re going to be exploring the neck in the rest of the conversation is, there’s tons of articles, tons of content out there already about bias that exists.

Chris Hemphill:
We want to talk about this quote here, algorithms can reinforce structural bias, or they can fight against them. So we want to talk, we want to focus on that fight against structural bias. And with that, I’m going to hand it over to Emily and Rebecca to discuss the algorithmic bias initiative. So, thank you for requesting control and hand it over to you. Thank you.

Dr. Emily Bembeneck:
Thank you so much, Chris. Just hearing you talk about this again, just always reminds me of how important it is and how needed this work is. So just to get started, I’ll kind of reintroduce myself and Becca my colleague. So I lead the Center for Applied AI at Chicago Booth here in lovely Hyde Park, Chicago. Chicago Booth is a business school at the University of Chicago, and our center is a new research center that I’ll talk about in a second, and involved in a variety of issues and topics related to AI.

Rebecca Nissan:
Hi, everyone. I’m Becca. Again, glad to be here. And like Emily, I work at the Center for Applied AI and also have joint appointment at ideas42, which is a behavioral science nonprofit. And through my role at ideas42, I am leading the work that we’re doing, working with partners and providers and governments, and lots of folks across the healthcare industry and people that relate to it to reduce bias and algorithms, and try to take some of the research that Emily’s going to talk about and apply it in the real world for impact.

Dr. Emily Bembeneck:
Thanks. So here is our center. So today we’re going to talk about our work in healthcare, but we actually work in a variety of fields, finance, public policy, criminal justice, education. Our work is founded in behavioral science and heavily influenced by the work of our faculty directors, Sendhil Mullainathan, who is well known for his work in a variety of topics that touch on bias. So, poverty, decision-making and under stress and in episodes of scarcity. Under Sendhil supervision, we try to use our research in behavioral science and AI to help humans make better decisions. We often think that AI can help people make better decisions by making the decisions for them, but it turns out that a lot of the decisions humans have made in the past that algorithms are modeling and trying to improve are problematic. And so we actually need to work closely in concert with algorithms to make sure that what information they’re providing us is actually helping us make better decisions. And when I say better today, I’m mostly talking about less biased or even unbiased decisions.

Rebecca Nissan:
So I’ll just jump in here to say a little bit more about ideas42. So we’re innovation, nonprofit applying behavioral science. So the study of human decision-making and why people do what they do to help improve lives and build better products and systems and drive social change at scale. So we work across many domains, not only in healthcare, though our work and algorithmic bias right now is focused in healthcare. Our machine learning team specifically uses a behavioral approach to support algorithmic tool adoption and ethical AI. So as I was saying before, we’re working directly with providers, payers, governments, and vendors, to identify ways that bias creeps into algorithms that people are using, and then to help guide those partners in ways to mitigate bias and algorithms, and also to create sustainable systems for proactively avoiding bias. So trying to equip the folks that we work with to have systems that allow them to proactively identify bias and not only catch it after the fact.

Dr. Emily Bembeneck:
And if you’re wondering how we’re connected, it’s because Sendhil, our faculty director used to go back. So this guy also co-founded ideas42 over a decade ago, and we share our leadership and similar vision and work very closely together on these kinds of topics, and are happy to both be able to talk to you today about our work. So we’re here today to talk about specifically algorithmic bias. What we’ve essentially done through our research is developing a framework that industry leaders, vendors and governments can use to think about how to assess and then mitigate the bias in their algorithms. I’m going to walk you through an example to kind of help you understand how we think about the kinds of bias that we look at. Chris mentioned label bias. And so I’ll define that for you and give you an example. And then we’ll kind of talk about that framework itself. Like, how do we think about the steps involved in mitigating and assessing your bias, and where’s the place to start if you might be in a company that’s considering these kinds of [inaudible 00:18:12].

Dr. Emily Bembeneck:
So in late 2019, we published a paper in science focused on algorithmic bias in healthcare. We looked at an algorithm that’s very commonly used across the United States, and a variety of systems that scores patients on how good of a fit they would be for a high-risk care program. So essentially it’s looking for patients, it thinks will benefit the most from a program like this. Intuitively when we think about that, we think, well, we want the patients who are the most sick to be put into a care program like this. They’ll get more care upfront in the hopes that this alleviates problems down the road and stops flare ups from happening or other kinds of high costs and high pain, high inconvenience scenarios for patients, that’s human intuition.

Dr. Emily Bembeneck:
But the algorithm didn’t quite think about it like that. The other of them had a bunch of information given to it, but what it actually ended up predicting was which patients will incur the most healthcare costs over the next year. So we’re thinking we care about need, how much do people need care? And the algorithm is thinking, well, how much are people going to cost on care? Okay. Those are a little different. So we can think about them as being related intuitively. Of course, if someone needs care, it’s going to cost something, right? But they’re not the same. There’s a significant discrepancy between the two that results in bias. We can like dig into that a little bit. So why would it matter that those are not exactly the same if they are correlated?

Dr. Emily Bembeneck:
Well, the problem is that they may be correlated, but they’re not equal. I could need a lot of care, but I may not be able to go to the hospital, I may not have a good doctor, I may have mobility issues, I may be afraid of being able to pay the hospital. Bill, I may not trust my doctor, my doctors may have mistreated me in the past. There are host of issues that could influence my decision of whether to get the care I need or not. Those kinds of problems we see in populations that are primarily less advantaged. So often it’s black patients who live in areas that are lower income and have lower access to care. What happens then is that our dataset, if we’re looking at cost, it’s only focusing on those patients who actually had care and paid for that care, it’s not looking at the actual health of a person underneath. Label choice bias is that discrepancy, it’s thinking, well, we care about how sick someone is, but the algorithm is thinking, well, that’s pretty close to how much someone costs. So that’s what we’re going to focus on.

Dr. Emily Bembeneck:
You can think of an algorithm like a genie. If I ask a genie to be rich, and then the genie says, “Hi, rich, what’s your next wish?” Well, I didn’t mean that kind of rich, but the genie is only answering the thing that I told it. That’s how an algorithm is, it’s extremely literal and it’s only going to do exactly what we ask it to do. Okay. So how do we fix this? Before we talk about how to fix it, let’s think about how we actually understand that it’s happening. In this graph, we can see black patients in the purple and white patients in the yellow. What we’re looking at is how the algorithm scored them. So high score appear in 95 to 100, means that they’re going into the High-Risk Program.

Dr. Emily Bembeneck:
But here on the left axis, this is the number of active chronic conditions, which we can kind of use to think of how healthy or unhealthy someone might be. If someone has three active chronic conditions going along the line here, they should be scored the same way. Unfortunately, the algorithm didn’t score them the same way. As you can see, the black patient is only a 90 and the white patient here is a 100. The white patient gets into the program, the black patient does not, not automatically. You can see that all along this line, that there’s a strong discrepancy there. That is not because the algorithm said black patients get less care or the algorithms thought black patients were in some way inferior, the algorithm didn’t even know they were black patients.

Dr. Emily Bembeneck:
The algorithm instead is picking up on the systemic issues that showcase themselves in actual care and how people are treated. It didn’t need to know that they’re black because it had the data from society, which is evidence that black people are treated differently than white people. And that historic data set is what’s being used to predict the future, and that’s where the problem is. Okay, so how do we solve this? I sent the discrepancy between what we want it to do and what it’s actually doing, is label choice bias. So the solution is making that discrepancy smaller, making that gap smaller, making what we want it to predict and what it’s actually predicting much closer. So here in this example, I’ve been talking about the decision we it to make is which patients will benefit most from the High-Risk Care Program? Which patients need this care the most?

Dr. Emily Bembeneck:
And so let’s change the algorithm to instead of saying, well, who costs the most? Unfortunately we don’t have a variable in the dataset for need. I don’t have an inherent score of 2.2 on me. There’s no way for us to actually measure that. We have to measure it some other information. And so instead of having the algorithm focus on how much I cost, it’s going to focus on the highest number of co-morbidities, what is my actual health level? Once we did this, once we made the algorithm more closely matched, it lets nearly twice as many black patients being referred to the program, same health level as those white patients, but before they had been [inaudible 00:24:32]. Here’s some citations that we’ll share around afterwards. You can kind of look at more of the research that we’ve done that shows how this shows up in healthcare in a lot of different scenarios. Just starting the horse, there’s a lot more to be done. I’m going to hand it over to Becca to talk about the applied framework of how we actually now tackle this in real life situations.

Rebecca Nissan:
Thanks. So the Algorithmic Bias Playbook is a resource that we created to give people who are interested in applying this to their organizations, a practical guide for doing so. So it’s one thing to understand how bias works and how to potentially fix it in theory, but it’s another thing to understand and execute the steps that are needed to look at the algorithms you’re using as an organization, and then assess whether they’re biased and then try to actually address the bias. So if you go to the next slide, I’ll talk through at a very high level what the playbook suggests as a framework. And then I’m sure we can make it available after for people to read more closely.

Rebecca Nissan:
So the playbook breaks down a framework into four steps. The first is to inventory algorithms that are being used in an organization. What we find is this is actually a pretty substantial step. So in most organizations there isn’t an algorithms team. I mean, there may be teams that focus more on data than others, but actually algorithms are used across lots of different teams in organizations. So there’s actually a lot of work to be done in figuring out what is going on and who’s using algorithms and putting all of that information in one place. And then we also recommend designating somebody, we call them a steward, to maintain the inventory. Since algorithms change all the time, people decide to create new algorithms or update the ones that they’re using. And keeping track of that is a really important way to start to take the first step towards addressing bias, to even know what’s going on in the first place.

Rebecca Nissan:
The second step is to screen for bias. And so the way we lay that out is through exactly the exercise that Emily just talked through, which is to articulate the ideal target for each algorithm. So in the example that we just used that would be health or need. And then you note the actual target, which in this example was cost, and then note the discrepancies between health and costs and how those discrepancies might differ across groups that we’re interested in looking at. So for example, between different racial groups.

Rebecca Nissan:
The next part of this exercise is to then analyze performance of the algorithm for populations of interest against the ideal target. So actually in the example that Emily just gave, when you look at how the algorithm performs at predicting cost, it does a really good job for both black and white patients at predicting costs. So if you just looked at algorithmic performance on the actual target, and you would’ve missed the fact that the algorithm doesn’t perform equally well relative to the ideal target, which is health. So that’s a really important component of identifying label choice bias.

Rebecca Nissan:
The next step is to start to address the problem. So retrain algorithms, ideally to address the ideal target that we’re interested in. So in some cases, this actually means picking a different algorithm. For example, there may be algorithms that exist. In fact, there are algorithms that exist that predict active chronic conditions instead of cost. So in that example, choosing a different algorithm is a relatively simple way to address this problem. And of course, that’s not a perfect fix, but it’s a very good first step in addressing the specific type of bias that we’re talking about. In some cases, it’s not feasible to retrain an algorithm or to pick one that already exists. In some cases, the recommendation is to justice suspend the use of an algorithm that you found to be biased.

Rebecca Nissan:
And then finally, the last step is to prevent future bias through establishing structures in your organization for both existing and future algorithms. So making sure that somebody is assigned to maintain the inventory in an ongoing way, making sure that people who have strategic oversight over the organization and set the policies and practices that people across teams need to follow, making sure somebody at that level is aware and able to address this and create those systems so that we’re catching bias before it happens.

Dr. Emily Bembeneck:
There’s lots more to say, there’s lots more to do. So we encourage you to join our community. This is our website, @chicagobooth. You can download the playbook in full. We have events coming up and a conference in the fall. We’re going to have to bring together experts and practitioners who are really leading the on ground effort to change the situation. So we encourage you to join us. We also encourage you to reach out directly to us. If you have any questions or want to talk more, our email addresses are here on the slide, and I know we’ll share them afterwards. Ziad of course, is our co-author and couldn’t be here, but is also very open to speaking. So thank you very much, Chris, back to you.

Chris Hemphill:
All right. Fantastic. Yeah, I want to encourage everybody to Google that, find that playbook. It was extremely well-written, extremely accessible to. So if you’re not directly working for a team that’s generating algorithms or anything like that, it’s just well worth the read, just to understand what might be going through your part of the organization and set standards around it and ask the right questions around what’s going on. That playbook gives you really concrete examples that you can use. Transitioning. So to speak of another concrete example, we were just going to share just to give life to some of the ideas that we’ve talked about. What does that look like on the ground for a company like ours? So we’re just going to review our work and our relationship with the Center for Applied AI because they really did open up some great capabilities and conversations within our organization. So on our team, Ling, I’ll throw you a chance to introduce and then I’ll carry it on.

Ling Hong:
Yeah. Again, I’m Ling. I’m the data scientist at SymphonyRM. Yeah, I think in this session, I’m going to show you some of the works we have done here at SymphonyRM to examine model bias.

Chris Hemphill:
Excellent. So just to orient everybody on who SymphonyRM is, basically when it comes to these bias questions, how you answer them, the ideal answer is going to depend on what problem you’re solving. Our focus is taking things like clinical history, the types of communications and outreach that have been done in the past, such as call centers, emails, text messaging, things like that, and take information known on consumer behavior and use that to develop personalized engagement plans. So the basic idea is to use all this past clinical and communication data to identify needs and identify what the best pathway is to help people identify and address those needs.

Chris Hemphill:
This uses a lot of data-driven approaches, a lot of various types of algorithms, including machine learning to accomplish that. So it raised the question, especially when that science article came out of our methods contributing to this problem that’s been highlighted, are we using labels, are we using choices, or are we pursuing ends that might be disadvantaging members of our population, or member members of health system populations, are we perpetuating bias within AI? There was a lot of inspiration that came around this work too. There’s a book that came out a few years ago, I think in 2016 called Weapons of Math Destruction, encourage anybody who’s really interested in this subject. Again, it’s really accessible, really easily written, but it talks about the challenges with algorithms. There’s a lot of coverage that STAT magazine and STAT news has been doing to cover algorithmic bias as well, and really opened our eyes to the subject and we wanted to make sure that we weren’t contributing to that problem.

Chris Hemphill:
So very similar to what Becca just shared, we went through our own process, looking introspectively, looking at what algorithms are in play and what challenges they might be causing. Ultimately from that analysis, and we’re going to get a little bit deeper into it, but I have to say that Pandora’s box is open. If you are using data or using past data and even looking at … It doesn’t have to be with the machine learning algorithm, if you’re even saying, oh, well, our average patient who comes in for cardio looks like this, and then making future marketing or outreach decisions, then Pandora’s box is open. There’s a potential that there’s bias decisions being made. So algorithms, they don’t have to be made from people who graduated from MIT or Carnegie Mellon, specifically studying data science, as long as we’ve looked at some previous population and we’re using data to make our decision, we’re creating algorithms every day, even if it’s not a team that’s devoted to doing that.

Chris Hemphill:
So the result of those algorithms and the result of a lot of the bias that we were talking about, such as financial bias and systemic bias, that starts showing up when we start looking at how patient populations are composed. So this is just an example of a health systems racial breakdown. The orange, we have the local census, the surrounding zip codes, the population breakdown by percent of the surrounding area. And then we have the various racial populations. You see that within healthcare data, there’s a significant amount of unknown people who haven’t shared their race. So those are kind of the real problems that you start encountering once you dig past the layer of how do we address this bias. And these imbalanced datasets, they help us learn more strongly the behaviors and characteristics of the larger class, which then disadvantages any kind of algorithms or outreach that we do for people that are in the other classes, but there’s major racial imbalance.

Chris Hemphill:
So even if a model score looks great, you got that nine times lift, you’re operating at 9,000 times the accuracy of some other approach. If you’re not breaking it down by race, if you’re not looking at that and doing that assessment, you might be significantly disadvantaging another group. That’s what we found in some of our algorithm forms. This was something that we were kicking off last year. The initiative started in 2019, but we put a significant amount of focus on it. It really just all brought everything to our attention with all of the problems that were being unearthed when George Floyd was murdered. So that became a company uniting initiative. We developed a mission around this, specifically around committing to find and read out bias in the way that we can, which is focusing on outreach and algorithms and things like that. The gaps were there, and it became something that we focused the entire company on.

Chris Hemphill:
So again, not just the data science team and its algorithmic development, but questions of implementation, how we collect the types of data that we collect so that we can address these types of issues, how it’s discussed within the market with our customer success group, our analytics, team sales and marketing. The data ethics team consists of people that can execute on this stuff, as well as people that focus on the algorithms. When you bring all those people together, it’s a blessing to bring all those people together, but there’s always going to be debate, discussion, conflict, things that arise. So there was all kinds of questions, it’s extremely difficult philosophical question to ask about what is ethical outreach? How do we quantify that? By doing this research, are we revealing ourselves to have been problematic and then become an industry scapegoat?

Chris Hemphill:
Our approach is racist. These are all important questions that I think are likely to come up when you start addressing these types of issues. There should be an open conversation within organizations about these types of things. We shouldn’t shy away from these just because we think that we’re going to open Pandora’s box, because I’m telling you Pandora’s box is already open. So let’s have a discussion on what we actually do about the problem. So with all those questions floating in the air, and without our own perspective, that’s why we decided to engage Dr. Ziad. We’ve seen the research and we knew that there was a big conversation about label choice bias, and some of the things that the organization that they were studying or doing. And we want to learn more about, hey, how do we solve some of these questions? And again, even if we were to do everything internally, how great is it for some organizations to give themselves a pat on the back to say that they’re unbiased? I always think that it’s better to get that feedback from a disinterested third party.

Chris Hemphill:
And again, the same quote, I’ll say at time and time again, algorithms we’re in a place, were at a crossroad, where we can either perpetuate the bias that’s existed systemically, or we can look at how these approaches can reverse some of those trends. Ziad and team just had a lot to share around that and gave us a lot of inspiration for the path that we chose to go down. So to give light to that path and some of the things that we did, I’m going to hand it over to Ling.

Ling Hong:
Okay. Thanks, Chris. So just like Chris mentioned, we are taking this issue seriously at SymphonyRM. So right now at SymphonyRM, we have built the analysis, a model bias as part of our data science pipeline, which means for every single new model we develop, we will try our best to generate models, showing the model performance for different populations. And just as the study shows, there are a couple of different approaches we are taking to alleviate bias. So the first one, we are tackling the label choice bias. I can give you an example about chronic kidney disease. So we know that for some communities that are on the resource patients living in their communities, they will have fewer assess to nephrologist.

Ling Hong:
And the result of that is that it’s very unlikely for them to have a diagnosis records within the hospital system. So if we use the diagnosis code only to train the model, it’s very likely that we’ll be meeting those patients. So for this issue, we not just include diagnosis codes, procedure codes, we are also looking at, rather than that readouts, like [inaudible 00:40:11] or potassium or other EGF or values, to make sure that we really have a very inclusive laboring criteria. The second approach here is the dynamics thresholding approach that we developed. So we know that some subpopulations tend to use last medical resources in the first place. So for this type of patients, we would design our strategy to reach out more to this group of patients. So, therefore when you look at without, you will not be disproportionally reaching out to only the white population.

Ling Hong:
And the third approach that we are currently experimenting with is to simulate a data set that will give us more data points on subpopulations. So one basic mechanism of how machine learning model works is that it would be easier for the model to pick up patterns for larger data sets. So if you don’t have enough data points for minority groups, it’s very hard for the algorithm in the first place to really learn the patterns from those groups. So we are hoping that by having this simulated data set, we can make the algorithm to be more intelligent to learn about the health conditions of this minority groups.

Ling Hong:
So this slide shows the effectiveness of our strategy. So before we applied the model bias analysis approach, you can see that we have reached out to a lot of white patients, but then we use the dynamic thresholding approach, and we adjust the proportion, considering both [inaudible 00:42:00], as well as the demographic distribution in that area. And after this, we’ll be able to reach out to 23% more of the minority groups in that community. So that’s a huge boost, and we hope that they keep applying this approaching the future, we’ll be able to resolve the housing inequity issue.

Chris Hemphill:
Thank you very much, Ling. Thank you for that explanation. There a lot that somebody listening could have gotten out of that, especially when you make the point about using codes. There’s often looking at previous diagnosis codes to determine what that future outreach should be, but there’s going to be a lot of people for which they might have that illness but not even have an appropriate code. So it’s important to find other ways to determine who that highly acute population is.

Ling Hong:
Yeah, so for this future steps, we are not just believing in ourselves to just look at racial bias. So in the future, we will be also looking at gender bias and also bias arise from income levels considering more, like our social, economic factors, and we hope that through this efforts AI can read the [inaudible 00:43:26].

Chris Hemphill:
Appreciate that, Ling. So what we wanted to cover in this first part was an overview of algorithmic bias, what it is and steps that organizations are taking to combat it and get that really awesome broad perspective from Emily and Rebecca, who are working with many organizations to do this. Now it’s time, we’re opening that up for you to take part in this conversation. So I noticed that there was a question that came in through Q&A. Keep the questions coming. If there are questions or thoughts that you’d like to share, because what we’ll focus on now is just an active and open conversation.

Chris Hemphill:
If there are questions that come up later that you might not want to address right now in public, that’s okay. Feel free to shoot those over to us, but really excited to get into some of the work that Emily, Rebecca, Ling and team have been doing. So to kick it off, I guess for Rebecca and Emily, what are the most common use cases that you’re running into ours? The example that we shared was around conducting outreach programs and outreach campaigns to people of need, but what are some of the other use cases that you’re seeing out there?

Dr. Emily Bembeneck:
Go ahead. Or you probably have more specific.

Rebecca Nissan:
I was going to say, I’m happy to start there and then feel free to add. But one of the cases that we’ve seen across working specifically with payers is cases similar to the one in the science paper, where care management algorithms are being used to decide who gets more access to care. Sorry, to decide who gets care management programs. And so because there’s inequities and access to care, often there’s actually a whole suite of algorithms that use cost to decide on how to allocate those care management programs. And so that very specific problem has actually come up with a lot of different partners across the industry. So, I’d say that’s the most common one that we’ve seen.

Dr. Emily Bembeneck:
There’s also one, I think maybe slightly similar to what Ling was talking about. And Becca, you may have more info on this as well, where we’ll notice or an organization will notice that they don’t have complete information on some group, let’s say a group with a certain condition. And the way that their models are constructed is separating those who’ve been tested for the condition and got a positive result versus everyone else. And so what this ends up doing is neglecting to include those who may have a positive result if they had been tested, but had never been tested. And so we’ve also been working on how do you create a more accurate base for the model to work on when your information itself maybe only half there, it’s another common one.

Chris Hemphill:
We got a question that came in from Matt Tewkesbury in the audience, it addresses a really detailed issue. I want to put it out to the group here, and there’s a little bit thinking around it. So where does it start and how does circumstance contribute? And the example that Matt gave is that as a hiring manager in healthcare, there are other people ahead of him in the hiring process and a variety of circumstances that occur before he receives a candidate pool to review and select. So there’s the concept that he can be perceived as discriminatory, only hiring from a particular group, because that’s the pool that he had to choose from. So that’s really, parallels the idea that once there’s been a pool of beneficiaries or patients that you are working from that particular dataset. So I’m curious, it kind of has parallels with selection bias. I wanted to get thoughts from the table around this question. I’ll throw at Emily first.

Dr. Emily Bembeneck:
Well, one of the points we make in the playbook is that it really requires not just one person trying to solve this problem, but a lot of different people trying to solve this problem. I think that’s true, we all find ourselves wherever we find ourselves in our organization, and we have a limited ability to make change from that position. However, I think recognizing that problem and then voicing it is actually huge. If you recognize that your pool of candidates or your data status is lacking, you have a choice to make. That can be simply the way things are, or that can be an opportunity for change. I think that we have to be proactive about fixing those problems, they are not going to go away on their own. Our candidate pools here too, we have a body of students. We have to be extremely proactive about making sure that our candidates are as diverse as possible.

Dr. Emily Bembeneck:
And that doesn’t mean imposing silly quotas or something like that, it means finding the good people that don’t recognize it that they would be a good fit here or that they would be welcomed here. It means changing our outreach strategies. It means changing the way that we talk to people, it’s changing how we are inclusive to others. I think that that mindset and using your voice and your organization to encourage that is one key step I think that anyone can do.

Chris Hemphill:
So I’ll throw the next question. Ling, do you want to tackle the next question? Oh, you’re on mute.

Ling Hong:
This question is kind of associated with, that Chris just mentioned. So we know that algorithms sometimes can reinforce structural biases, or we can actually use algorithm to fight against the biases. So if a healthcare leader wants to choose to use algorithm to fight against the biases, what are some of the peak force or potential challenges they will be facing based on your experience?

Dr. Emily Bembeneck:
So let me make sure I heard you correctly. You say, if someone is attempting to use an algorithm to, not perpetuate bias too but to make progress against it, what are some pitfalls or challenges that they will encounter in that attempt?

Ling Hong:
Yeah, so maybe I can share some of my experience at SymphonyRM. So for us, when we were starting to adjust the model bias, one thing is that there’s a possibility that some models just cannot be faced, but they are like the cash cow for the company. So it’s hard for you to give up on them. Another more common issue I think within the industry is that we just don’t have the race and ethnicity data to analyze bias, and it’s hard for us to get this from our clients.

Rebecca Nissan:
So I can just pick up on the second piece of that around race and ethnicity data, that is another common issue that comes up for a lot of people because it often can be hard to get that data. Something that we’ve explored is using imputation techniques to make up for that. I think that’s one way to do it. There are some suggestions in the playbook about different methods that you can use to essentially create better data given whatever you have. That said, I do think that one of the conversations we’ve had with a lot of different companies that we’ve worked with is around how to actually create better systems for gathering data from people and getting self-reported race, ethnicity, language, data. So sometimes that can mean coming up with a new system for actually asking people that question or coming up with different ways to directly address the problem head-on at least for the future.

Dr. Emily Bembeneck:
And just to touch on the first part of your question, what if you have an algorithm that you consider integral and it turns out to be biased? I think that might happen, those cases could come up. But I don’t think that there’s always going to be a trade-off of that. I think a lot of the times, what we end up actually doing is making algorithms do their job better. So you want an algorithm to actually solve some particular problem and it’s doing a bad job, that’s why it’s biased. Or it’s biased and it’s doing a bad job. That’s what’s happening when we fix it. When we make it actually create an accurate decision, that’s going to improve the bottom line. So those two things, equity and profits or whatever, don’t have to be mutually exclusive, they’re often actually more aligned than you would perhaps think.

Ling Hong:
Yeah. Also, when I’m thinking about this problem, I feel like there should be some regulations in place, because housekeeper can be a very lucrative industry and people just have the tendency to go up to profits if there’s no something in place to put them off back. So do think that we need kind of lighten regulatory environment to post limitation on this.

Dr. Emily Bembeneck:
I can start. So I do think regulation is an important part of the puzzle. I think one of the challenges that regulators face is trying to regulate an industry that is incredibly innovative and fast-moving. So it’s hard to make very prescriptive rules around specific use cases because those may change right after the regulations are out. So I think that’s one challenge. But I think that there is a strong role for regulators to play, and that is around two key points. One is a set of standards. So what are the goalposts, what is bias? How do you define it? What matters to regulators? What should companies be looking at?

Dr. Emily Bembeneck:
And then what are the goalposts? What do we actually think is unacceptable? What do we think is acceptable? What needs to be done as a baseline in order to avoid litigation or some other kind of falling outside of the law? And the other is that litigation piece, is being able to enforce those standards and rules that they have developed and developing them in a way that still leads to innovation and has a lot of room for development and change, but at the same time keeps the interests of the public at heart. So, that’s been my thought.

Chris Hemphill:
There’s a question that came in from the audience that runs kind of parallel to these conversation points, and it was around the comfort that people might have in providing demographic data. One thing that one might’ve noticed on our previous slide was that around 20% of the people that we were modeling for chose not to disclose their race. So the question is around patient reluctance to answer questions like that. Gosh, that opens up a whole nother door when talking about things like sexual orientation and gender identity data, which is a huge point of conversation. So just questions around creating a safe space for patients to share and educating clinicians on the bias that they might be expressing in their care. Just curious about thoughts or encounters that you’ve had with that.

Ling Hong:
Yeah. I actually has ideas on this because my another identity is patient. So when people are asking me to disclose my personal information, the first question I have in mind is, what’s the cash? What’s the benefit I’m going to get out of this? So one, I’m not sure if this system profit analogies, but if you [inaudible 00:56:25] giving out my information to the hospital is just like paying cost. Eventually they all promise me that I would benefit from the money they are going to spend, but I just don’t know what’s the eventual outcome of it. So I’m not sure if that’s a perspective of looking at this question.

Dr. Emily Bembeneck:
I think that’s a really important and common perspective, honestly. I talked to a lot of students who are interested in this. I think there’s a lot of distrust around the system, how is the system going to use this information? Is this going to disadvantage me because now they know that I’m part of some group that is historically disadvantaged? When is this going to come back and bite me? Like you said, what’s the catch? I think there is so much distrust there.

Dr. Emily Bembeneck:
And fixing that is partially on those of us who have the power to make change, to address issues that exists, there is that inequity, but also educating people around how we’re fixing this and how that data can actually help. So Becca can probably speak more on this, but I know that we continually tell people, not putting race data in your algorithm does not make it unbiased. In fact, it can have the opposite effect. That is not relative with bias or lack of bias. And so talking to patients, talking to doctors, talking to systems and all of these about the problem and about how to solve it, I think is one important step.

Rebecca Nissan:
I also wanted to add that I think this could really be a project into itself and probably a field, even. I think if you think the context that that happens in the encounter between a doctor and a patient and all of the things that the patient is thinking about and how those contexts could differ, depending on the situation that that patient is in and where they’re coming from, part of it is probably an issue of distrust, but there might also be issues of hassles that the patient is going through and just trying to fill out a form, if you’re trying to get into an appointment and you’re concerned about your health, and in order to get to the question about your race, you’ve already gone through three pages of medical documents, then the chances that people fill it out are going to be a lot smaller.

Rebecca Nissan:
And so I think that’s where behavioral science can help us reorganize some of the context there to help people gather better data in a way that works for the people who want to collect the data, but also for the patients who are being asked to fill it out.

Chris Hemphill:
That’s so … Oh, go ahead.

Dr. Emily Bembeneck:
Sorry, Chris. [inaudible 00:59:12] on that too is something that came up with people, I was talking with the other day, which is just the intersectional nature of these questions and answers. As we’ve seen on some of these slides, we’re putting people in one bucket, we’re saying they are white or they are black. A lot of times people have a set of identities and understanding from a data point of view, how do we structure that? How do we understand that? How does that inform our decisions and how can we better allow patients to accurately self-identify and then make sure that we’re representing them accurately in the work that we do as well?

Chris Hemphill:
Excellent. Yeah. I think that often things become problems before people focus on bringing in behavioral science to focus on it. So we have these forms, we have these intake points and things like that. But without taking that particular lens we can say, when you’re asked about race data. But if it was on page three of a form, then that might be impacting why we’re not getting the accurate information. It’s three o’clock Eastern here, but we still have another question from the audience. This one from Diego Galen Hoffman who asked, so we’ll do this and we’ll have time for just final thoughts and round table there. I don’t want to take everybody’s time on this, but it is a really good question, which is, how do we keep private companies accountable for bias in their algorithms, if they’re not transparent about their secret sauce?

Chris Hemphill:
And just to go a little bit deeper. I mean, there’s probably been an all kinds of scenarios where a health leader or a leader of some sort might ask, well, how did you identify that population? Or how did you say that these are the people that we should reach out to for this particular service? And the company responds, well, that’s proprietary, that’s our sauce. That algorithm that we were going through as far as criminal justice earlier on proprietary, nobody knows the inner workings of the algorithm, they refused to reveal it. So Emily or Rebecca, what are your thoughts on organizations that choose not to be transparent?

Dr. Emily Bembeneck:
I guess I would take a hopeful approach. A lot of the organizations we work with, we don’t need to know always the secret sauce to discover whether that algorithm is working as intended or working as it should be. That’s one of the good things about label choice bias, that you can tell if it exists based on the outcomes, you can judge the decisions that the algorithm is advising next to real world outcomes and you can see if there’s a discrepancy there. Now, once it gets more challenging is when you want to adjust that algorithm to change those outcomes. You need to make some changes on your variables and your training data, perhaps. But I think that honestly, we can see if there’s a problem without knowing that, and then we can test whether their solutions have been effective without knowing necessarily what they did. So that’s one good thing, I think.

Rebecca Nissan:
Yep. Just to build on that, I think to your point about Pandora’s box being open, it’s no longer going to be a viable path for companies to just say, well, we’ve never looked into it, therefore it’s not a problem. And so, I think even at the expense of sometimes needing to share, at least with people under an NDA of some kind, but at least with people who are interested in looking into the issue, I think that’s going to be a necessary step. And then the way that people talk about it after can be, I guess, addressed. But the point is just that, I think it’s no longer going to be an option to just not deal with the problem.

Chris Hemphill:
So we’re a little bit past the hour, but I wanted to give everybody the opportunity to just share final thoughts here, because there’s a reason that you embarked on this work and started doing this, and have this particular focus. And there’s a reason that you came in and had a conversation with us about this today. So if there’s anything that you want the audience to get out of this conversation, Ling, Emily, Rebecca, I just want to allow the opportunity to share those final thoughts before we conclude and get back to our weeks. Start with Rebecca.

Rebecca Nissan:
I really just want to say that it’s been a pleasure working with this group, and thanks so much for having me and for a really useful discussion today.

Chris Hemphill:
And Emily.

Dr. Emily Bembeneck:
Likewise, thank you so much. Every time we do something like this, I’m reminded of how important the work is, like I said, and how much more there is to do. It’s a great feeling to know that something that you’re doing can make a difference. And so I’m just thrilled to be able to work with you guys and have this opportunity.

Chris Hemphill:
Thank you. And Ling.

Ling Hong:
Yeah, just Becca and Emily mentioned, this is very important work, and I’m also hoping that not just researchers, not just practitioner within the industry, but also patients would take the initiative and realize it’s easy, because it is something that really requires joint effort to really put things forward.

Chris Hemphill:
Great point, Ling, and that is exciting. We talked about the patient’s role, they’re not even sure how the data’s going to be used when being asked about it. So there needs to be a way to get them to participate as well. I appreciate everybody who has stayed on with us for just a couple of minutes longer. Matt, Carroll, Diego, for your questions and all that, appreciate that. We have talks like this on a weekly basis and we’re actually launching a podcast next week called Hello Health Podcast.

Chris Hemphill:
The first episode is going to be focused on healthcare disparities and healthcare equity issues. I believe it’s going to come out on the 29th, next week, so seven days from now, but keep your eyes out on that. And if you want to be a part of these conversations, have discussions with these industry experts, feel free to subscribe to SymphonyRM on LinkedIn or Twitter. Thank you.

Find the Clarity You’ve Been Missing

Learn how Actium Health is driving improved quality, outcomes, and revenue for innovative health systems nationwide.

Request Demo
Meet Some of Our Customers