A new Chilmark Research report by Dr. Jody Ranck, the firm's senior analyst, explores state-of-the-art processes for bias and risk mitigation in artificial intelligence that can be used to develop more trustworthy machine learning tools for healthcare.

WHY IT MATTERS
As the usage of artificial intelligence in healthcare grows, some providers are skeptical about how much they should trust machine learning models deployed in clinical settings. AI products and services have the potential to determine who gets what form of medical care and when – so the stakes are high when algorithms are deployed, as Chilmark's 2022 "AI and Trust in Healthcare Report," published Sept. 13, explains.

Growth in enterprise-level augmented and artificial intelligence has touched population health research, clinical practice, emergency room management, health system operations, revenue cycle management, supply chains and more.

Efficiencies and cost-savings that AI can help organizations realize are driving that array of use cases, along with deeper insights into clinical patterns that machine learning can surface.

But there are also many examples of algorithmic bias with respect to race, gender and other variables that have raised concerns about how AI is being deployed in healthcare settings, and what downstream effects of "black box" models could be.

The Chilmark report points to the hundreds of first-year COVID-19 pandemic algorithms analyzing X-rays and CT scans to aid diagnosis that could not be reproduced in scientific study. Clinical decision support tools based on problematic science are still in use, according to the research.

Along with the tech industry, the report criticizes the U.S. Food and Drug Administration for falling behind in addressing the challenges the rapidly growing industry presents for the healthcare sector.

An intra-industry consortium is proposed to address some of the critical areas of AI that are central to patient safety and to build "an ecosystem of validated, transparent and health equity-oriented models with the potential for beneficial social impact."

Available by subscription or purchase, the report outlines steps that should be taken to ensure good data science – including how to build diverse teams capable of addressing the complexities of bias in healthcare AI, based on government and think tank research.

THE LARGER TREND
Some in the medical and scientific communities have pushed back on AI-driven studies that fail to share enough details about their codes and how they were tested, according to an article on the AI replication crisis in the MIT Technology Review.

The same year, Princeton University researchers released a review of scientific papers containing pitfalls. Of 71 papers related to medicine, 27 papers contained AI models with critical errors.

Some research shows that the tradeoff between fairness and efficacy in AI can be eliminated with intentional thoughtfulness in development – by defining fairness goals up front in the machine learning process.

Meanwhile, rushed AI development or deployment practices have led to overhyped performance, according to Joachim Roski, a principal in Booz Allen Hamilton's health business.

Roski spoke with Healthcare IT News prior to a HIMSS22 educational session addressing the need for a paradigm shift in healthcare AI during which he presented prominent AI failures and key design principles for evidence-based AI development.

"Greater focus on evidence-based AI development or deployment requires effective collaboration between the public and private sectors, which will lead to greater accountability for AI developers, implementers, healthcare organizations and others to consistently rely on evidence-based AI development or deployment practices," said Roski.

ON THE RECORD
Ranck, the Chilmark report's author, hosted an April podcast interview with Dr. Tania Martin-Mercado, digital advisor in healthcare and life sciences at Microsoft, about combating bias in AI. (Read our interview with Martin-Mercado here.)

Based on her findings researching race-adjusted algorithms currently in use, she said increasing developer responsibility and accountability could ultimately reduce damage to patients.

"If you are not empowering the [data] people that are creating the tools to protect patients, to protect populations, to get people involved in clinical studies, if you’re not empowering these people to make [the] change and giving them the authority to drive action, then it’s [just] performance," said Martin-Mercado.