Starting with electrocardiogram algorithms used in healthcare settings, Dandelion Health will run algorithms on its dataset – which spans 10 million patients – to measure the performance and bias of artificial intelligence across key racial, ethnic and geographic subgroups.

WHY IT MATTERS

Dandelion is aggregating and de-identifying clinical data through its consortium of three major health systems – Sharp HealthCare in San Diego, Sanford Health in Sioux Falls, South Dakota, and Texas Health Resources in Arlington, Texas – to validate algorithms, according to today's announcement. 

The early-stage, New York-based startup, also known as Dandelion Health Data Consortium, says it developed this no-cost program that can deliver overall performance metrics and ones for pre-specified groups of interest under a patient-privacy-first, open-access model. 

Dandelion says the first-of-its-kind service addresses the fundamental challenge of validating the performance of algorithms against intended populations – securing high-quality clinical datasets that encompass diverse populations and putting patient privacy first imposes high costs and is hard to find.

The company pledges to make it accessible to as many algorithm developers working on predictive analytics for cardiology using electrocardiogram waveforms as possible.

Any researcher or company with the legal right to do so can upload its ECG algorithm to a HIPAA- and SOC-2-compliant platform and specify the performance metrics they would like to query. 

Users will sign a contract to protect their IP address, pre-process the algorithm on sample data provided, write ~10 lines of wrapper code using a template, submit the container and confirm they are ready to validate. Dandelion will then run the algorithm on ~500,000 records, collect and calculate results, and send the report.

"When an algorithm is trained on a dataset from Cambridge or Palo Alto, and then applied on other populations – a different geography, a different care setting, or a different racial and ethnic mix of patients – the performance of the algorithm can degrade dramatically," said Elliott Green, CEO and cofounder of Dandelion Health.

"It’s critical to know how algorithms do, both overall and for the most vulnerable patient populations, who are not typically represented in algorithm training datasets. Our pilot program answers the question, does your algorithm do what it’s supposed to do? And does it do it fairly, for everyone?"

Dandelion says that the names of participating researchers and companies will never be made public and results will only be provided to algorithm developers if they grant permission to others.

It also says Sendhil Mullainathan, cofounder and advisor and the Roman Family University Professor of Computation and Behavioral Science at Chicago Booth, and Dr. Ziad Obermeyer, cofounder and chief scientific officer, and associate professor and Blue Cross of California distinguished professor at the Berkeley School of Public Health, will oversee the free AI validation program. 

Dandelion will begin accepting ECG waveform algorithms designed to be representative of the U.S. population for validation from developers worldwide, beginning July 15, for an initial period of three months. Future plans beyond the pilot include adding clinical data modalities like clinical notes and radiology imaging.

THE LARGER TREND

With the rapid growth, development and integration of AI tools in healthcare, the U.S. Food and Drug Administration, the Coalition for Health AI and others have called for methods to rigorously measure and validate the safety and effectiveness of algorithms. 

At HIMSS23, Sonya Makhni, medical director of the Mayo Clinic Platform, discussed the importance of validation and platform-based collaboration in developing and deploying effective healthcare AI. On the clinic's development platform, collaborators agree to data standards and retain ownership of their data as they partner to train and validate ecosystems, she explained.

She said with access to de-identified 10.4 million longitudinal patient records, Mayo Clinic facilitates external innovation at a much faster pace. That platform is deploying validated algorithms – such as EKG algorithms – and integrating them into the clinic's workflows.

Sharp Healthcare and Sanford Health, the largest rural health system in the United States, joined the Dandelion Health Data Consortium in August to build diversity in data science.

"People with great ideas cannot access research-quality, AI-ready data in a secure, principled way. This means that AI products that could fundamentally transform clinical care aren't getting built," Obermeyer said last year.

ON THE RECORD

"There is intense interest in algorithms that input electrocardiogram waveforms – which test the function of the heart by measuring its electrical activity – and output predictions about arrhythmias, lab values, or the results of other, more expensive tests like echocardiograms," Obermeyer said in a statement.

"Questions about data quality and patient privacy have become a global conversation among scientific and healthcare communities, governments and the public alike. The goal of our pilot program is to make it easy for developers to gain insight into how well their algorithms perform overall – and for specific groups of interest – using a scientifically rigorous validation process that protects patient privacy and is aligned with the public good."

Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.