Researchers at New York University's Langone Health academic medical center developed a large language model, now deployed at three of its hospitals, that predicts a patient’s risk of 30-day readmission and other clinical outcomes.

WHY IT MATTERS

Coinciding with its study published in Nature this week, the release of the NYUTron model's code base in GitHub enables other healthcare organizations to train their own LLM and provide doctors with insights that could help them identify which patients may require intervention to reduce readmissions.

The model has been used to evaluate 50,000 patients discharged from NYU’s healthcare system. NYUTron shares its predictions of readmission risk with physicians by email. 

NYU partnered with NVIDIA to develop and run its LLM on several of the company's artificial intelligence platforms – its stack, library and software.

"Not all hospitals have the resources to train a large language model from scratch in-house, but they can adopt a pretrained model like NYUTron and then fine-tune it with a small sample of local data using GPUs in the cloud," said Dr. Eric Oermann, assistant professor of neurosurgery, radiology, and data science at NYU Langone Health, said in a blog post on NVIDIA's website.

"That’s within reach of almost everyone in healthcare," he said

"Oermann’s team found that after pretraining their LLM, fine-tuning it onsite with a specific hospital’s data helped to significantly boost accuracy," NVIDIA said in its announcement.

NYUTron was pretrained on 10 years of health records from NYU Langone Health, which contain more than four billion words of clinical notes representing nearly 400,000 patients. Oermann said the team is using medium-sized models trained on highly refined data to accomplish healthcare-specific tasks.

The team then developed four other algorithms that predict the length of a patient’s hospital stay, the likelihood of in-hospital mortality and the chances a patient’s insurance claims may be denied.

In a webinar last year, researchers said their approach was to treat readmission prediction as a natural language processing task involving the creation of an LLM pretrained on a health-system-scale corpus of clinical text on high-end multi-node GPU servers. 

As NYUTron was developed, they said they aimed to address the following questions: 

• How to handle long sequence length?
• How to address label imbalance?
• How to assess the impact of noisy labels on model evaluation?

First, they "collected a vast set of unlabelled clinical notes and five task-specific labeled clinical notes from the NYU Langone EHR," before training the model, refining, deploying and testing it in a real-world environment, the researchers said in the report. 

"On small samples, NYUTron was competitive with a small group of physicians at predicting 30-day readmission," they said. 

Testing a group of six physicians at different levels of seniority against NYUTron in a head-to-head comparison, they said they established a baseline difficulty for predicting all-cause readmission at the time of discharge.

"Median physician performance was worse than that of NYUTron," researchers said.

"For physicians and NYUTron, the median false positive rate was 11.11%, whereas the median true positive rate was 50% for physicians compared with 81.82% for NYUTron. Physicians had a median F1 score [a machine learning evaluation metric that measures a model's accuracy] of 62.8% and substantial variance of 22.2% compared with NYUTron, which had a median F1 score of 77.8%."

At an advance press briefing NVIDIA on Tuesday, Oermann also said that NYU Langone Health is looking at licensing its models for those organizations that do not have the resources to do a build from scratch.

The next phase for Oermann’s team is a planned clinical trial to test whether interventions based on NYUTron’s analyses reduce readmission rates.

THE LARGER TREND

Research published in Journal of Multidisciplinary Healthcare last year said that nearly 15% of all hospital patients are readmitted to the hospital within 30 days after initial discharge.

Readmission rates are affected by countless variables – including five common treatments at emergency departments – that not only impact a patient's overall care, but they can also divert beds and resources from patients that may have more intensive healthcare needs.

"Industry research as well as our own experience indicates as many as 20% of readmissions may be preventable," said Teresa Radford, clinical program coordinator at University of Virginia Health.

She told Healthcare IT News in December that after finding that UVA Health's 30-day readmission rate for patients with complex and costly medical conditions was as high as 17% to 18% per year, the healthcare provider created a hospital-at-home program that reduced hospitalizations and readmissions by 46%.

A multidisciplinary team of physicians, nurses and mental health professionals created an intensive care plan for every individual that had at least one chronic disease or behavioral condition that contributed to frequent UVA Health hospital visits and high utilization of the hospital-based services. 

ON THE RECORD

"While there have been computational models to predict patient readmission since the 1980s, we’re treating this as a [NLP] task that requires a health system-scale corpus of clinical text," Oermann said in the announcement. 

"We trained our LLM on the unstructured data of electronic health records to see if it could capture insights that people haven’t considered before."

Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.