The pivotal trial data that led to the clearance of the first AI-based diagnostic system to not require clinician interpretation was published on Aug. 28 by nPJ Digital.

The 900-patient investigation showed that the IDx-Dr system for autonomous detection of diabetic retinopathy "robustly exceeded the pre-specified primary endpoint goals" and can be realistically deployed and used by non-specialist providers.

"The AI system's primary role is to identify those people with diabetes who are likely to have diabetic retinopathy that requires further evaluation by an eye-care provider," Michael Abràmoff, MD, IDx founder and president, said in a statement.

"The study results demonstrate the safety of autonomous AI systems to bring specialty-level diagnostics to a primary care setting, with the potential to increase access and lower cost," he added.

While these results were enough for the FDA to grant its approval, an accompanying editorial raised a few warning flags over the researcher's robustness claims.

Penned by Pearse A. Keane, MD, a clinician scientist at the National Institute for Health Research (NIHR), and Eric J. Topol, MD, director of the Scripps Translational Science Institute and EVP at Scripps Research Institute, it highlighted sampling limitations, protocol decisions and specific clinical limitations that will need to be addressed in future validation studies.

However, they still concluded the study is "undoubtedly a milestone and an important benchmark for future research."

In July, the University of Iowa Healthcare became the first healthcare organization to implement IDx-DR.

“Early detection of diabetic retinopathy is an essential component of comprehensive diabetes care," said Dale Abel, MD, director of the division of endocrinology and metabolism at UI Health Care, in a statement. "This innovation further strengthens our ability to provide state-of-the-art care for our patients with diabetes."

IDx-DR's algorithm analyzes images taken with the Topcon NW400 retinal camera and uploaded to a cloud server. Within minutes, the software provides doctors with a binary result, either indicating that more than mild diabetic retinopathy is present and that the patient should be referred to an eye care professional, or that the screen is negative and should be repeated in 12 months.

Notably, since the system's results do not require clinician interpretation, it can be used by providers who aren't normally involved with patients' eye care, according to the agency.

To test the system, Abràmoff and colleagues collected the retinal images of 900 diabetes patients from 10 U.S. sites. These images, taken by AI-assisted primary care staff, were fed into IDx-DR's algorithm for diagnosis.

The researchers compared the algorithm's conclusions to those reached by experienced, Wisconsin Fundus Photograph Reading Center (FPRC)-certified retinal photographers who took their own images and evaluated each case using the FPRC's gold standard grading system.

Among the 900 participants, 892 completed all of the study procedures, 852 received an FPRC diagnosis, and 819 received both an FPRC and IDx-DR diagnosis. Among the first group, 198 patients received a mild diabetic retinopathy diagnosis using the gold standard, while IDx-DR's algorithm diagnosed 173. Of the 621 determined by FPRC imaging not to have the disease, the AI corroborated this conclusion on 556 patients.

These two rates translated to 87.2 percent sensitivity and 90.7 percent specificity. The system also demonstrated an imageability rate of 96.1 percent, and according to the researchers, it was well integrated into the primary care setting.

"Early detection of retinopathy is an important part of managing care for the millions of people with diabetes. Yet many patients with diabetes aren’t adequately screened for diabetic retinopathy, since about 50 percent of them do not see their eye doctor on a yearly basis," Malvina Eydelman, MD, director of the Division of Ophthalmic and Ear, Nose, and Throat Devices at the FDA's Center for Devices and Radiological Health, said in a statement announcing the clearance.

"Today's decision permits the marketing of a novel artificial intelligence technology that can be used in a primary care doctor's office," she added. "The FDA will continue to facilitate the availability of safe and effective digital health devices that may improve patient access to needed healthcare."

Some peers are skeptical

However groundbreaking the technology at hand may appear to be, the investigation into its validity wasn't above critique.

In their accompanying editorial, Keane and Topol stressed that Abràmoff and colleagues' patient sample was relatively small for a diagnostic accuracy study and instituted preferential recruitment to bolster its numbers.

Along with concerns that 40 participants excluded from the analysis could have a significant impact on the study's final findings, they wondered how the tool's sole capacity for diabetic neuropathy diagnosis would realistically fit scenarios where patients could present with a number of other, more severe retinal conditions.

"There is also the question as to whether the now-approved device will have significant uptake in the clinic. Besides the expense, it remains to be determined how and where it would be implemented," they wrote. "Will primary care clinics incorporate retinal screening into their practice? This is not really an 'autonomous system' since someone needs to acquire the image – who will perform that?"

Keane and Topol did note that the FDA's De Novo premarket review pathway, by necessity, allows a bit more leeway than the standard 510(k) clearance. Keeping in mind that future investigations are likely to face more stringent inquiry prior to regulatory approval, they applauded the study as a much-need trailblazer for what's to come in AI diagnosis technology.

"While it is always easy to be critical of studies that forge new ground, it is important to applaud the authors for this pivotal work,” they wrote. “Although deep learning will not be a panacea, it has huge potential in many clinical areas where high dimensional data is mapped to a simple classification and for which datasets are potentially stable over extended periods.”

"As such, it will be incumbent on healthcare professionals to become more familiar with this and other AI technologies in the coming years to ensure that they are used appropriately," they concluded. "This study represents an important first step in that direction."

Of note, Keane reported a consulting relationship with DeepMind, which earlier this month released its own eye disease diagnosis study. Topol holds an advisory position at Verily Life Science.

Twitter: @dave_muoio
Email the writer: dave.muoio@himssmedia.com