Three-dimensional reconstruction and modeling technology is not new in hospitals. In fact, many organizations have dedicated radiology technologists that perform 3D reconstructions from cross-sectional images.

THE PROBLEM

The results of their work are typically saved back to PACS, essentially as a video of predetermined angles – often referred to as a rotational cine or turntable cine. While these "videos" are accessible to surgeons, the prescribed view and structures create an impediment to a complete understanding of the patient's morphology.

"If surgeons were able to leverage the complete PACS suite, they would be able to create their own views; however, there is a significant user-mismatch between the radiology-focused software and the surgical or interventional user," explained Justin Ryan, director and research scientist in the Webster Foundation 3D Innovations Lab at Rady Children's Hospital in San Diego.

Ryan, who is also an adjunct professor in the department of neurological surgery at the University of California at San Diego, holds a doctorate in biomedical engineering.

"Simply stated, surgeons are not typically trained how to use the totality of radiology software to best leverage the available 3D data," he added.

This challenge may be best described by a recent case, he continued.

"A cross-sectional CT scan was acquired for a patient with double outlet right ventricle anatomy as well as an aberrant airway," he recalled. "As per our institution's standard-of-care, the CT received a 3D reconstruction in radiological post-processing software. Two cines featuring the 3D reconstructions were produced: a spin and a tumble.

"The surgeon provided feedback that they wanted the same spin/tumble with the airway included; then another request came in to make the aorta partially transparent followed by additional transparent requests and unique angles," he continued. "Our team had produced over 12 cines in an effort to fulfill the surgeon's requests – where a wholistic solution would have allowed the surgeon to do all of these processes in real time."

PROPOSAL

Ryan looked to other interactive domains, especially video game development, and saw there is a focused effort on creating an easy-to-use (user interface) yet effective (user experience) application. This theory is known as UIUX.

"Our team sought to leverage real-time 3D viewing present in the video game development domain as well as UIUX principles to create a solution for surgeon and interventionalists as a primary user, not radiologists," he noted. "We were fortunate to have recruited a team member who previously worked for a video game firm. We engaged in an ideation process between our surgeons, interventionalists and this innovation engineer to settle on using the Unity platform to visualize anatomy in real time.

"Our team sought to leverage real-time 3D viewing present in the video game development domain as well as UIUX principles to create a solution for surgeon and interventionalists as a primary user, not radiologists."

Justin Ryan, Rady Children's Hospital

"In working with our stakeholders, we recognized the primary goal was to visualize 3D structures with as few impediments as possible," he continued. "To recognize this goal, our team implemented periodic focus group and pilot tests. We would give a simple prompt such as 'open the cardiac model located on the desktop, then hide the aorta and make the airway partially transparent' with no additional instructions."

The team would monitor the user and take note at each stumbling block. Perhaps the iconography was less-than-ideal or the implemented method – while making sense to the engineers on the team – ultimately didn't make sense to the intended user.

"These stumbling blocks would push our development team back to the drawing board," Ryan said. "This constant feedback from the intended user group during development was a significant benefit to our process. While medical software companies may employ a medical advisor board, we had the benefit of constant feedback from our team members."

The result is Arc 3D Model Viewer, a 3D viewing platform designed specifically for reviewing medical 3D reconstructions.

MEETING THE CHALLENGE

Ryan and the team ultimately started with the Unity platform for real-time visualization. Many people may be familiar with popular video games such as Pokémon Go, Marvel Snap and Among Us. These games were built on Unity.

"Readers may not be aware that Unity also has non-video game applications in the entertainment, education and defense industries, as these industries also have a need for real-time visualization," Ryan noted. "The Unity platform enabled us to create a solution where surgeons could interact fully with 3D datasets.

"We designed Arc 3D Model Viewer to fill the niche of real-time 3D viewing, not segmentation and reconstruction," he continued. "Our process is to leverage FDA-cleared segmentation software – we use Mimics by Materialise, but other platforms exist – to perform segmentation and 3D reconstruction. We output the final 3D model that can then be imported into Arc 3D Model."

During the development process, staff constantly engaged with their information security team as the ultimate goal was to deploy the software on hospital computers.

"As we completed a release candidate application, we provided our information security team the code to review for vulnerabilities; ultimately a third-party information security firm was also retained to provide additional review," Ryan said. "With the acceptance of our software, it could then be deployed to hospital PCs.

"Now our users can hide and show structures or virtually clip through structures in real time; this would not be possible with the status quo cines in PACS," he added. "To simplify the process further, our engineers are building in DICOM query and retrieve capabilities."

RESULTS

Ryan said the team's biggest accomplishment has been going from ideation to deployment within the organization.

"Walking into a clinic room and seeing Arc 3D Model Viewer on the computer being used to explain a patient's condition to a family has been rewarding personally," he said. "Now we are spinning off branches of our software for education and patient education. After deployment of our solution, we saw the number of 3D requests increase, which I see as a testament to the software's UIUX promise.

"We've been performing 3D reconstructions at point of care for years, but the volume greatly increased after the deployment of our solution," he noted. "When we released the not-for-diagnostic-use software publicly, it was encouraging to see the download count spike, but spike in ways we didn't anticipate."

Almost half of downloads have been from academic or medical organizations outside of the United States. It is rewarding, Ryan said, seeing potential 3D use well beyond just Rady Children's Hospital patients.

FINDING THE FREE TOOL

"We have an academic and engineering mindset," Ryan concluded. "We encourage people to reach out to us if they encounter an impediment to adoption or an idea on how to visualize 3D in novel ways. We would want to build collaborations to give more people greater control of their 3D data."

To get the Arc 3D Model Viewer, click here. To contact Ryan and his team, write to 3DPrintLab@rchsd.org.

Follow Bill's HIT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication.