Artificial intelligence tutoring outperforms expert instructors in neurosurgical training

The COVID-19 pandemic has presented both challenges and opportunities for medical training. Remote learning technology has become increasingly important in several fields. A new study finds that in a remote environment, an artificial intelligence (AI) tutoring system can outperform expert human instructors.

The Neurosurgical Simulation and Artificial Intelligence Learning Centre at The Neuro (Montreal Neurological Institute-Hospital) recruited seventy medical students to perform virtual brain tumour removals on a neurosurgical simulator. Students were randomly assigned to receive instruction and feedback by either an AI tutor or a remote expert instructor, with a third control group receiving no instruction.

An AI-powered tutor called the Virtual Operative Assistant (VOA) used a machine learning algorithm to teach safe and efficient surgical technique and provided personalized feedback, while a deep learning Intelligent Continuous Expertise Monitoring System (ICEMS) and a panel of experts assessed student performance.

In the other group, remote instructors watched a live feed of the surgical simulations and provided feedback based on the student’s performance.

The researchers found that students who received VOA instruction and feedback learned surgical skills 2.6 times faster and achieved 36 per cent better performance compared to those who received instruction and feedback from remote instructors. And while researchers expected students instructed by VOA to experience greater stress and negative emotion, they found no significant difference between the two groups.

Surgical skill plays an important role in patient outcomes both during and after brain surgery. VOA may be an effective way to increase neurosurgeon performance, improving patient safety while reducing the burden on human instructors.

“Artificially intelligent tutors like the VOA may become a valuable tool in the training of the next generation of neurosurgeons,” says Dr. Rolando Del Maestro, the study’s senior author. “The VOA significantly improved expertise while fostering an excellent learning environment. Ongoing studies are assessing how in-person instructors and AI-powered intelligent tutors can most effectively be used together to improve the mastery of neurosurgical skills.”

“Intelligent tutoring systems can use a variety of simulation platforms to provide almost unlimited chances for repetitive practice without the constraints imposed by the availability of supervision,” says Ali Fazlollahi, the study’s first author. “With continued research, increased development, and dissemination of intelligent tutoring systems, we can be better prepared for ever-evolving future challenges.”

This study, published in the Journal of the American Medical Association (JAMA Network Open) on Feb. 22, 2022, was funded by the Franco Di Giovanni Foundation, the Royal College of Physicians and Surgeons of Canada, and the Brain Tumour Foundation of Canada Tumour Research Grant along with The Neuro. Cognitive assessment was led by Dr. Jason Harley at McGill University’s Department of Surgery.

The Neuro

The Neuro – The Montreal Neurological Institute-Hospital – is a bilingual, world-leading destination for brain research and advanced patient care. Since its founding in 1934 by renowned neurosurgeon Dr. Wilder Penfield, The Neuro has grown to be the largest specialized neuroscience research and clinical center in Canada, and one of the largest in the world.

The seamless integration of research, patient care, and training of the world’s top minds make The Neuro uniquely positioned to have a significant impact on the understanding and treatment of nervous system disorders.

In 2016, The Neuro became the first institute in the world to fully embrace the Open Science philosophy, creating the Tanenbaum Open Science Institute. The Montreal Neurological Institute is a McGill University research and teaching institute. The Montreal Neurological Hospital is part of the Neuroscience Mission of the McGill University Health Centre. For more information, please visit www.theneuro.ca

Story source:

McGill University

Journal reference

Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students

Abstract

Importance

To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches.

Objective

To determine how VOA and remote expert instruction compare in learners’ skill acquisition, affective, and cognitive outcomes during surgical simulation training.

Design, Setting, and Participants

This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021.

Interventions The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback.

Main Outcomes and Measures The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, −1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports.

Results

A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group.

All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001).

Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups.

However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found.

Conclusions and Relevance

In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training.