A robot, for the first time trained by observing videos of experienced surgeons, successfully performed surgical tasks with a level of skill matching human doctors.
This achievement, using imitation learning, represents a breakthrough in surgical robotics by eliminating the need to manually program each precise movement required in surgery. It brings robotic surgery closer to full autonomy, where robots could eventually conduct complex surgeries independently of human intervention.
The research, led by Johns Hopkins University, is being showcased this week at the Conference on Robot Learning in Munich, a premier event in the fields of robotics and machine learning.
“It’s incredible—we feed the model just camera input, and it predicts the robotic movements needed for surgery,” said senior author Axel Krieger, assistant professor in JHU’s Department of Mechanical Engineering. “This is a major step toward a new frontier in medical robotics.” The team, including researchers from Stanford University, used imitation learning to train a da Vinci Surgical System robot to complete three core tasks in surgery: needle manipulation, tissue lifting, and suturing. In each task, the robot trained with this model performed as skillfully as a human surgeon.
The model integrates imitation learning with the machine learning architecture that powers language models like ChatGPT. However, instead of language, this model operates in the “robotic language” of kinematics, breaking down robotic motion into precise mathematical angles.
To train the model, the team utilized hundreds of videos from wrist cameras on da Vinci robots during surgical procedures worldwide. These recordings, used by surgeons for post-operative reviews and stored in extensive archives, provided a rich data set of real-world examples for the robot to “imitate.” Currently, almost 7,000 da Vinci systems are in use globally, with over 50,000 trained surgeons contributing to this valuable archive.
Although the da Vinci system is widely used, it is known to lack precision. The researchers overcame this by training the model to perform relative movements rather than absolute actions, allowing for adjustments when input data is imperfect.
"All we need is visual input, and this AI system figures out the correct action,” explained lead author Ji Woong “Brian” Kim, a postdoctoral researcher at Johns Hopkins. “With just a few hundred examples, the model learns the procedure and can adapt to new environments it hasn’t seen.” Krieger added, “The model is so capable that it can handle unexpected situations—if it drops the needle, it automatically picks it up and keeps going. That’s not something we explicitly taught it to do.”
This approach could make it possible to quickly train a robot for nearly any surgical procedure. The team is now expanding this imitation learning to train a robot to perform entire surgeries, not just isolated tasks.
Previously, programming a robot for even a single surgical task required painstakingly hand-coding each movement. Krieger noted that developing a model for suturing one type of surgery could take years.
“This is so much more efficient,” said Krieger. “By collecting imitation data from various procedures, we can train a robot in just a few days. This advancement accelerates our journey toward autonomous surgery, aiming for greater accuracy and fewer medical errors.”
The Johns Hopkins research team includes PhD student Samuel Schmidgall, Associate Research Engineer Anton Deguet, and Associate Professor of Mechanical Engineering Marin Kobilarov. Stanford’s team includes PhD student Tony Z. Zhao and Assistant Professor Chelsea Finn.
Validate your login
Sign In
Create New Account