Real-time 3D representations of surgical scenes for automating surgical robots

Join in MY580 or online (Zoom)

Efficient and effective surgery requires accurate models of the surgical scene. Surgeons must learn to create mental maps of complex 3D structures and register them to the operative field. To enable generalizable, robust automation in surgical robots, the robot also must learn to create 3D maps of the operative field from limited training data. This talk focuses on using multi-modal fusion to create 3D representations of surgical scenes. This 3D representation can improve learning efficiency of automation algorithms and generalization between cases. We use simultaneous localization and mapping to create to-scale models of surgical scenes, monocular depth estimation to update the scene in real time, and the soft-tissue models to track motion of unseen portions of the surgical scene. We have demonstrated surgical autonomy in central airway obstruction in ex vivo and cadaver models. Building interpretable signals through intermediate representations within a learning framework may also increase trust in the algorithm’s outputs and facilitate adoption and human-in-loop use.

Speaker bio

Jie Ying Wu is an assistant professor at Vanderbilt University’s Department of Computer Science. Before joining Vanderbilt, she obtained her Ph.D. from Johns Hopkins University (2021), M.Sc. from École normale supérieure Paris-Saclay (2016), and B.Sc. from Brown University (2015). Her work explores using machine learning and augmented reality techniques to enable the surgical tools to provide more active guidance. Her augmented reality work focuses on increasing collaboration in the operating room while her robotics work focuses on increasing automation and improving surgical training. As a part of this endeavor, Jie Ying improved the stability of an open-source surgical robotics platform, the da Vinci Research Kit, and laid out the framework for the next generation of that system.