Augmented Reality in Minimally Invasive Surgery

Laparoscopic surgery is a Minimally Invasive Surgical (MIS) procedure using endoscopes with small incisions to carry out internal operations on patients. A video image from a camera at the end of the endoscope provide the surgeon with visual feedback from inside the patient. While MIS offers considerable advantages over open surgeries, it also imposes challenges on surgeons' performance due to issues associated with the field of view (FOV), hand-eye disalignment and disorientation. Augmented Reality (AR) technology can help overcome the limitations by overlaying additional information with the real scene through augmentation of target surgical locations, annotations, labels, tumour measurement or even 3D reconstruction of anatomic structures.

With our colleagues at Bournemouth University we are working to provide AR in MIS for both monocular and stereoscopic endoscopes. Our framework provides real-time computation of 3D markerless tracking through a robust feature-based simultaneous localization and mapping (SLAM) approach. An interactive geometry-aware AR environment is implemented by incrementally building a geometric mesh via zero mean normalised cross correlation (ZNCC) stereo matching.

Area highlight and measurement onto the vessel walls displayed on the video feed.
Figure 1: Area highlight and measurement onto the vessel walls displayed on the video feed. [PNG]

Monocular MIS surgical scenes are particularly challenging to augment with correct spatial calibration. We are also addressing this problem so as to prevent drifting and inaccurate planar mapping.

Collaborators: Bournemouth University

Related Publications

  1. Chen, L., Tang, W., John, N. W., Wan, T. R., & Zhang, J. J. (2018). SLAM-based Dense Surface Reconstruction in Monocular Minimally Invasive Surgery and its Application to Augmented Reality. Computer Methods and Programs in Biomedicine, 158, 135–146. [bib]
  2. Chen, L., Day, T. W., Tang, W., & John, N. W. (2017). Recent Developments and Future Challenges in Medical Mixed Reality. In Proceedings of 16th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 123–135). IEEE Computer Society. (missing reference) [bib]