Augmented Reality in Minimally Invasive Surgery

Laparoscopic surgery is a Minimally Invasive Surgical (MIS) procedure using endoscopes with small incisions to carry out internal operations on patients. A video image from a camera at the end of the endoscope provide the surgeon with visual feedback from inside the patient. While MIS offers considerable advantages over open surgeries, it also imposes challenges on surgeons’ performance due to issues associated with the field of view (FOV), hand-eye disalignment and disorientation. Augmented Reality (AR) technology can help overcome the limitations by overlaying additional information with the real scene through augmentation of target surgical locations, annotations, labels, tumour measurement or even 3D reconstruction of anatomic structures.

With our colleagues at Bournemouth University we are working to provide AR in MIS for both monocular and stereoscopic endoscopes. Our framework provides real-time computation of 3D markerless tracking through a robust feature-based simultaneous localization and mapping (SLAM) approach. An interactive geometry-aware AR environment is implemented by incrementally building a geometric mesh via zero mean normalised cross correlation (ZNCC) stereo matching.

Monocular MIS surgical scenes are particularly challenging to augment with correct spatial calibration. We are also addressing this problem so as to prevent drifting and inaccurate planar mapping.

Collaborators: Bournemouth University

Related Publications

1. Chen, L., Tang, W., John, N. W., Wan, T. R., & Zhang, J. J. (2020). De-smokeGCN: Generative Cooperative Networks for Joint Surgical Smoke Detection and Removal. IEEE Transactions on Medical Imaging, 39(5), 1615–1625. [bib]
2. Chen, L., Tang, W., John, N. W., Wan, T. R., & Zhang, J. J. (2020). Context-Aware Mixed Reality: A Learning-based Framework for Semantic-level Interaction. Computer Graphics Forum, 39(1), 484–496. [bib]
3. Chen, L., Tang, W., John, N. W., Wan, T. R., & Zhang, J. J. (2018). SLAM-based Dense Surface Reconstruction in Monocular Minimally Invasive Surgery and its Application to Augmented Reality. Computer Methods and Programs in Biomedicine, 158, 135–146. [bib]
4. Chen, L., Tang, W., & John, N. W. (2017). Real-time Geometry-Aware Augmented Reality in Minimally Invasive Surgery. Healthcare Technology Letters, Special Issue on Augmented Environments for Computer-Assisted Interventions, 4(5), 163–167. [bib]
5. Chen, L., Day, T. W., Tang, W., & John, N. W. (2017). Recent Developments and Future Challenges in Medical Mixed Reality. In Proceedings of 16th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 123–135). IEEE Computer Society. (missing reference) [bib]