Multimodal augmented reality for enhanced learning in robotic surgery
What Used to Be
The current training on navigated minimally invasive surgery is primarily done in a realistic clinical setup with many costly devices. Clinical procedures are then planned based on the pre-operative patient specific model using path planning and optimization theories. During the intra-operative procedures, the patient needs to be registered to the preoperative image based on registration theory and marker or marker-less techniques. After the registration, tracking and navigation systems are employed to track and guide the surgical instrument accurately with respect to the surgical targets.
Problems We Identified
Therefore, teaching the key fundamental engineering aspects of image-guided robotic interventions is a challenging task, particularly for the biomedical major students with weak mathematics background. Hands-on learning is an effective way to inspire their interest and help them to understand the insight philosophy, but this approach is limited primarily by high equipment costs and lack of accessibility. Thus, it is desirable to leverage on cost-effective and immersive multimodal augmented reality for robotic surgery training.
What We Did
The objective of the project is to develop a cost-effective and immersive hands-on educational platform – a novel multimodal augmented reality training system comprised of multiple low cost and actuation modules – for teaching the fundamental engineering theoretical aspects involved in robotic surgeries. The following key modules were developed:
1) preoperative anatomical modeling;
2) surgical instrument tracking using Kinect sensors;
3) treatment planning;
4) surgical navigation and robotic assisted executions using LEGO kits and SensAble omni-phantom.
How This Helped
The overall AR-based robotic surgery system was demonstrated for selected skull-based phantom experiments. The students did 1 mini-project and demonstrated the learning outcomes. The physical prototypes constructed by the students showed that the hands-on project had helped them to understand the underlying theories.