Marker-less real-time intra-operative camera and hand-eye calibration procedure for surgical augmented reality

[摘要] Accurate medical Augmented Reality (AR) rendering requires two calibrations, a camera intrinsic matrix estimation and a hand-eye transformation. We present a unified, practical, marker-less, real-time system to estimate both these transformations during surgery. For camera calibration we perform calibrations at multiple distances from the endoscope, pre-operatively, to parametrize the camera intrinsic matrix as a function of distance from the endoscope. Then, we retrieve the camera parameters intra-operatively by estimating the distance of the surgical site from the endoscope in less than 1 s. Unlike in prior work, our method does not require the endoscope to be taken out of the patient; for the hand-eye calibration, as opposed to conventional methods that require the identification of a marker, we make use of a rendered tool-tip in 3D. As the surgeon moves the instrument and observes the offset between the actual and the rendered tool-tip, they can select points of high visual error and manually bring the instrument tip to match the virtual rendered tool tip. To evaluate the hand-eye calibration, 5 subjects carried out the hand-eye calibration procedure on a da Vinci robot. Average Target Registration Error of approximately 7mm was achieved with just three data points.

[学科分类] 肠胃与肝脏病学

[关键词] medical robotics;medical image processing;cameras;endoscopes;calibration;robot vision;surgery;phantoms;rendering (computer graphics);augmented reality;image registration;biomedical optical imaging;hand-eye calibration procedure;surgical augmented reality;pre-operative medical data;camera intrinsic matrix estimation;hand-eye transformation;camera calibration;endoscope;high visual error;virtual rendered tool tip;marker-less real-time intra-operative camera;augmented reality rendering;subsequent gradient descent steps;da Vinci robot;average target registration error;prostate phantom