Abstract:

Laparoscopic liver surgery holds many advantages for patients, among other things less pain and a quicker recovery time. Since it is technically more challenging, an intraoperative navigation system should be implemented to support the surgeon. This requires a reliable segmentation of the liver in laparoscopic images. The segmentation of RGB images can be difficult, when observing similar color and texture. As this occurs frequently in laparoscopic images, we aim to include depth maps as additional data. Moreover, depth maps are unaffected by environmental disturbances like reflections, lightning or pollution of the camera lens, which were reported to be challenging for an accurate segmentation. In our work we want to show, how depth maps can be used as additional input to an artificial neural network, supporting the segmentation of the liver in laparoscopic images.The data used in this work is divided into a synthetic and a clinical dataset. The synthetic dataset consists out of 20,000 image-depth pairs from ten patients. The clinical dataset is smaller, containing 525 image-depth pairs, from five patients. It is important to mention, that the depth maps of the synthetic dataset are ground truth depth maps, whereas the clinical depth maps are estimated by an artificial neural network. There is a visible gap between them, which was attempted to bridge by different pre-processing methods.Both datasets were utilized for the training and testing of an artificial neural network, more precisely the DFormer, which is an encoder-decoder network with a backbone pre-trained on RGB-D data and two attention fusion modules.In the first experiment the network is trained on synthetic data and tested on clinical data, through what the transferability of the weights from the synthetic to clinical data is explored. The second experiment investigates the changes, when fine-tuning the synthetic weights on clinical data and performing a 5-fold cross-validation. Lastly, experiment 3 and 4 analyse the impact of depth maps on the prediction by using ideal and non-informative clinical depth maps for experiment 3 and noisy synthetic depth maps for experiment 4.Through the four performed experiments we have found, that depth maps can be helpful with imprecise RGB images, as they are able to compensate the quality of the laparoscopic images. Moreover, we have shown, that the quality of the prediction is dependent on the quality of the depth maps. Nonetheless, even if we achieved interesting and promising results as described beforehand, we have found limitations due to the quality of the depth maps and the limited available data, on particular laparoscopic scenarios, for example incisions of the liver. Through these limitations, future opportunities and challenges arise, including further experiments, the optimization of depth maps and the increase of data.