From detection to grasp: solutions for challenges in autonomous robotic surgical instrument handling

The global healthcare sector faces a critical shortage of healthcare workers, posing a substantial threat to the sustainability of modern healthcare systems. Despite various socioeconomic measures, these efforts have seen limited success, prompting increased interest in automation to alleviate medical staff workloads. One promising approach is the development of robotic scrub nurses (RSNs), autonomous surgical assistants in charge of surgical instrument handling. Despite commendable efforts, significant challenges continue to hinder RSN implementation, including the need for large datasets of annotated images to train AI-based detectors, unreliable tool localization performance, and the absence of a versatile gripper that can securely handle various surgical instruments. This dissertation proposes solutions that address these key challenges to help enable RSNs to effectively perform instrument detection, localization, and grasping. The primary contributions of this work include: 1) a novel data augmentation method based on a limited number of manually annotated images to improve detection performance and generalization with minimal annotation effort, 2) a multi-view voting approach for improved tool localization by filtering out detection errors, and 3) the design of a hybrid gripper based on granular jamming technology, capable of securely grasping a wide range of instruments while promoting compatibility with human collaboration. Experimental results demonstrated the efficacy of the proposed solutions, showing high detection performance achieved through data augmentation, a significant reduction in localization errors with multi-view aggregation, and reliable performance of the hybrid gripper in handling diverse surgical instruments. These advancements represent a significant step forward in RSN development, offering the potential to enhance surgical efficiency and help mitigate the impact of the healthcare workforce shortage.