In this project, we developed an image guided robotic needle placement system using the Robot Operating System (ROS) framework. You used a depth camera (Kinect Azure by Microsoft) mounted on a robot arm (Panda by Franka Emika) to record 3D images of a chest phantom. We had to find the transformation between the robot’s endeffector and the camera with an eye-in-hand calibration. Using this transformation we're required to be able to stich the individual images to a combined scan while the robot drives the camera around the phantom. We had then register this scan to a high resolution model, obtained from computer tomography. Within the high resolution model a target for the needle will be given. We were required to perform trajectory planning to find a collision free and kinematically feasible path to the target. Lastly, exchange the camera for a needle (mock-up) and let the robot perform the insertion.
Task 1: Robot kinematics
In the final application you will need to move the robot
- for recoding of poses for the hand-eye calibration
- for scanning the chest phantom from different sides
- for the needle insertion
Task 2: Trajectory planning
In order to move the robot to a desired target it needed to send the robot a continous trajectory of joint positions.
Task 3: Camera calibration
The Kinect Azure has an RGB and a depth camera. The calibration of the Kinect had to be done so that it captures the true geometries of the world.
Task 4: Eye-in-hand calibration
In order to stich the pointclouds of the chest phantom, it is needed to know the transformation between the mounted camera (”eye”) and the robot’s end effector (”hand”)
Task 5: Model recording and registration
In order to obtain a 3D Scan of the chest phantom, the camera has to be driven around the phantom. Based on given a 3D CAD model of phantom, the model registration had to be done.
