visual feature selection for visual servo for multiple detected objects

[shrutichakraborty]

Hello! I am working on a problem where I would like to detect mutiple diefferent objects and visual servo to each one after the other. In this case what vidual features would you recommend to use? I would like to position the robot arm perpendicular to the centre of each detected bounding box of my detected object. Using a detection network such as Yolov8 I am able to get the bounding box or perhaps even segmentation mask of the desired object to track and servo, would you say that using the centre of gravity and the depth would be sufficient? Is it possible to combine different types of features and then compute the interaction matrix? Any code examples/suggestions would be great ! Thank you!

[fspindle]

I would suggest to implement a position based visual servo where with the depth you can estimate the 3D position of the center of the bounding box. For the orientation you can use the depth to estimate the plane of the surface where the object is located.

[shrutichakraborty]

Thank you! I am trying to write a code to do the above. In this case when you say that we can use the depth to estimate the plane of the surface would I use vpFeautureDepth. Using the example on this page :

#include <visp3/visual_features/vpFeatureDepth.h>
#include <visp3/vs/vpServo.h>
 
int main()
{
  [vpServo](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html) task; // Visual servoing task
 
  [vpFeatureDepth](https://visp-doc.inria.fr/doxygen/visp-daily/classvpFeatureDepth.html) [s](https://visp-doc.inria.fr/doxygen/visp-daily/classvpBasicFeature.html#a11fd7153b97dc170c020b1927ee6a8d6); //The current point feature.
  //Set the current parameters x, y, Z and the desired depth Zs
  double x;   // You have to compute the value of x.
  double y;   // You have to compute the value of y.
  double Z;   // You have to compute the value of Z.
  double Zs;  // You have to define the desired depth Zs.
  //Set the point feature thanks to the current parameters.
  [s](https://visp-doc.inria.fr/doxygen/visp-daily/classvpBasicFeature.html#a11fd7153b97dc170c020b1927ee6a8d6).buildfrom(x, y, Z, log(Z/Zs));
 
  // Set eye-in-hand control law.
  // The computed velocities will be expressed in the camera frame
  task.[setServo](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#a7a4a10cd84ab77868ae35167a05a340b)([vpServo::EYEINHAND_CAMERA](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#a2a92ea050512c23e38b84d5fefd56b07a631f45d50b3e36d024c0b1d90a7e04ab));
  // Interaction matrix is computed with the desired visual features sd
  task.[setInteractionMatrixType](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#a29cd059011a841f177094f4207212dbe)([vpServo::CURRENT](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#ae60923730a0ffde59cdc66dba7aec72aaa9166a8c30c2ea3d66523df1071fb7a9));
 
  // Add the 3D depth feature to the task
  task.[addFeature](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#a488b73bde3d15cd2296bd933417179ba)([s](https://visp-doc.inria.fr/doxygen/visp-daily/classvpBasicFeature.html#a11fd7153b97dc170c020b1927ee6a8d6)); // s* is here considered as zero
 
  // Control loop
  for ( ; ; ) {
    // The new parameters x, y and Z must be computed here.
 
    // Update the current point visual feature
    [s](https://visp-doc.inria.fr/doxygen/visp-daily/classvpBasicFeature.html#a11fd7153b97dc170c020b1927ee6a8d6).buildfrom(x, y, Z, log(Z/Zs));
 
    // compute the control law
    [vpColVector](https://visp-doc.inria.fr/doxygen/visp-daily/classvpColVector.html) v = task.[computeControlLaw](https://visp-doc.inria.fr/doxygen/visp-daily/classvpServo.html#ad56bcea0ee335df15a431b263ee82d8f)(); // camera velocity
  }

Would the x,y,z here correspond to the 3D centre of the bounding box? and the desired depth a few centimeters above to ensure the object is still in the frame?
I also looked at the tutorial-pose-from-planar-object.cpp file and it seems like that is similar to what I'd like to d o except rather than using a defined model I'd like to use the corners of my bounding box but I am not sure about how to define other keypoints?

It would be really helpful to know your suggestions on which modules/examples I could use/adapt? Thanks a lot!