Suggestion on batch processing & how to estimate the scale and orientation components of local features

[GabbySuwichaya]

Hello! Thank you very much for releasing the implementation of this work.
The runtime & performance are quite impressive.

• Here, I would like to ask for your suggestion on how to extend this work. Actually, do you have a version of this work for batch processing over multiple images at the same time? Is it possible?

• Also, could you please recommend how to estimate the scale and orientation components of local features? I have actually used the SuperPoint as the local feature. I am not sure how to get these two components ?

[22wei22]

Hello! Thank you very much for releasing the implementation of this work. The runtime & performance are quite impressive.

• Here, I would like to ask for your suggestion on how to extend this work. Actually, do you have a version of this work for batch processing over multiple images at the same time? Is it possible?
• Also, could you please recommend how to estimate the scale and orientation components of local features? I have actually used the SuperPoint as the local feature. I am not sure how to get these two components ?

@GabbySuwichaya
I want to use the SuperPoint as the local feature. I meet the same thing how to get scale and orientation. do you have some idea?

[cavalli1234]

Hi @GabbySuwichaya, hi @22wei22

Let me comment point by point:

• I don't have other versions of this work, and this does not support processing multiple images together in batches. It is a feasible extension, and could be useful in the case when you have lots of images with few correspondences, so that you can fit all of them into a single batch in GPU. Implementation-wise, if you wish to proceed to such extension, it could be made by adding another layer of indexing (currently each local neighborhood is indexed, you can also index each image in batch to distinguish them while keeping variable-length vectors to compute). It's a bit tricky to code, but surely feasible. However, from my personal taste, I would rather suggest to extend it in some direction which could be interesting research-wise. Examples of open questions could be:
  • Can we make the selection more robust to noisy correspondences, to alleviate the need for orientation+scale as extra filters?
  • Can we add some meaningful learned component to catch some complex correlations on top of affine regularities?
  • How can we go beyond the first-nearest-neighbor matches to alleviate the need for good descriptors?
• Regarding scale and orientation, these don't always come defined with a given keypoint. When they are not defined like with SuperPoint, my best suggestion could be to try to estimate them like SIFT does, and emulate them. They will not be exactly the same, since scale is intrinsically connected with the way SIFT selects where to pick keypoints, but they might be good estimates. In AdaLAM you can also change the confidence you give to the scale and orientation filters, so it might be the case to adjust those confidences for emulated scale and orientation. Unfortunately I don't have ready-made code for this, but I think you should be able to find something online, worst case there's a lot of documentation on how SIFT works. You can also use AdaLAM without scale and orientation, but from my experience (also with SuperPoint) it works far from optimally.

In any case, thank you a lot for your interest and let me know if you plan to work on an extension of AdaLAM!

[ducha-aiki]

@GabbySuwichaya @22wei22 I can advice something for SuperPoint.

• Orientation. It is probably safe to assume all zero -- otherwise SP would probably match them. If that does not suit you, see the recommendation for the scale.

• Scale. I would recommend to start with small fixed scale, e.g. 32 pix, extract patches around SP keypoints and feed them into https://github.com/bluedream1121/self-sca-ori Thus you will have a scale and orientation. I am going to integrate the methods I mentioned into kornia
  Alternatively, you can estimate orientation using OriNet from kornia

However, if you are using SuperPoint, it is much better to use SuperGlue with it instead of AdaLAM. And for AdaLAM I would recommend KeyNetHardNet instead