MMDet MaskRCNN ResNet50/SwinTransformer Decouple

[eugene123tw]

Summary

• Moved MMDet MaskRCNN
• Moved MMDet ResNet50
• Moved MMDet SwinTransformer
• Moved MMDeploy MaskRCNN components

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[harimkang]

At first glance, it looks like you're importing and replacing modules.
@jaegukhyun Can you take a look to make sure there's no duplication with detection-related tasks?

Will the MaskRCNN in src/otx/algo/instance_segmentation/maskrcnn.py, which is used as the main, be replaced with OTXInstanceSegModel instead of MMDetInstanceSegCompatibleModel in another PR?

[jaegukhyun]

I have one question during review.
Will MaskRCNN family be exported using NativeExporter or MMDeployExporter?

[eugene123tw]

At first glance, it looks like you're importing and replacing modules. @jaegukhyun Can you take a look to make sure there's no duplication with detection-related tasks?

Will the MaskRCNN in src/otx/algo/instance_segmentation/maskrcnn.py, which is used as the main, be replaced with OTXInstanceSegModel instead of MMDetInstanceSegCompatibleModel in another PR?

Hi @harimkang, maybe I need to clarify some points. I've transferred all the essential MaskRCNN code and backbone code from MMDet to OTX 2.x, reflecting a similar approach with MMDet detection models or the mmcv conv/norm module you did. Also, reuse as many components as Jaeguk moved.

I'm not sure of the rationale behind replacing MMDetInstanceSegCompatibleModel with OTXInstanceSegModel. As far as I understand, we still require MMXCompatibleModel classes to execute mmlab models on OTX 2.x without installing MMDet. I believe this PR aims to enable Geti MMDet models in OTX without having to install mmdet or mmcv.

[jaegukhyun]

I think separate mmdet folder has pros and cons. My opinion is we don't need to specify that all of algorithms come from mmdet. @harimkang What do you think?

[eugene123tw]

I have one question during review. Will MaskRCNN family be exported using NativeExporter or MMDeployExporter?

@jaegukhyun Currently, it relies on MMDeployExporter. However, if we intend to remove mmdeploy dependency, this action should be addressed in a separate PR due to its large scope.

[harimkang]

Hi @harimkang, maybe I need to clarify some points. I've transferred all the essential MaskRCNN code and backbone code from MMDet to OTX 2.x, reflecting a similar approach with MMDet detection models or the mmcv conv/norm module you did. Also, reuse as many components as Jaeguk moved.

I'm not sure of the rationale behind replacing MMDetInstanceSegCompatibleModel with OTXInstanceSegModel. As far as I understand, we still require MMXCompatibleModel classes to execute mmlab models on OTX 2.x without installing MMDet. I believe this PR aims to enable Geti MMDet models in OTX without having to install mmdet or mmcv.

Yes, I think we're on the same page. But doesn't MMDetInstanceSegCompatibleModel also have a dependency on mm after all, and if we want to use it without installing mm, shouldn't eventually make sure that existing models don't use the MMDetInstanceSegCompatibleModel class in MaskRCNN?

Or are you saying you're going to leave the old model alone and add a new recipe (without mm)?

[eugene123tw]

Hi @harimkang, maybe I need to clarify some points. I've transferred all the essential MaskRCNN code and backbone code from MMDet to OTX 2.x, reflecting a similar approach with MMDet detection models or the mmcv conv/norm module you did. Also, reuse as many components as Jaeguk moved.
I'm not sure of the rationale behind replacing MMDetInstanceSegCompatibleModel with OTXInstanceSegModel. As far as I understand, we still require MMXCompatibleModel classes to execute mmlab models on OTX 2.x without installing MMDet. I believe this PR aims to enable Geti MMDet models in OTX without having to install mmdet or mmcv.

Yes, I think we're on the same page. But doesn't MMDetInstanceSegCompatibleModel also have a dependency on mm after all, and if we want to use it without installing mm, shouldn't eventually make sure that existing models don't use the MMDetInstanceSegCompatibleModel class in MaskRCNN?

Or are you saying you're going to leave the old model alone and add a new recipe (without mm)?

@harimkang MMDetInstanceSegCompatibleModel is a bridge for training MMDet models. With the decoupling of MMDet MaskRCNN in this PR, we also need to untangle data structures (such as BitmapMasks and PolygonMasks) and the instance segmentation data pipeline. However, this will require a separate PR to avoid overwhelming the current one.

Of course, there are still several mm-related components, including mmcv opset, mmdet data structures, mmdeploy, and mmengine, that still remain. It's possible to migrate all these components to OTX 2.x, it's also time-consuming. Given the scope, this migration cannot be achieved in a single PR.

Looking ahead, for torchvision.MaskRCNN, a new recipe will be necessary. And, to maintain backward compatibility, we can continue utilizing MMDetInstanceSegCompatibleModel until we find a solution for full compatibility between torchvision.MaskRCNN and MMDet MaskRCNN.

[harimkang]

Of course, there are still several mm-related components, including mmcv opset, mmdet data structures, mmdeploy, and mmengine, that still remain. It's possible to migrate all these components to OTX 2.x, it's also time-consuming. Given the scope, this migration cannot be achieved in a single PR.

Looking ahead, for torchvision.MaskRCNN, a new recipe will be necessary. And, to maintain backward compatibility, we can continue utilizing MMDetInstanceSegCompatibleModel until we find a solution for full compatibility between torchvision.MaskRCNN and MMDet MaskRCNN.

Yes I understand, thanks for the clarification! :)