Skip to content

Latest commit

 

History

History
127 lines (88 loc) · 5.74 KB

README.md

File metadata and controls

127 lines (88 loc) · 5.74 KB
Raw

TAPPS: Task-aligned Part-aware Panoptic Segmentation (CVPR 2024)

[Project page] [Paper]

PWC

PWC

Code for 'Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations', Daan de Geus and Gijs Dubbelman, CVPR 2024.

Installation

Getting Started

Training

To train a model, you need a configuration file. We provide default configuration files for the models presented in our work. Our configs are designed to be trained on 4 GPUs, using a different number of GPUs likely requires changes to the learning rate.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml

Evaluation

Evaluating a model on the PartPQ metric requires two steps: (1) making and saving the predictions, (2) evaluating the predictions and calculating the PartPQ.

1. Making predictions and saving them

This step requires a configuration file and the model weights. We provide default configuration files and weights for the models presented in our work. You also need to define a directory where you wish to store the predictions.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml \
                    --eval-only \
                    MODEL.WEIGHTS /PATH/TO/MODEL/WEIGHTS.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR /PATH/WHERE/PREDICTIONS/ARE/STORED/

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml \
                    --eval-only \
                    MODEL.WEIGHTS checkpoints/tapps_pascal_r50_cocoinit.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR predictions/

2. Evaluating the predictions and calculating the PartPQ

Provide the path where the predictions are stored and specify the dataset.

python eval/eval_partpq.py --save_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                           --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Example:

python eval/eval_partpq.py --save_dir predictions/ \
                           --dataset pascal

Visualization

To visualize the part-aware panoptic segmentation predictions that you stored (see Evaluation step 1), run:

python eval/visualize_pps.py --pred_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                             --save_dir /PATH/WHERE/VISUALIZATIONS/WILL/BE/STORED/ \
                             --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Models

Check this page for trained models and associated config files.

License

Shield: License: MIT

This code builds upon the official Mask2Former code. The majority of Mask2Former is licensed under an MIT License. However, portions of the Mask2Former project are available under separate license terms: Swin-Transformer-Semantic-Segmentation is licensed under the MIT license, Deformable-DETR is licensed under the Apache-2.0 License.

This code also uses the panoptic_parts code, which is licensed under the Apache-2.0 license.

The remaining code, specifically added for TAPPS, is licensed under an MIT license.

Citing us

Please consider citing our work if it is useful for your research.

@inproceedings{degeus2024tapps,
  title={{Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations}},
  author={{de Geus}, Daan and Dubbelman, Gijs},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

If you use the part-aware panoptic segmentation task in your work, consider citing:

@inproceedings{degeus2021pps,
  title = {{Part-aware Panoptic Segmentation}},
  author = {{de Geus}, Daan and Meletis, Panagiotis and Lu, Chenyang and Wen, Xiaoxiao and Dubbelman, Gijs},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2021},
}

Acknowledgement

This project is built on top of the official code of Mask2Former, we thank the authors for their great work and useful code.