Skip to content
/ tapps Public

[CVPR 2024] Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations

License

MIT, MIT licenses found

Licenses found

MIT
LICENSE
MIT
LICENSE_MASK2FORMER
Notifications You must be signed in to change notification settings

tue-mps/tapps

Repository files navigation

TAPPS: Task-aligned Part-aware Panoptic Segmentation (CVPR 2024)

PWC

PWC

Code for 'Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations', Daan de Geus and Gijs Dubbelman, CVPR 2024.

Installation

Getting Started

Training

To train a model, you need a configuration file. We provide default configuration files for the models presented in our work. Our configs are designed to be trained on 4 GPUs, using a different number of GPUs likely requires changes to the learning rate.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml 

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml

Evaluation

Evaluating a model on the PartPQ metric requires two steps: (1) making and saving the predictions, (2) evaluating the predictions and calculating the PartPQ.

1. Making predictions and saving them

This step requires a configuration file and the model weights. We provide default configuration files and weights for the models presented in our work. You also need to define a directory where you wish to store the predictions.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml \
                    --eval-only \
                    MODEL.WEIGHTS /PATH/TO/MODEL/WEIGHTS.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR /PATH/WHERE/PREDICTIONS/ARE/STORED/

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml \
                    --eval-only \
                    MODEL.WEIGHTS checkpoints/tapps_pascal_r50_cocoinit.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR predictions/

2. Evaluating the predictions and calculating the PartPQ

Provide the path where the predictions are stored and specify the dataset.

python eval/eval_partpq.py --save_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                           --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Example:

python eval/eval_partpq.py --save_dir predictions/ \
                           --dataset pascal 

Visualization

To visualize the part-aware panoptic segmentation predictions that you stored (see Evaluation step 1), run:

python eval/visualize_pps.py --pred_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                             --save_dir /PATH/WHERE/VISUALIZATIONS/WILL/BE/STORED/ \
                             --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Models

Check this page for trained models and associated config files.

License

Shield: License: MIT

This code builds upon the official Mask2Former code. The majority of Mask2Former is licensed under an MIT License. However, portions of the Mask2Former project are available under separate license terms: Swin-Transformer-Semantic-Segmentation is licensed under the MIT license, Deformable-DETR is licensed under the Apache-2.0 License.

This code also uses the panoptic_parts code, which is licensed under the Apache-2.0 license.

The remaining code, specifically added for TAPPS, is licensed under an MIT license.

Citing us

Please consider citing our work if it is useful for your research.

@inproceedings{degeus2024tapps,
  title={{Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations}},
  author={{de Geus}, Daan and Dubbelman, Gijs},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

If you use the part-aware panoptic segmentation task in your work, consider citing:

@inproceedings{degeus2021pps,
  title = {{Part-aware Panoptic Segmentation}},
  author = {{de Geus}, Daan and Meletis, Panagiotis and Lu, Chenyang and Wen, Xiaoxiao and Dubbelman, Gijs},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2021},
}

Acknowledgement

This project is built on top of the official code of Mask2Former, we thank the authors for their great work and useful code.

About

[CVPR 2024] Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations

Resources

License

MIT, MIT licenses found

Licenses found

MIT
LICENSE
MIT
LICENSE_MASK2FORMER

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages