SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images

Make OVSS possible in remote sensing contexts

Kaiyu Li¹  Ruixun Liu¹  Xiangyong Cao^✉1  Xueru Bai²  Feng Zhou²  Deyu Meng¹  Zhi Wang¹ 

¹Xi'an Jiaotong University  ²Xidian University 

• [Project] • [arXiv] • [Colab] •

Visualization and performance of SegEarth-OV on open-vocabulary semantic segmentation of remote sensing images. We evaluate on 17 remote sensing datasets (including semantic segmentation, building extraction, road extraction, and flood detection tasks), and our SegEarth-OV consistently generates high-quality segmentation masks.

Abstract

Remote sensing image plays an irreplaceable role in fields such as agriculture, water resources, military, and disaster relief. Pixel-level interpretation is a critical aspect of remote sensing image applications; however, a prevalent limitation remains the need for extensive manual annotation. For this, we try to introduce open-vocabulary semantic segmentation (OVSS) into the remote sensing context. However, due to the sensitivity of remote sensing images to low-resolution features, distorted target shapes and ill-fitting boundaries are exhibited in the prediction mask. To tackle this issue, we propose a simple and general upsampler, SimFeatUp, to restore lost spatial information in deep features in a training-free style. Further, based on the observation of the abnormal response of local patch tokens to [CLS] token in CLIP, we propose to execute a straightforward subtraction operation to alleviate the global bias in patch tokens. Extensive experiments are conducted on 17 remote sensing datasets spanning semantic segmentation, building extraction, road detection, and flood detection tasks. Our method achieves an average of 5.8%, 8.2%, 4%, and 15.3% improvement over state-of-the-art methods on 4 tasks. All codes are released.

Dependencies and Installation

# 1. install SimFeatUp
# refer to https://github.com/likyoo/SimFeatUp

# 2. git clone this repository
git clone https://github.com/likyoo/SegEarth-OV.git
cd SegEarth-OV

# 3. create new anaconda env
conda create -n SegEarth python=3.9
conda activate SegEarth

# install torch and dependencies
pip install -r requirements.txt
# The dependent versions are not strict, and in general you only need to pay attention to mmcv and mmsegmentation.

Datasets

We include the following dataset configurations in this repo:

1. Semantic Segmentation: OpenEarthMap, LoveDA, iSAID, Potsdam, Vaihingen, UAVid^img, UDD5, VDD
2. Building Extraction: WHU^Aerial, WHU^Sat.Ⅱ, Inria, xBD^pre
3. Road Extraction: CHN6-CUG, DeepGlobe, Massachusetts, SpaceNet
4. Water Extraction: WBS-SI

Please refer to dataset_prepare.md for dataset preparation.

Quick Inference

python demo.py

Model evaluation

Single-GPU:

python eval.py --config ./configs/cfg_DATASET.py --workdir YOUR_WORK_DIR

Multi-GPU:

bash ./dist_test.sh ./config/cfg_DATASET.py

Evaluation on all datasets:

python eval_all.py

Results will be saved in results.xlsx.

Results

Citation

@article{li2024segearth,
  title={SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images},
  author={Li, Kaiyu and Liu, ruixun and Cao, Xiangyong and Bai, Xueru and Zhou, Feng and Meng, Deyu and Wang, Zhi},
  journal={arXiv preprint arXiv:2410.01768},
  year={2024}
}

Acknowledgement

This implementation is based on ClearCLIP and FeatUp. Thanks for the awesome work.