SAM-guided Graph Cut for 3D Instance Segmentation

Project Page | Video | Paper

SAM-guided Graph Cut for 3D Instance Segmentation
Haoyu Guo^*, He Zhu^*, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, Xiaowei Zhou

TODO

• Segmentation with / without GNN
• Graph construction and SAM based annotation
• Processing of point clouds
• Processing of triangle meshes

Setup

The code is tested with Python 3.8 and PyTorch 1.12.0.

1. Clone the repository:

git clone https://github.com/zju3dv/SAM_Graph.git

2. Install dependencies:

   pip install -r requirements.txt

Clone ScanNet repository and build the segmentor and modify segmentor_path in scripts/run.py.

Download the checkpoint of segment-anything model and modify sam_ckpt_path in scripts/run.py.

Data preparation

Please download the example data from here and modify data_path in scripts/run.py for fast testing. If you want to use your own data, please organize the data as the same format as the example data.

Run

The pipeline of our method is illustrated as follows:

graph TD
    subgraph Input
        A[multi-view images]
        B[sensor depth]
        C[camera pose]
        D[point cloud / mesh]
    end

    E[rendered depth]
    F[superpoints]
    G[sam encoder feature]
    H[depth difference]
    I[superpoint projections]
    J[graph structure]
    K[predicted masks]
    L[edge weights]
    M[node feature]
    N[graph segmentation]

    A --> G
    B --> H
    C --> E
    C --> I
    D --> E
    D --> F
    E --> H
    F --> I
    F --> J
    G --> K
    G --> M
    H --> I
    I --> K
    I --> M
    J --> L
    J --> N
    K --> L
    L --> N
    M --> N

Loading

Note that depth difference step is optional, but is recommended if accurate sensor depth is available and the point cloud / mesh contains large holes or missing regions.

To run the pipeline, simply run:

cd scripts
python run.py

The results of each step will be saved in the individual folders.

Citation

@inproceedings{guo2024sam,
  title={SAM-guided Graph Cut for 3D Instance Segmentation},
  author={Guo, Haoyu and Zhu, He and Peng, Sida and Wang, Yuang and Shen, Yujun and Hu, Ruizhen and Zhou, Xiaowei},
  booktitle={ECCV},
  year={2024}
}