Skip to content

SA-DNet is an end-to-end Semantic-Aware on-Demand registration-fusion network. SA-DNet can specify registration and fusion regions based on semantic information, achieving on-demand conceptualization.

License

Notifications You must be signed in to change notification settings

Meng-Sang/SA-DNet

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SA-DNet

SA-DNet is an end-to-end Semantic-Aware on-Demand registration-fusion network. SA-DNet can specify registration and fusion regions based on semantic information, achieving on-demand conceptualization. Furthermore, the sam, reg, and fuse modules are separated, allowing integration of new functionalities. The current architecture implemented in this project is as follows:

Network
Note:

In this project, the HOL operators in reg have not been implemented. When SAM perceives regions that are too large, it leads to excessive computation in HOL, causing the system to freeze.

TODO

  • Fixed bug.
  • Update the new sam, reg, and fuse modules.
  • To optimize and implement the HOL operator

Install

Download this project:

git clone [email protected]:Meng-Sang/SA-DNet.git
cd SA-DNet

Runtime environment

This project requires a runtime environment with Python=3.8 or higher. Additionally, it depends on the libraries specified in the requirements.txt file and 2 GitHub repositories (Grounding DINO and Segment Anything). The environment configuration is as follows:

Step 1: install python and requirements.txt
conda create -n sa-dnet  python=3.8
pip install -r requirement.txt
Step 2: Configure Grounding DINO

For Grounding DINO installation, please refer to Grounding DINO.

Download weight
# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/grounding_dino/weight
# then:
cd model/sam/grounding_dino/weight
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth
Note:

If the network is unavailable, the following error may occur:

OSError: Can't load tokenizer
Step 3: Configure Segment Anything
git clone [email protected]:facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

or

pip install git+git://github.com/facebookresearch/segment-anything.git
Download weight
# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/segment_anything/weight
# then:
cd model/sam/segment_anything/weight
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Demo

Run with Python

import cv2
from utils.predict import sa_dnet

ir_image = cv2.imread("assets/ir/006393.jpg")
vi_image = cv2.imread("assets/vi/006393.jpg")
_, fuse_image = sa_dnet(ir_image, vi_image, "car", is_mask=True)
cv2.imwrite("assets/demo.jpg", fuse_image)

Show result

ir image
IR Image
vi image
VI Image
matching image
Matching Image (Car)
unmask fuse image
Unmask Fuse Image (Car)
Image 4
Mask Fuse Image (Car)

Acknowledgments:

This project depends on the following repositories:

About

SA-DNet is an end-to-end Semantic-Aware on-Demand registration-fusion network. SA-DNet can specify registration and fusion regions based on semantic information, achieving on-demand conceptualization.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages