Introduction

Code for my master thesis: A Sketch based 3D Point Cloud Modeling System based on Deep Generation Network and Detail Editing

The interactive system can generate the 3D point clouds from sketches and apply further point cloud editings.

Demo Video

Demo video on youtube: https://www.youtube.com/watch?v=Vch4nURRrAE&t=50s

Sketch-based 3D Point Cloud Modeling

System overview:

Main functions:

1. Generate 3D point clouds from sketches by reconstruction module.
2. Add erasing hints to improve point clouds' details by erasing module.
3. Add thin structures.
4. Apply deformation.

After editing, user can save the result point clouds.

Code

Set up environment

Create Python Virtual Env

virtualenv -p virtualenv -p /usr/bin/python3.6 sketch2pc_venv
source sketch2pc_venv/bin/activate

Install Pytorch

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Compile the EMD loss function, we use the code from the SpareNet

cd cuda/emd
python setup.py install

For other packages, install the packages in requirements.txt

pip install -r requirements.txt

This code is tested under CUDA 11.0 on Ubuntu 20.04 and a RTX2080 super GPU.

For model training and testing

Before training and testing, please check the config files in configs/ to make sure the paths are correct

Training

For reconstruction module:

python runner.py --train_gan

For erasing module:

python runner.py --train_refine

Testing

Compute CD, EMD:
For reconstruction module:

python runner.py --test_gan

For erasing module:

python runner.py --test_refine

Evaluating

Visualize the point clouds created from models:
For reconstruction module:

python runner.py --evaluate_gan

For erasing module:

python runner.py --evaluate_refine

Running the User Interface

Please run:

python sketch_3d_app.py

Downloads

Dataset

We render the 3D models in ShapeNet to different views and convert the rendering images to sketch images by Photo-Sketching, please download data from following links:

Point clouds

Sketch images

Sketch images and erasing annotations

Pretrained Weight

Please download pretrained weight from following links: (Reconstruction module: best-gan-ckpt.pth , Erasing module: best-refine-ckpt.pth)

Pretrained weight

Reference

https://github.com/hzxie/Pix2Vox.git
https://github.com/justanhduc/graphx-conv
https://github.com/kangxue/P2P-NET
https://github.com/microsoft/SpareNet
https://github.com/mathLab/PyGeM