Skip to content

Latest commit

 

History

History
114 lines (80 loc) · 14.5 KB

README.md

File metadata and controls

114 lines (80 loc) · 14.5 KB
Raw

BasicVSR++ (CVPR'2022)

BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment

Task: Video Super-Resolution

Abstract

A recurrent structure is a popular framework choice for the task of video super-resolution. The state-of-the-art method BasicVSR adopts bidirectional propagation with feature alignment to effectively exploit information from the entire input video. In this study, we redesign BasicVSR by proposing second-order grid propagation and flow-guided deformable alignment. We show that by empowering the recurrent framework with the enhanced propagation and alignment, one can exploit spatiotemporal information across misaligned video frames more effectively. The new components lead to an improved performance under a similar computational constraint. In particular, our model BasicVSR++ surpasses BasicVSR by 0.82 dB in PSNR with similar number of parameters. In addition to video super-resolution, BasicVSR++ generalizes well to other video restoration tasks such as compressed video enhancement. In NTIRE 2021, BasicVSR++ obtains three champions and one runner-up in the Video Super-Resolution and Compressed Video Enhancement Challenges. Codes and models will be released to MMagic.

Results and models

The pretrained weights of SPyNet can be found here.

Model Dataset PSNR (RGB) SSIM (RGB) PSNR (Y) SSIM (Y) Training Resources Download
basicvsr_plusplus_c64n7_8x1_600k_reds4 REDS4 (BIx4) 32.3855 0.9069 - - 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_8x1_600k_reds4 UDM10 (BDx4) - - 34.6868 0.9417 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_8x1_600k_reds4 Vid4 (BIx4) - - 27.7674 0.8444 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_8x1_600k_reds4 Vid4 (BDx4) - - 24.6209 0.7540 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_8x1_600k_reds4 Vimeo-90K-T (BIx4) - - 36.4445 0.9411 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_8x1_600k_reds4 Vimeo-90K-T (BDx4) - - 34.0372 0.9244 8 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi REDS4 (BIx4) 31.0126 0.8804 - - 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi UDM10 (BDx4) - - 33.1211 0.9270 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi Vid4 (BIx4) - - 27.7882 0.8401 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi Vid4 (BDx4) - - 23.6086 0.7033 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi Vimeo-90K-T (BIx4) - - 37.7864 0.9500 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi Vimeo-90K-T (BDx4) - - 33.8972 0.9195 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd REDS4 (BIx4) 29.2041 0.8528 - - 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd UDM10 (BDx4) - - 40.7216 0.9722 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd Vid4 (BIx4) - - 26.4377 0.8074 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd Vid4 (BDx4) - - 29.0400 0.8753 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd Vimeo-90K-T (BIx4) - - 34.7248 0.9351 4 (Tesla V100-PCIE-32GB) model | log
basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd Vimeo-90K-T (BDx4) - - 38.2054 0.9550 4 (Tesla V100-PCIE-32GB) model | log

NTIRE 2021 checkpoints

Note that the following models are finetuned from smaller models. The training schemes of these models will be released when MMagic reaches 5k stars. We provide the pre-trained models here.

Model Dataset Download
basicvsr-pp_c128n25_600k_ntire-vsr NTIRE 2021 Video Super-Resolution - Track 1 model
basicvsr-pp_c128n25_600k_ntire-decompress-track1 NTIRE 2021 Quality Enhancement of Compressed Video - Track 1 model
basicvsr-pp_c128n25_600k_ntire-decompress-track2 NTIRE 2021 Quality Enhancement of Compressed Video - Track 2 model
basicvsr-pp_c128n25_600k_ntire-decompress-track3 NTIRE 2021 Quality Enhancement of Compressed Video - Track 3 model

Quick Start

Train

Train Instructions

You can use the following commands to train a model with cpu or single/multiple GPUs.

# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py

# single-gpu train
python tools/train.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py

# multi-gpu train
./tools/dist_train.sh configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py 8

For more details, you can refer to Train a model part in train_test.md.

Test

Test Instructions

You can use the following commands to test a model with cpu or single/multiple GPUs.

# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth

# single-gpu test
python tools/test.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth

# multi-gpu test
./tools/dist_test.sh configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth 8

For more details, you can refer to Test a pre-trained model part in train_test.md.

Citation

@InProceedings{chan2022basicvsrplusplus,
  author = {Chan, Kelvin C.K. and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  title = {BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment},
  booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition},
  year = {2022}
}