SBFBurst: Simple Base Frame Guided Residual Network

This is the official implementation of VISAPP 2024 "Simple Base Frame Guided Residual Network for RAW Burst Image Super-Resolution".

Abstract: Burst super-resolution or multi-frame super-resolution (MFSR) has gained significant attention in recent years, particularly in the context of mobile photography. With modern handheld devices consistently increasing their processing power and the ability to capture multiple images even faster, the development of robust MFSR algorithms has become increasingly feasible. Furthermore, in contrast to extensively studied single-image super-resolution (SISR), burst super-resolution mitigates the ill-posed nature of reconstructing high-resolution images from low-resolution ones by merging information from multiple shifted frames. This research introduces a novel and effective deep learning approach, SBFBurst, designed to tackle this challenging problem. Our network takes multiple noisy RAW images as input and generates a denoised, super-resolved RGB image as output. We demonstrate that significant enhancements can be achieved in this problem by incorporating base frame-guided mechanisms through operations such as feature map concatenation and skip connections. Additionally, we highlight the significance of employing mosaicked convolution to enhance alignment, thus enhancing the overall network performance in super-resolution tasks. These relatively simple improvements underscore the competitiveness of our proposed method when compared to other state-of-the-art approaches.

Architecture and some results

A table displaying images generated by different methods on Sythetic Dataset is shown below:

Base Frame	DBSR	MFIR	BIPNet	SBFBurst	Ground Truth

Refer to our paper for more details.

Installation and Requirements

1. This project was tested in Python 3.11 and CUDA 11.8 environments. It is recommended to set up a virtual environment. All necessary requirements can be installed using pip by running the bash install.sh script.

2. The data can be downloaded from the creator repository: Synthetic and BurstSR. Shortcuts to the datasets are given below, but it is important to check if they are still working or updated according to the original authors' pages. Download the datasets and place them in the _DATASETS/ folder in their respective subfolders:

   • Synthetic: Zurich RAW to RGB Canon set and SyntheticBurst validation set
   • BurstSR: BurstSR dataset (cropped)

3. Update the paths in local.py according to your local environment. The environment setting file admin/local.py contains the paths for pretrained networks, datasets, etc.

Pretrained Model

All the checkpoints including the SpyNet and PWCNet weights are located in the pretrained_models/ folder. These files are in git-lfs format, so it's recommended to use git clone to properly get them.

Model	PSNR	SSIM	LPIPS
SBFBurst Synthetic	42.1918	0.9684	0.0367
SBFBurst Real World BurstSR	48.8719	0.9870	0.0224

Evaluation

The scripts run_test_synthetic.py and run_test_burstsr.py are configured to load models from the pretrained_models/ folder. They will save the predictions in the workspace/ folder. Ensure you adjust these settings according to meet your requirements.

Synthetic testing

python run_test_synthetic.py

Real World BurstSR testing

python run_test_burstsr.py

Training

The training configurations are located in the train_settings/sbfburst directory. You can run the training using the following commands:

Synthetic training

python run_training.py sbfburst default_synthetic

Real World BurstSR training

python run_training.py sbfburst default_burstsr

Citations

If our code contributes to your research or projects, please consider citing our paper using the following BibTeX reference:

@conference{visapp24,  
	author={Anderson Cotrim and Gerson Barbosa and Cid Santos and Helio Pedrini},  
	title={Simple Base Frame Guided Residual Network for RAW Burst Image Super-Resolution},  
	booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP},  
	year={2024},  
	pages={77-87},  
	publisher={SciTePress},  
	organization={INSTICC},  
	doi={10.5220/0012348300003660},  
	isbn={978-989-758-679-8},  
}

Acknowledgement

The training and testing codes are based on Deep-Burst-SR, BipNet and BasicVSR_PlusPlus.

Contact

My email is [email protected]. Please feel free to contact me regarding anything related to the paper. I'm also glad to discuss any topic related to Super-Resolution!