---

SuperBench is a benchmark dataset and evaluation framework for super-resolution (SR) tasks in scientific domains. It provides high-quality datasets and baseline models for evaluating and comparing SR methods in various scientific applications.

An overview of super-resolution for weather data

Highlights

• Diverse datasets: SuperBench includes high-resolution fluid flow (left), cosmology (right), and weather datasets (middle) with dimensions up to $2048\times2048$.

• Evaluation metrics: The framework provides comprehensive evaluation metrics for assessing SR performance, including:
  • Pixel-level difference
  • Human-level perception
  • Domain-motivated error metrics
• Baseline models: Pre-trained baseline models are provided to facilitate comparison with state-of-the-art methods.
• Extensible framework: SuperBench is designed to be easily extendable, allowing the inclusion of new datasets and baseline models.

Datasets

• Navier-Stokes Kraichnan Turbulence
  • Two fluid flow datasets are simulated with Reynolds numbers of $Re=16000$ and $Re=32000$. The spatial resolution of this dataset is $2048\times2048$.
  • Three variables are considered: two velocity variables in the $x$ and $y$ directions, as well as with the vorticity field.
• Cosmology Hydrodynamics
  • The spatial resolution is $2048\times2048$. The temperature and baryon density are provided in log scale.
  • A corresponding low-resolution simulation data is provided for a realistic super-resolution task.
• Weather data
  • The weather data is modified from ERA5. The spatial resolution is $720\times1440$.
  • Three channels are considered: Kinetic Energy (KE) at 10m from the surface, the temperature at 2m from surface, and total column water vapor.

Table 1: A summary of SuperBench dataset.

Baseline models

• Bicubic interpolation
• SRCNN,
  • Paper: Image Super-Resolution Using Deep Convolutional Networks
  • Venue: TPAMI, 2015 (the first paper using deep NNs)
• Subpixel CNN
  • Paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
  • Venue: CVPR, 2016
• EDSR
  • Paper: Enhanced deep residual networks for single image super-resolution
  • Venue: CVPR, 2017 (NTIRE2017 Super-Resolution Challenge winner)
• WDSR
  • Paper: Wide activation for efficient and accurate image super-resolution
  • Venue: BMVC, 2019 (NTIRE 2018 Super-Resolution Challenge winner)
• FNO
  • Paper: Fourier Neural Operator for Parametric Partial Differential Equations
  • Venue: ICLR, 2021
• SwinIR
  • Paper: Swinir: Image restoration using swin transformer
  • venue: CVPR, 2021

Evaluation metrics

To assess the performance of these methods, we employ three distinct types of metrics: pixel-level difference metrics; human-level perception metrics; and domain-motivated error metrics.

• Pixel-level difference:
  • relative Forbenius norm error (RFNE)
  • infinity norm (IN)
  • peak signal-to-noise ratio (PSNR)
• Human-level perception:
  • structural similarity index measure (SSIM)
• Domain-motivated error metrics:
  • physics errors (e.g., continuity loss)
  • Energy Specturm
  • Anomaly Correlation Coefficient (ACC)
  • ...

Results

We have evaluated several state-of-the-art SR models on the SuperBench dataset across different degradation scenarios. Here are an example result on weather dataset.

Baseline Performance

We present the baseline performance of various SR models on weather data with bicubi-downsampling degradation. Figure 1 shows visual comparisons of the baseline model reconstructions against the ground truth high-resolution images. (a) and (b) are x8 and x16 up-sampling tasks, respectively. Table 2 below provides quantitative evaluation results for weather data in terms of RFNE, IN, PSNR and SSIM metrics.

Figure 1: An example snapshot of baseline performance on weather data.

Table 2: Results for weather data with bicubic down-sampling.

Additional Results

For more detailed results and analysis, please refer to our paper.

Contribution to trained models

We welcome contributions from the scientific machine learning community. If you would like to contribute to the baseline models of SuperBench, please open an issue on the GitHub repository. You may either request to push code to src.models or provide a link to the trained models with model details.

Getting Started

Installation

To use SuperBench, follow these steps:

1. Clone the repository:

git clone https://github.com/erichson/SuperBench.git

2. Install the required dependencies:

pip install -r requirements.txt

Usage

1. Download the SuperBench datasets:

   Example:

   # for Cosmology data
   wget https://portal.nersc.gov/project/dasrepo/superbench/cosmo.tar
   
   # for Climate data
   wget https://portal.nersc.gov/project/dasrepo/superbench/climate.tar
   
   # for Fluid data
   wget https://portal.nersc.gov/project/dasrepo/superbench/nskt_16k.tar

2. Training Baseline Models

   To train all baseline models with the same configuration as described in the SuperBench paper, follow these steps:

   2.1. Generate the .sh code by running the following command;

   Note: Make sure to update the PATH variable in generate_train_sh.py to match the path where you have downloaded the data.

   python generate_train_sh.py

   2.2. Execute the generated .sh code to train all baseline models:

   sh train_all.sh

3. Evaluating Trained Models (Download trained weights from here.)

   To evaluate the performance of your trained model, you can use the eval.py script provided. This script requires several arguments to be specified:

   • --data_name: The name of the dataset you are using for evaluation.
   • --data_path: The path to the dataset directory.
   • --model_path: The path to the trained model file.
   • --in_channels: The number of input channels for the model.

4. Visualize the Super-Resolution (SR) Results

   To visualize snapshots as presented in the paper:

   python analysis/plot_snapshots.py

   To visualize the accuracy results as shown in the paper:

   python analysis/plot_ACC.py

   To visualize the energy spectrum in the paper:

   python analysis/plot_Energy_Spectrum.py

Contribution to datasets

We also welcome dataset contributions from the community. If you would like to contribute to SuperBench, please open an issue on the GitHub repository and provide a link to your datasets with data details.

Issues and Support

If you encounter any issues or have any questions, please open an issue on the GitHub repository.

License

SuperBench is released under the GNU General Public License v3.0.