Camera Exposure Control for Robust Robot Vision with Noise-Aware Image Quality Assessment

In this paper, we propose a noise-aware exposure control algorithm for robust robot vision. Our method aims to capture best-exposed images, which can boost the performance of various computer vision and robotics tasks. Our metric consists of a combination of image gradient, entropy, and noise metrics. The synergy of these measures allows the preservation of sharp edges and rich texture in the image while maintaining a low noise level. Using this novel metric, we propose a real-time and fully automatic exposure and gain control technique based on the Nelder-Mead method.

Overview

[Full paper] [YouTube] [Project Page]

Dataset

In this paper, we provide a unique dataset developed specifically to compare exposure control algorithms. The composition of this dataset is as follows.

• HW setup: a stereo camera system with 20 cm baseline acquiring synchronized 1600 x 1200 px images.
• # of scene: Total 25 scene (10 indoor, 15 outdoor)
• # of image: Each scene consist of 550 x 2 images
  • Outdoor Environment
    • Exposure time : [0.1 - 7.45 ms] with 0.15 ms interval
    • Gain : [0 - 20]dB with 2dB interval
  • Indoor Environment
    • Exposure time : [4 - 67 ms] with 3 ms interval
    • Gain : [0 - 24]dB with 1dB interval
• # of object class: 13 Object Class
  • Person, Bicycle, Car, Firehydrant, Backpack, Sports ball, Chair, Mouse, Keyboard, Cellphone, Book, Scissors, and TV.
  • However, some objects appear very rarely, we need to acquire more dataset.
• Download link: https://drive.google.com/file/d/1DUeByL_ADzGyRpqc0iXA-DxZwLOd5Q-U/view?usp=sharing

Quick Start: Demo Script

1. Download the datset

clone this repositroy
donwload datset from [here](https://drive.google.com/file/d/1DUeByL_ADzGyRpqc0iXA-DxZwLOd5Q-U/view?usp=sharing)
unzip "DataSet_AE.zip"

2. Run MetricEvaluation.m

run "Run_MetricEvaluation.m"

Then, this matlab code create and save a landscape of the proposed image quaility assessment metric for each dataset.

Note that, we have cleaned and optimized the code for better readability. However, the result could be slightly different from the result reported in the paper.

The original results, including related works, can be available here.

If you want use original results, unzip this file into "Result_Exp" folder.

3. Run NMbasedControl.m

run "NMbasedControl.m"

This matlab code runs the paper version of the Nelder-Mead optimization based control algorithm for each dataset.

4. Run FeatureMatching.m

run "FeatureMatching.m"

This matlab code conducts a experiment about a feature extraction & matching. To do this experiment, you need to install OpenCV on matlab by following this link.

C++ version code

The C++ code "NoiseAwareAE_Code_cpp.zip" includes the c++ version of the proposed image quality assessment metric and the Nelder-Mead optimization based control algorithm. The implemented code is designed to operate on a Flir blackfly color camera in a moving scenario. Therefore, the NM optimization part's implementation differs from the Matlab version. Also, this code is not well-organized and has dependencies on OpenCV, spinnaker, and boost library. The user needs to adjust the dependency for target hardware or selectively use image quality assessment metric C++ code or other C++ code.

Citation

Please cite the following paper if you use our work or parts of this code in your own work.

@inproceedings{shin2019camera,
  title={Camera exposure control for robust robot vision with noise-aware image quality assessment},
  author={Shin, Ukcheol and Park, Jinsun and Shim, Gyumin and Rameau, Francois and Kweon, In So},
  booktitle={2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={1165--1172},
  year={2019},
  organization={IEEE}
}