Single-Shot Auto Focus (SSAF) for the Remoscope

This is work done by the Bioengineering team at the CZ Biohub SF, a non-profit research institution.

This repository houses a simple CNN used for running single-shot autofocus on the Remoscope (preprint here: https://www.medrxiv.org/content/10.1101/2024.11.12.24317184v1). On Remoscope, we flow whole blood through a flow cell whose thickness is such that the red blood cells flow in a monolayer. We want to be able to image continuously without needing to pause and acquire Z-stacks to re-focus (in the case of focal drift caused by mechanical vibrations or thermal effects).

To keep the sample in focus throughout the run, we have trained a CNN to recognize focus offsets based on the images (there is an asymmetry in the focus stack due to non-idealities in the optics which causes the images to look slightly different when above or below focus). The model outputs the number of steps away from focus and the direction (i.e +3 steps, -5 steps, etc.).

Installation

These instructions are for installation onto the BRUNO HPC at the Chan Zuckerberg Biohub San Francisco.

Create (or activate) a conda environment if you have not already.

Install PyTorch and Torchvision from here, selecting the installation settings for

• Stable
• Linux
• Conda
• Python
• Cuda (most recent, or the Cuda version that the HPC uses)

conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge

For exporting models, we need onnx:

conda install -c conda-forge onnx

And for converting to the intermediate representation,

pip3 install openvino-dev

If you find that you need a package that is not included here, please create an issue!

Training

The file ssaf_training.sh is the batch job file. Run

sbatch ssaf_training.sh

Exporting Models

We need to convert your .pth file to .onnx and then to the Intel intermediate representation (a .xml and a .bin file). To export your nmodel to .onnx, run

./to_onnx.py <PATH_TO_YOUR_pth>

which should create a .onnx file, a .xml file, and a .bin file. The .bin and .xml files are the intermediate representation of your model. You can then rsync or scp the .bin and .xml files from wherever you need them!

Tips for preparing OpenVino

this link has helpful information

also see here

After installing opencv and cmake, you can run

git clone https://github.com/openvinotoolkit/openvino.git
git checkout tags/2022.1.0
git submodule update --init --recursive
mkdir build && cd build

# only the MYRIAD plugin is supported, and is default "on"
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DTHREADING=SEQ \
  -DOpenCV_DIR="/home/pi/opencv/platforms/linux/arm.toolchain.cmake" \
  -DENABLE_PYTHON=ON \
  -DPYTHON_EXECUTABLE="/usr/bin/python3" \
  -DPYTHON_LIBRARY="/usr/lib/arm-linux-gnueabihf/libpython3.7m.so" \
  -DPYTHON_INCLUDE_DIR="/usr/include/python3.7" \
  -DENABLE_OV_TF_FRONTEND_ENABLE=OFF \
  -DENABLE_OV_PDPD_FRONTEND_ENABLE=OFF \
  -DENABLE_SAMPLES=OFF .. \
  && make --jobs=$(nproc --all)

And you have to add openvino to the PYTHONPATH variable. Add this to ~/.bashrc:

export PYTHONPATH=/home/pi/openvino/bin/armv7l/Release/lib/python_api/python3.7:$PYTHONPATH

OpenVino Performance Optimizations!

OpenVino benchmark app

Measuring Performance

Getting Performance Numbers

Improving Performance

Training & executing with half-precision?

• Intel speaks a bit to this WRT the NCS
• and this also from pytorch
• specific fp16 optims
• Great explanation of fp16

Preprocess Faster?

General Model Optimization Guide

Runtime Inference Options

Model Caching or Compiling