u-Segment3D ("Turn 2D segmentations into 3D")

A Python library for generating 3D consensus cell segmentation from 2D segmented stacks

May 22, 2024

u-Segment3D is a Python library to merge 2D slice-by-slice segmented 3D volumes in x-y, x-z, or y-z views into a single consensus 3D segmentation. u-Segment3D can use the probabilities and gradients directly predicted by popular existing Neural Network segmentation models such as Cellpose (Direct Method) or operates on the provided 2D segmentations directly (Indirect Method). u-Segment3D does not require all 3 views, any number can be used, nor do each view need be generated by the same algorithm. u-Segment3D also provides a number of postprocessing algorithms to recover missing features after obtaining the 3D segmentation. Therefore it can also be used as a postprocessing module for 3D segmentations in general. Despite the name, the postprocessing and segmentation functions in u-Segment3D have 2D equivalents which can be used to enhance 2D segmentation too. Please see the tutorials.

The primary motivation in developing u-Segment3D is to take advantage and leverage existing state-of-the-art 2D segmentation models for 3D segmentation without further data training.

It is associated with the BioArxiv paper, A general algorithm for consensus 3D cell segmentation from 2D segmented stacks, bioRxiv, 2024, written by Felix Y. Zhou, Clarence Yapp, Zhiguo Shang, Md Torikul Islam, Benjamin Nanes, Edward Jenkins, Gabriel M. Gihana, Bo-Jui Chang, Andrew Weems, Michael Dustin, Sean Morrison, Reto Fiolka, Kevin Dean, Andrew Jamieson, Peter K. Sorger and Gaudenz Danuser.

The current library is still a work-in-progress. It is fully functional but documentation still require work and more tutorials are upcoming.

A universal 2D-to-3D segmentation algorithm

Dependencies

u-Segment3D has a number of dependencies detailed in the requirements.txt. GPU dependencies are based on installing Cellpose to use as the default 2D segmentation method in u-Segment3D.

Installation

u-Segment3D can be installed by git cloning the repository and running pip in the cloned folder with python>=3.9. We have tested on Python==3.9 on Red Hat Enterprise Linux Server 7.9. We suggest first creating a new conda environment for install and use conda to install cudatoolkit and cudnn first:

conda create -n u_Segment3D_env python=3.9 cudatoolkit=11.8.* cudnn==8.* -c anaconda
conda activate u_Segment3D_env
pip install .

Then,

pip install git+https://www.github.com/mouseland/cellpose.git

to update to the latest version of Cellpose (If necessary). We find recent version of Cellpose doesn't need this last step.

The most difficult part of the install is the GPU dependencies, if using Cellpose. The 2D-to-3D segmentation of u-Segment3D itself can be run using only CPU. If there are any problems in installing, we provide an alternative install path which we have had success internally with colleagues, based on using the included u_Segment3D_env.yml environment file:

conda env create -f u_Segment3D_env.yml
conda activate u_Segment3D_env
pip install git+https://www.github.com/mouseland/cellpose.git
pip install torch==2.0.1 torchvision==0.15.2 cupy-cuda11x
pip install dask[complete]
pip install tiler
pip install edt
pip install connected-components-3d

If on a HPC cluster, you may need to module load the cuda, module load cuda118/toolkit/11.8.0 prior to activating the conda environment

Errors we have encountered:

1. Installing pyicu through pip may give an error when building. This is to do with the gcc version. You may need to use a later gcc compiler e.g. module load gcc/8.3.0. If this does not work, try removing the dependency from requirements.txt altogether and reinstall.

2. Installing cucim-cu11 via pip seems to fail occasionally with the error it finds only a placeholder. This error can be fixed by directly downloading the wheel matching your Python and OS version from https://pypi.nvidia.com/cucim-cu11/, installing this first before running pip install . As example, using Python 3.9 and wget on Linux:

wget https://pypi.nvidia.com/cucim-cu11/cucim_cu11-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip install cucim_cu11-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip install .

Getting Started

The simplest way to get started is to check out the included notebook tutorials which aims to showcase various use cases of u-Segment3D for 3D segmentation and how parameters may be tuned and algorithms adapted.

Example Data

Please download the zipped folder containing example data from the link. The following examples assume you have unzipped the data to the example_data/ directory of this repository, and is running the examples after installation from their current location in the repository. Please adjust filepaths accordingly, if otherwise.

Example Scripts

Examples for 3D Single Cell segmentation with morphological motifs:

1. tutorials/single_cells/segment_blebs_3D.py : "demonstrates direct method, how to run on downsampled version to get base segmentation, and additionally recover cellular protrusions at original image resolution for efficiency"
2. tutorials/single_cells/segment_lamellipodia_3D.py : "demonstrate direct method and basic usage"
3. tutorials/single_cells/segment_filopodia_3D.py : "demonstrates direct method and basic usage, and using a median filter to pre-smooth salt-and-pepper like noise which may affect guided filter postprocessing"
4. tutorials/single_cells/segment_ruffles_3D.py : "demonstrates direct method and basic usage, and extra use of padding and median filter, if single cells are cropped too close to the border"

Examples for 3D Multiple Cell segmentation:

1. tutorials/multi_cells/segment_Tcells_3D.py : "demonstrates the same direct method with Cellpose backend for single cells can be applied to resolve multiple cells and their subcellular protrusions"
2. tutorials/multi_cells/segment_zebrafish_macrophages_3D.py : "demonstrates more extensive preprocessing prior to running Cellpose 2D and then running u-Segment3D aggregation"
3. tutorials/multi_cells/segment_zebrafish_macrophages_3D_with_independent_fg_binary.py : "demonstrates how to 'paint' an external binary with cellpose gradients within the gradient descent aggregation in u-Segment3D. Regions not covered by gradients are segmented as a single cell or fragment"

Examples for 3D Tissue segmentation:

1. tutorials/tissue/segment_organoid_3D.py : "demonstrate direct method and basic usage, with minimal processing, how to resize images according to voxel resolution and eval performance"
2. tutorials/tissue/segment_ovules_3D.py : "demonstrate direct method and basic usage, with minimal processing, how to resize images according to voxel resolution and eval performance"

Examples for 2D segmentation for anisotropic: These examples demonstrate the indirect method, whereby we first generate the 2D segmentation masks and also showcases how to aggregate on only one view i.e. 'xy' to segment 3D.

1. tutorials/anisotropic/segment_single-cell_tracking_challenge_2D_stacks.py : "demo segmentation of MDA231 cells from the single cell tracking challenge, using an independent Otsu thresholded foreground binary to mask and get cell shape boundary"
2. tutorials/anisotropic/segment_epithelial_organoid_2D_stacks.py : "demo segmentation of an air-liquid interface grown skin cell culture that is thin"

Overview of Library Features

For those interested, u-Segment3D is a library organized by function similar to scipy / numpy / opencv allowing re-usable functions that can be used to build-up complex processing pipelines. A brief summary is below.

Module	Functionality
file_io.py	Functions for reading and writing images and intermediate outputs including pickled objects
filters.py	Functions for 'filtering', 'enhancing' and 'cleaning-up' images and segmentations e.g. label diffusion, keep largest component
flows.py	Functions for computing different 2D flows used for 2D-to-3D segmentation
gpu.py	Functions for GPU-based processing, primarily array resizing to speed up I/O operations if GPU is available and CUDA is installed.
meshtools.py	Functions for basic mesh processing, e.g. reading/writing meshes, extracting surface mesh from binary segmentation and computing mesh properties
metrics.py	Functions for evaluating segmentation performance, such as average precision
parameters.py	Functions for generating the default parameters for key u-Segment3D steps
plotting.py	Functions for plotting, such as coloring the gradient and segmentations
segmentation.py	Functions for running Cellpose with automatic parameter tuning
watershed.py	Functions for doing suppressed gradient descent in 2D/3D
usegment3d.py	Wrapper Functions that align with the conceptual steps in the paper - most often will use functions from here

Questions and Issues

Feel free to open a GitHub issue or email me at [email protected].

Danuser Lab Links

Danuser Lab Website

Software Links