Add auto_compress=True option to HDF5IO.write()

[yarikoptic]

Per our discussion at SfN with @oruebel it might be greatly beneficial to enable compression for at least some datasets within HDF5 files. Upon initial palpation of .nwb files in the wild it is possible to find some instances where even gzip level 1 compression would bring 10x storage savings. (e.g. for http://datasets.datalad.org/?dir=/labs/svoboda/Chen_2017 I saw:

smaug:/mnt/btrfs/datasets/datalad/crawl-misc/svoboda-rclone/Exported NWB 2.0
$> du -scm Chen*                        
35113	Chen 2017
3579	Chen 2017_comp-gzip1
3324	Chen 2017_comp-gzip5
3319	Chen 2017_comp-gzip9
35114	Chen_2017-ds
3298	Chen 2017.tgz
83744	total

Ref: our very initial "discussion" at @dandiarchive : dandi/dandiarchive-legacy#15
Attn: @bendichter @mgrauer

[yarikoptic]

On the related note - are you aware of any way to "index" compressed (in HDF5) datasets? Indexing could allow for efficient access to portions of the dataset without chunking and requiring to decompress the preceding data just to get to the offset.

Related

• for nibabel, neuroimaging .nii.gz files, https://pypi.org/project/indexed-gzip/ was developed to provide similar capabilities for gzip'ed files.
• XZ is an efficient archiving format with indexing capability. Unfortunately I do not see it even listed among registered external compression filters

But may be, instead of indexing, it would indeed be better to take advantage of the domain knowledge here and chunk data wisely (e.g. separate channels, and then some reasonable time durations within each chunk, e.g. for each 10k samples) instead. I expect the extra chunking "metadata" not being a significant storage demand. Not yet sure how much of performance hit it would be to reconstruct upon loading entire dataset(s), but this report suggests that increasing chunk cache could be of tremendous benefit. In summary chunking should allow for quite efficient data access regardless of the usage scheme (slicing in time or in channels space).

[oruebel]

Chunking and indexing address two orthogonal problems. Chunking mostly addresses the data load whereas indexing is concerned with data search. Chunking is something we already support.

For indexing in HDF5, FastBit may be an attractive option.

[mavaylon1]

Related to #1158 ; however, we will close for now until it comes back up again. We already have neuroconv to do automatic compression. Feel free to reopen.

[bendichter]

@yarikoptic we now how the method that does this here: https://neuroconv.readthedocs.io/en/main/api/tools.nwb_helpers.html#neuroconv.tools.nwb_helpers.configure_and_write_nwbfile