Implementation Details

[bnlawrence]

This issue provides a summary of the outstanding issues required for the POSIX implementation of the PyActiveStorageAPI to be fully compliant with the native (no active storage) and S3 implementations.

Most, but not all of these issues are fully described here, but there are a couple on the PyActiveStorage repo as well, which are linked from here.

API Issues

The full reduced chunk API is described in depicted here (with native source code here and S3 source code here). In the case of S3, the reduce_chunk implementation packages up the API into an http request handled by the rust server code in this repository. These are all called from the conditional which starts here (i.e from within process_chunk).

To fully support that API we expect the following steps make the most sense:

1. Implement the dtype in the C API (i.e. here in CSFS_Python/active_proxy.c), and internally select the appropriate reduction library for the right method and type.
2. Implement the order character
3. Implement the shape and chunk_selection parameters
4. Implement the compressor and filters parameters (integers which allow the server to decompress and/or carry out other filter operations according to a lookup table), and
5. Implement support for the missing constructor.

Once these are in the API (#3), in the first instance we should raise an error for any actual use of shape, chunk_selection,missing,compressor and filters, although it should be relatively easy to support all but the compression and filters which will require additional server side support (but the others can be built following example code from the S3 rust implementation).

Then we should proceed through #4, #5 and #6.

Sequencing (issues for PyActiveStorage)

To aid in this process, it'd be helpful for us to build a python wrapper for the expected C, and make sure we can get the API into the C and hello world the arguments (PyActiveStorage issue 143)

We also need finalise the method for discovering whether or not the posix storage is active (PyActiveStorage Issue 56).

We then need to build and deploy a full stack implementation against the fuse implementation, and do it in such a way that DDN can do the same against their Red implementation. (#7)

We then need to define a shared lookup table for filters and compressors (we probably have one, but I'm not yet sure where). (PyActiveStorage issue 145).