Multiprocessing option

[leguillf]

Since a filtering process, specific to one time of the window, seems to be independent from others, it would be computationally effective to implement a multiprocessing architecture. Have you already developed such an architecture ? Otherwise, I would be happy to help you on that issue.

[angus-g]

Sorry for not getting back to you on this! You're absolutely correct that this is pretty amenable to multiprocessing. At the moment we're using OpenMP within the custom OceanParcels fork to parallelise over particles for advection. Because we're doing filtering in 2D, we manually split into tasks over different Z levels (and/or timesteps) within batch jobs on our HPC. This works fine for the moment, but it's a bit labour-intensive. Additionally, it doesn't give us a single, collated output file so some post-processing is required.

I think it would be pretty cool if we could get some task-level parallelism for this. It could be farmed out to dask or something fairly easily.