Virtualizarr + Coiled Serverless Example Notebook

[norlandrhagen]

Inspired by @thodson-usgs's Lithops example, I've created a Virtualizarr example using coiled serverless functions.

• 1TB virtual dataset from 924 NetCDF files
• 9 minutes and ~$0.24 of cloud cost on coiled

Would love some feedback if anyone has thoughts.

• Changes are documented in docs/releases.rst

[TomNicholas]

Awesome!!

I'm a bit unclear where the in-memory datasets live at each point in the computation when using coiled functions. You do the reference generation on a bunch of separate instances, but in order to to combine_by_coords they all have to be on the same instance. At what point does that transfer occur?

[norlandrhagen]

I'm a bit unclear where the in-memory datasets live at each point in the computation when using coiled functions. You do the reference generation on a bunch of separate instances, but in order to to combine_by_coords they all have to be on the same instance. At what point does that transfer occur?

I'm pretty sure the .map returns a generator of all the virtual datasets to my local laptop.
Coiled allows you to run notebooks in the cloud. I successfully tried running all the reference generation serverless functions from a coiled cloud notebook and that also worked great. It might be a good option for a larger dataset, but I think since the manifest arrays are so memory efficient, I didn't see any issues doing the reduce on a laptop.
I also tried starting the reference generation severless functions from a larger serverless function (that was meant to be the reduce machine) and it was kinda wacky.

It would be nice to find the limits to where this becomes a problem! At that point, maybe the lithops map-reduce executor or beam would be a better option.