Solve Alaska HAND FIM Processing Memory bottleneck

[shawncrawley]

Weeks ago while first testing Alaska FIM, we came face to face with the fact that the underlying inputs for creating HAND FIM polygons for Alaska are too large. They surpass the maximum memory that can be allocated to the hv-vpp-*-viz-hand-fim-processing Lambda.

Here are the priorities we identified as steps for solution to getting Alaska FIM working as services:

• Taking reprojection out of lambda running processing
• Running the GMS only
• Run for just Anchorage AK to get a rapid solution

Here are the longer term solutions to get a stable pipeline for the service after we get something available for Workshop:

• Have Ryan look into HAND Processing code for efficiencies
• Setup running on smaller HUC 10 domains for Alaska
• Filter out smaller stream orders (whether in GMS or branch 0 depending on other tests)

Getting a service up to UAT is the primary goal, the next will be getting the service stable and tested fully for all the services in UAT, then lastly before pushing is making any budgetary efficiency measurements on the service.

[shawncrawley]

@groutr had made some HAND FIM improvements that we thought would solve this (see #944). There must have been a mistake made when putting it in place and testing it, because I swear it was working. But I just discovered yesterday that it's still failing due to running out of memory. Thus, I'm tackling the "Running the GMS only" approach as an interim solution until we can work out a long-term solution. Commit and PR to come here in a moment...

[RobHanna-NOAA]

We have many more options for future releases actually and more options are opening up all of the time.

• HV can help solve part of it, but on the FIM-Dev sides, we already have put in code that cuts the size of the AK Hucs way down. This feature is not part of the dataset we currently have in 2.1.8 but AK HUCs will come from FIM-dev considerably smaller.
• We would like to work with HV to see how moving some datafiles to be send over to HV in parquet format. In Dec, I would like to work with Ryan to experiment with this with me. If it works and we can prove some performance gains, we can quickly fix the data coming from FIM-dev to come in parquet or other formats.
• We can re-examine how/when we load hydrotable data. ie). FIM-dev has hydrotables at each branch level, but also has aggregated hydrotables at the huc level which aggregates all of the branches into one. And it too could be parquet or pre-loaded into RDS if not already there.
• Ultimately any logic that HV needs to apply must be done on short term emergency basis only. Logic and/or filtering, file types, etc has to come from FIM-dev as it can affect a wide range other areas including our QA systems if the data FIM-dev gives is not the same as being used in the UI.

I am sure we can work together to fix anything we need as long as either Carson or Derek approves it and Carson, Derek and myself are aware of it. Easiest if I can help with those options as I have more options available to help solve these problems then might be seen. Over time, I can teach more of the details of the FIM-dev datasets so that you won't need me to help with these types of details eventually.

[shawncrawley]

@RobHanna-NOAA Awesome - thanks for updating this ticket with all of this info. We should meet again soon to discuss which direction we should take for the official and long-term solution. This ticket should remain open until that long-term fix is in place. The short-term fix was only put in place to allow us to get Alaska FIM released in v2.1.8.