Errors that did not stop the pipeline

[jds485]

I just ran the pipeline in the container for #66 and realized that some targets had errors that did not cause the pipeline to stop.

Example:
An error was thrown by one of the new targets in #66, which led me to find that p2_prms_nhdv2_xwalk was NULL. When I tried running the function called for that target, I got Error: C stack usage 7972148 is too close to the limit. I'll look into this error, but I am mostly concerned that the pipeline continued even when an error was thrown. I'm not sure if this is a problem with how I set up the container, with Rstudio server, or with how we made the pipeline.

[lekoenig]

Bummer, thanks for the example. I think these are still separate issues, but I'm curious - do you think p2_prms_nhdv2_xwalk was NULL due to the mutate error you experienced before?

[jds485]

No, this was a rebuild of the whole pipeline

[jds485]

To make things more interesting, I also had this happen outside of the container, and also did not stop the pipeline. I got this more detailed error message, though:
Error: evaluation nested too deeply: infinite recursion / options(expressions=)?
Error during wrapup: evaluation nested too deeply: infinite recursion / options(expressions=)?
Error: no more error handlers available (recursive errors?); invoking 'abort' restart

[lekoenig]

@jds485 That infinite recursion error is coming from a recursive function traverse_nhd in 2_process/src/pair_nhd_reaches.R. So something's up with the xwalk target either not getting the inputs it's expecting or behaving differently across our local builds.

I'm curious if the pipeline encounters an error, if it won't use an outdated version of the target in question. We didn't change the name of the xwalk target, so maybe upon failing, it's using that older version?

[jds485]

Okay, thanks for letting me know which function to check. The version of p1_nhdv2reaches_sf downloaded using the container had a different geometry than the one obtained from a local build. The attribute table was the same. Running outside of the container seems to be working to make the crosswalk target. So, maybe this is an R version/package version issue. Which version are you running? I'm using 4.0.5 locally and 4.1.2 in container

[lekoenig]

I'm running 4.1.1 locally. It's been awhile since I've rebuilt p1_nhdv2reaches_sf and I'm hesitant to tar_destroy (unless I can selectively destroy specific targets). If p1_nhdv2reaches_sf is the culprit, I wonder if this bug is coming from a difference in nhdplusTools?

[jds485]

Update: I updated my local Windows and container versions of R to 4.1.2 and also updated packages. The packages that are specified in library loads in our code have the same versions for Windows and the container. The repo built without error on my Windows computer. I receive an error running R (or R studio server) within the container. The error results from p2_prms_nhdv2_xwalk = NULL (as described above). So, I think this goes back to figuring out why the infinite recursion error is being thrown when the pipeline is run in the container

[lekoenig]

Thanks for the update, @jds485. OK, that is weird that the error comes up in the container but not on windows. I would have to dig in a bit further to assess whether/how the recursive function could be rewritten. So I might wait to see the outcome of your hacky fix! Do you mind also checking, is the geometry for p1_nhdv2reaches_sf being similarly returned in windows vs. the container? It seems possible to me that something's up with one of the inputs to the xwalk function.

[jds485]

All of the inputs to the xwalk function have the same summary information on Windows and the container.

However, some of the linestrings are different. I'm not sure the best way to continue diagnosing the problem. The function that generates p1_nhdv2reaches_sf only depends on a character vector that we specify (DRB huc8 codes), and functions in nhdplusTools and sf packages. I could remove the suppressMessages to see if that's useful, but I'm interested in your thoughts

[jds485]

well, it might just be a numerical precision issue. This is the summary of the absolute differences:
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.000e+00 0.000e+00 0.000e+00 3.720e-07 0.000e+00 4.974e-06

[jds485]

Documenting what I've tried:
I increased the C stack limit but the error still happened (with the higher limit).
I tried an "unlimited" limit, which resulted in a node stack overflow error.
I tried increasing the C stack limit and setting the maximum expressions in R to the max allowed with options(expressions=500000) but that also resulted in a node stack overflow error.

[jds485]

I just tried running on Linux outside of the container and I also had this problem. So it's narrowed down to an OS problem instead of how I set up the container. But at this point I have no further ideas for how to fix the issue. @lekoenig I think it would be best to place the crosswalk into a separate repo and we can pull the table as a target. That might be better for sharing the crosswalk among projects, anyway.

[lekoenig]

In a nutshell: I'm closer to diagnosing this problem - and p1_nhdv2reaches_sf is in fact the culprit - but I don't know why the differences in p1_nhdv2reaches_sf between local builds and the container build are occurring in the first place.

The infinite recursion error is coming from the pair_nhd_reaches() function which calls a recursive function to navigate the NHD and return all COMIDs that lie between two points (an upstream point and a downstream point). I compared the NHD lines (target p1_nhdv2reaches_sf) built locally with the NHD lines built within the container. The target dimensions are the same and all of the captured COMIDs are the same. The only difference I noted was very slight differences in the geometries (side note: why??). Using the container-built NHD lines, I looped pair_nhd_reaches over each of our PRMS segments and identified the following segments as causing the break: 1254_1, 1255_1, 1264_1, 2749_1.

If we look at 1254_1, for example, we see that the local NHD lines fall along the PRMS line, whereas the NHD lines built within the container fall slightly off of the PRMS line due to those small differences in the geometries. The recursive function depends on an object called nhd_paired that is created from a spatial join between the PRMS segment and the NHD lines. For segment 1254_1, that small difference in geometry results in NHD lines no longer being joined to the PRMS segment and so the recursive function is understandably upset that it is not receiving the expected input.

In pair_nhd_reaches I have a hard-coded limit on the (small) buffered distance for which we allow a spatial join between a PRMS segment and NHD lines that is right now set at 0.3 meters. I don't want to add this distance as an argument to the function because you get other idiosyncrasies with improperly matched NHD lines if you allow this buffer to be too big. I tested it out and if we expand the buffer to 0.35 m the container NHD lines seem to build properly within the xwalk target. The underlying differences in geometries do result in the output being different for one PRMS segment although that difference is relatively minor.

All that being said, I think I'm also leaning toward moving the xwalk target to a separate repo to reduce bloat within the inland salinity pipeline and also to allow the xwalk output to be more easily shared across projects.

[jds485]

Wow, that is weird! The p1_nhdv2reaches_sf are downloaded, right? I didn't see any additional processing in our code. I don't see why they would have different geometry when downloaded on a Windows vs. Linux

[jds485]

The Container version of Linux (Ubuntu LTS 20.04) has GDAL version 3.0.4 as the latest version available to download. So I think the only way to get a newer version of GDAL would be to upgrade to a newer version of Ubuntu, which wouldn't be a stable release (could change over time). If we keep the container version of GDAL as-is, I'm not sure what other geospatial issues this may cause. I'll note that even with GDAL 3.4.1 the crosswalk did not build on my non-container Ubuntu (which is also 20.04 and has libgdal version 3.0.4, but somehow installed sf to link with GDAL 3.4.1...I'm not sure how that worked out).