[draft] tar_terra_rast_wrap: multi-target method to preserve SpatRaster metadata #63
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,47 @@ | ||||||
| #' Manage geotargets Cache Files | ||||||
| #' | ||||||
| #' The `geotargets` cache is a folder containing one subfolder for each target created using `tar_terra_rast_wrap()` method. Each subfolder contains one or more files containing data and metadata to support a `terra` `SpatRaster` object. | ||||||
| #' | ||||||
| #' The cache directory can be customized using `geotargets_option_set("cache.dir")` or environment variable `GEOTARGETS_CACHE_DIR`. | ||||||
| #' | ||||||
| #' Periodically you may want to purge files in the cache using `geotargets_destroy_cache()`. | ||||||
| #' | ||||||
| #' @param name character. Target name (default: `NULL` will delete all target cache files) | ||||||
| #' @param init logical. Re-create empty cache directory? Default: `FALSE` | ||||||
| #' | ||||||
| #' @return integer. 0 for success, 1 for failure, invisibly. | ||||||
| #' @export | ||||||
| #' @rdname geotargets-cache | ||||||
| #' @examples | ||||||
| #' \dontrun{ | ||||||
| #' | ||||||
| #' # delete cache folder for target named "foo" | ||||||
| #' geotargets_destroy_cache("foo") | ||||||
| #' | ||||||
| #' # create empty folder | ||||||
| #' geotargets_init_cache("foo") | ||||||
| #' | ||||||
| #' # delete and recreate folder | ||||||
| #' geotargets_destroy_cache("foo", init = TRUE) | ||||||
| #' | ||||||
| #' # delete all cache files | ||||||
| #' geotargets_destroy_cache() | ||||||
| #' | ||||||
| #' } | ||||||
| geotargets_destroy_cache <- function(name = NULL, init = FALSE) { | ||||||
| cachedir <- geotargets_option_get("cache.dir") | ||||||
| target_cache_dir <- file.path(cachedir %||% "geotargets_cache", name %||% "") | ||||||
| res <- unlink(target_cache_dir, recursive = TRUE) | ||||||
| if (init) geotargets_init_cache(name = name) | ||||||
| invisible(res) | ||||||
| } | ||||||
|
|
||||||
| #' @rdname geotargets-cache | ||||||
| #' @export | ||||||
| geotargets_init_cache <- function(name = NULL) { | ||||||
| cachedir <- geotargets_option_get("cache.dir") | ||||||
| target_cache_dir <- file.path(cachedir %||% "geotargets_cache", name %||% "") | ||||||
|
There was a problem hiding this comment.
Suggested change
Maybe? Just for consistency with |
||||||
| dir.create(target_cache_dir, | ||||||
| showWarnings = FALSE, | ||||||
| recursive = TRUE) | ||||||
| } | ||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,153 @@ | ||||||||
| #' Create a wrapped terra SpatRaster target | ||||||||
| #' | ||||||||
| #' Provides a target format for [terra::SpatRaster-class] objects backed by | ||||||||
| #' [terra::PackedSpatRaster-class] and file-based targets in the `geotargets` | ||||||||
| #' cache folder (`cachedir`). | ||||||||
| #' | ||||||||
| #' @param filetype character. File format expressed as GDAL driver names passed | ||||||||
| #' to [terra::writeRaster()] | ||||||||
| #' @param gdal character. GDAL driver specific datasource creation options | ||||||||
| #' passed to [terra::writeRaster()] | ||||||||
| #' @param cachedir character. Path to directory where file sources for `PackedSpatRaster` objects can be stored. Default: `"geotargets_cache"` when `geotargets::geotargets_option_get("cache.dir")` is not set. | ||||||||
| #' @param ... Additional arguments not yet used | ||||||||
| #' @seealso [geotargets_destroy_cache()] [geotargets_init_cache()] | ||||||||
| #' @note Although you may pass any supported GDAL vector driver to the | ||||||||
| #' `filetype` argument, not all formats are guaranteed to work with | ||||||||
| #' `geotargets`. At the moment, we have tested `GTiff` and `GPKG` and | ||||||||
| #' they appear to work generally. | ||||||||
| #' | ||||||||
| #' @inheritParams targets::tar_target | ||||||||
| #' @importFrom rlang %||% arg_match0 | ||||||||
| #' @seealso [targets::tar_target_raw()] | ||||||||
| #' @export | ||||||||
| #' @examples | ||||||||
| #' if (Sys.getenv("TAR_LONG_EXAMPLES") == "true") { | ||||||||
| #' targets::tar_dir({ # tar_dir() runs code from a temporary directory. | ||||||||
| #' library(geotargets) | ||||||||
| #' targets::tar_script({ | ||||||||
| #' list( | ||||||||
| #' geotargets::tar_terra_rast( | ||||||||
| #' terra_rast_example, | ||||||||
| #' system.file("ex/elev.tif", package = "terra") |> terra::rast() | ||||||||
| #' ) | ||||||||
| #' ) | ||||||||
| #' }) | ||||||||
| #' targets::tar_make() | ||||||||
| #' x <- targets::tar_read(terra_rast_example) | ||||||||
| #' }) | ||||||||
| #'} | ||||||||
| tar_terra_rast_wrap <- function(name, | ||||||||
| command, | ||||||||
| pattern = NULL, | ||||||||
| filetype = "GTiff", | ||||||||
| gdal = geotargets::geotargets_option_get("gdal.raster.creation.options"), | ||||||||
| cachedir = geotargets::geotargets_option_get("cache.dir"), | ||||||||
| ..., | ||||||||
| tidy_eval = targets::tar_option_get("tidy_eval"), | ||||||||
| packages = targets::tar_option_get("packages"), | ||||||||
| library = targets::tar_option_get("library"), | ||||||||
| repository = targets::tar_option_get("repository"), | ||||||||
| iteration = targets::tar_option_get("iteration"), | ||||||||
| error = targets::tar_option_get("error"), | ||||||||
| memory = targets::tar_option_get("memory"), | ||||||||
| garbage_collection = targets::tar_option_get("garbage_collection"), | ||||||||
| deployment = targets::tar_option_get("deployment"), | ||||||||
| priority = targets::tar_option_get("priority"), | ||||||||
| resources = targets::tar_option_get("resources"), | ||||||||
| storage = targets::tar_option_get("storage"), | ||||||||
| retrieval = targets::tar_option_get("retrieval"), | ||||||||
| cue = targets::tar_option_get("cue")) { | ||||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||||
| filetype <- filetype %||% "GTiff" | ||||||||
| gdal <- gdal %||% character(0) | ||||||||
| cachedir <- cachedir %||% "geotargets_cache" | ||||||||
|
|
||||||||
| #check that filetype option is available | ||||||||
| drv <- get_gdal_available_driver_list("raster") | ||||||||
| filetype <- rlang::arg_match0(filetype, drv$name) | ||||||||
|
|
||||||||
| check_pkg_installed("terra") | ||||||||
|
|
||||||||
| name <- targets::tar_deparse_language(substitute(name)) | ||||||||
|
|
||||||||
| envir <- targets::tar_option_get("envir") | ||||||||
|
|
||||||||
| command <- targets::tar_tidy_eval( | ||||||||
| expr = as.expression(substitute(command)), | ||||||||
| envir = envir, | ||||||||
| tidy_eval = tidy_eval | ||||||||
| ) | ||||||||
|
|
||||||||
| pattern <- targets::tar_tidy_eval( | ||||||||
| expr = as.expression(substitute(pattern)), | ||||||||
| envir = envir, | ||||||||
| tidy_eval = tidy_eval | ||||||||
| ) | ||||||||
|
|
||||||||
| .format_terra_rast_wrap_write <- eval(substitute(function(object, path) { | ||||||||
| # TODO: provide mapping of major file types to extensions | ||||||||
| extension <- switch(filetype, | ||||||||
| "GTiff" = ".tif", | ||||||||
| "GPKG" = ".gpkg", | ||||||||
| "") | ||||||||
|
Comment on lines
+88
to
+91
There was a problem hiding this comment. Is this necessary? We don't need file extensions when writing to |
||||||||
| saveRDS(terra::wrapCache(object, | ||||||||
| filename = file.path(cachedir, | ||||||||
| basename(path), | ||||||||
| paste0(basename(path), | ||||||||
| extension)), | ||||||||
| filetype = filetype, | ||||||||
| gdal = gdal, | ||||||||
| overwrite = TRUE), | ||||||||
| file = path) | ||||||||
| }, list(cachedir = cachedir, filetype = filetype, gdal = gdal))) | ||||||||
|
|
||||||||
| geotargets_init_cache(name = name) | ||||||||
|
|
||||||||
| # rast_cache_init <- targets::tar_target_raw( | ||||||||
| # paste0(name, "_cache_init"), | ||||||||
| # str2expression(paste0("normalizePath(file.path(", | ||||||||
| # shQuote(cachedir), ", ", | ||||||||
| # shQuote(name), "))")), | ||||||||
| # format = "file" | ||||||||
| # ) | ||||||||
|
|
||||||||
| rast_wrap <- targets::tar_target_raw( | ||||||||
| name = name, | ||||||||
| command = command, | ||||||||
| pattern = pattern, | ||||||||
| packages = packages, | ||||||||
| library = library, | ||||||||
| format = targets::tar_format( | ||||||||
| read = function(path) terra::unwrap(readRDS(path)), | ||||||||
| write = .format_terra_rast_wrap_write, | ||||||||
| marshal = function(object) terra::wrap(object), | ||||||||
| unmarshal = function(object) terra::unwrap(object) | ||||||||
| ), | ||||||||
| repository = repository, | ||||||||
| iteration = iteration, | ||||||||
| error = error, | ||||||||
| memory = memory, | ||||||||
| garbage_collection = garbage_collection, | ||||||||
| deployment = deployment, | ||||||||
| priority = priority, | ||||||||
| resources = resources, | ||||||||
| storage = storage, | ||||||||
| retrieval = retrieval, | ||||||||
| cue = cue | ||||||||
| ) | ||||||||
|
|
||||||||
| rast_cache_files <- targets::tar_target_raw( | ||||||||
| paste0(name, "_cache_files"), | ||||||||
| str2expression(paste0(" | ||||||||
| list.files( | ||||||||
| file.path(", shQuote(cachedir), ", ", shQuote(name),"), | ||||||||
| full.names = TRUE, | ||||||||
| recursive = TRUE | ||||||||
| )")), | ||||||||
| format = "file_fast", | ||||||||
|
There was a problem hiding this comment. You could allow the option of either "file" or "file_fast" here, but I'm guessing it doesn't really matter since it seems like nothing will ever depend on this target. |
||||||||
| deps = name | ||||||||
| ) | ||||||||
|
Comment on lines
+138
to
+148
There was a problem hiding this comment. I'm questioning whether this target even needs to exist. Unless there is a way to make it be "upstream" of the |
||||||||
|
|
||||||||
| list(#rast_cache_init, | ||||||||
| rast_wrap, | ||||||||
| rast_cache_files) | ||||||||
| } | ||||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if there is a way that this could set some "flag" that could be used to invalidate the "upstream" target through a custom
cue? Or perhaps it runstar_invalidate()on all targets created withtar_terra_rast_wrap()when run? I think it's fine that manually deleting a file from the cache breaks the pipeline, but I think any "official" way of deleting the cache should correctly invalidate targets.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, now that I think of it, the directory names inside the cache are target names, yeah? So if this could get all those dir names and pass them to
tar_invalidate(any_of(dirnames))I think it would make this function a lot more useful.