How to correctly spawn a process (mcparallel) from inside a RestRserve end point?

[dereckmezquita]

I ran into an interesting problem recently: how to spawn a parallel independent process from a RestRserve end point.

Related: #216

I am not an expert in parallel nor RestRserve's internals, my goal here is to start a discussion and get a better understanding. I want to get a deeper understanding so to better debug in the future.

The problem and miminal example: segfault/permission errors

I was encountering segmentation faults when using parallel and later packages together within a RestRserve endpoint. Setting the detached argument to TRUE in the mcparallel function seems to have resolved the immediate issue.

Here is a minimal example:

box::use(RestRserve[ Application, BackendRserve ])
box::use(fs)
box::use(later)
box::use(parallel)

messageParallel <- function(...) {
    system(sprintf('echo "%s"', paste0(..., collapse = "")))
}
app <- Application$new()

# NOTE: narrowed down the source of the segfault
# error happens when later is used with parallel to read write files
parallel_later_add_bot <- function(bot_id) {
    parallel$mcparallel({
        messageParallel("RUNNING PARALLEL LATER BOT_ID ", bot_id)
        parallel_loop <- later$create_loop()
        later$later(function() {
            messageParallel("EXECUTING LATER BOT_ID ", bot_id)
        }, 3, loop = parallel_loop)

        while(!later$loop_empty(loop = parallel_loop)) {
            later$run_now(loop = parallel_loop)
        }
        messageParallel("EXITING PARALLEL LATER BOT_ID ", bot_id)
    })
    return(TRUE)
}

app$add_get("/add-id", function(request, response) {
    bot_id <- request$parameters_query[["bot_id"]]
    if (!parallel_later_add_bot(bot_id)) {
        response$set_status_code(400)
        response$set_body(list(error = "Bot ID already exists"))
        return()
    }

    response$set_body(list(message = sprintf("Added bot %s", bot_id)))
})

backend <- BackendRserve$new()
backend$start(app, http_port = 8080, background = FALSE)

(base) work@Derecks-MacBook-Air dev % Rscript restrserve-rds-segfault/index.R
{"timestamp":"2024-08-18 14:52:39.162872","level":"INFO","name":"Application","pid":81807,"msg":"","context":{"http_port":8080,"endpoints":{"HEAD":"/add-id","GET":"/add-id"}}}
-- running Rserve in this R session (pid=81807), 2 server(s) --
(This session will block until Rserve is shut down)
RUNNING PARALLEL LATER BOT_ID 2kj34dd34235323
RUNNING PARALLEL LATER BOT_ID 2kj34dd34235323
RUNNING PARALLEL LATER BOT_ID 2kj34dd34235323
RUNNING PARALLEL LATER BOT_ID 2kj34dd34235323
EXECUTING LATER BOT_ID 2kj34dd34235323
EXITING PARALLEL LATER BOT_ID 2kj34dd34235323
Warning: stack imbalance in '=', 124 then 132
Warning: stack imbalance in '{', 120 then 134
Warning: stack imbalance in 'if', 118 then 135
Warning: stack imbalance in '=', 137 then 138

 *** caught segfault ***
address 0x0, cause 'invalid permissions'
EXECUTING LATER BOT_ID 2kj34dd34235323
EXITING PARALLEL LATER BOT_ID 2kj34dd34235323
EXECUTING LATER BOT_ID 2kj34dd34235323
EXITING PARALLEL LATER BOT_ID 2kj34dd34235323

 *** caught bus error ***
address 0x12d686e48, cause 'invalid alignment'
EXECUTING LATER BOT_ID 2kj34dd34235323
EXITING PARALLEL LATER BOT_ID 2kj34dd34235323

 *** caught segfault ***
address 0x4000000102ad7020, cause 'invalid permissions'
R(81913,0x16de9b000) malloc: *** error for object 0x1027ee318: pointer being realloc'd was not allocated
R(81913,0x16de9b000) malloc: *** set a breakpoint in malloc_error_break to debug
RUNNING PARALLEL LATER BOT_ID 2kj34dd34235323
EXECUTING LATER BOT_ID 2kj34dd34235323
EXITING PARALLEL LATER BOT_ID 2kj34dd34235323

 *** caught segfault ***
address 0x28, cause 'invalid permissions'

Observations and solution: mcparallel(detached = TRUE)

1. Using parallel alone in the endpoint worked without segfaults and without needing the detached argument:

parallel_syssleep_add_bot <- function(bot_id) {
    parallel$mcparallel({
        messageParallel("RUNNING PARALLEL SYSSLEEP BOT_ID ", bot_id)
        Sys.sleep(10)
        messageParallel("EXITING PARALLEL SYSSLEEP BOT_ID ", bot_id)
    })
    return(TRUE)
}

2. When combining parallel with later, using detached = TRUE seems to prevent errors:

parallel_later_add_bot <- function(bot_id) {
    parallel$mcparallel({
        messageParallel("RUNNING PARALLEL LATER BOT_ID ", bot_id)
        parallel_loop <- later$create_loop()
        later$later(function() {
            messageParallel("EXECUTING LATER BOT_ID ", bot_id)
        }, 3, loop = parallel_loop)

        while(!later$loop_empty(loop = parallel_loop)) {
            later$run_now(loop = parallel_loop)
        }
        messageParallel("EXITING PARALLEL LATER BOT_ID ", bot_id)
    }, detached = TRUE)
    return(TRUE)
}

Discussion

While this solution appears to work for now, I'm concerned that I might have gotten lucky in solving this and could encounter more issues down the line. I'm seeking expert insights and explanations on the following points:

1. Mechanics of detached = TRUE:
   Can someone explain why using detached = TRUE resolves the segmentation fault? What's happening under the hood?
2. RestRserve and process interaction:
   How does RestRserve's request handling (which involves forking) interact with the processes we're creating manually? Are there potential conflicts or race conditions? I know that RestRserve itself uses parallel.
3. Robustness of this solution:
   Is this a reliable approach for production use? Are there scenarios where it might break or cause unexpected behavior?
4. Performance and scalability:
   What are the performance implications of using detached processes within RestRserve endpoints? How might this affect the application's scalability?
5. Best practices and alternatives:
   Are there best practices for handling asynchronous operations or spawning processes in RestRserve? Are there alternative approaches that might be more robust or efficient?
6. Potential pitfalls:
   What kind of issues might I encounter in the future with this approach? Are there any specific areas I should be cautious about?

Thank you very much and I hope to contribute some of my experience for those wanting to do something similar.