conflict libcurl.so in Julia 1.8

[szcf-weiya]

I am using the latest v0.13.14 RCall.jl with R 4.2.1 and Julia 1.8.4 on Ubuntu 20.04.

julia> versioninfo()
Julia Version 1.8.4
Commit 00177ebc4fc (2022-12-23 21:32 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 4 virtual cores
Environment:
  LD_LIBRARY_PATH = /opt/R/4.2.1/lib/R/lib

julia> using RCall

R> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8          LC_NUMERIC=C             
 [3] LC_TIME=C.UTF-8           LC_COLLATE=C.UTF-8       
 [5] LC_MONETARY=C.UTF-8       LC_MESSAGES=C.UTF-8      
 [7] LC_PAPER=C.UTF-8          LC_NAME=C.UTF-8          
 [9] LC_ADDRESS=C.UTF-8        LC_TELEPHONE=C.UTF-8     
[11] LC_MEASUREMENT=C.UTF-8    LC_IDENTIFICATION=C.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.2.1

It works well but I found a problem related to libcurl.so.

I installed RCurl package in R, it works well, and the ldd status is as follows, which depends on/lib/x86_64-linux-gnu/libcurl.so.4

/opt/R/4.2.1/lib/R/library/RCurl/libs# ldd RCurl.so
	linux-vdso.so.1 (0x00007ffe7f5e3000)
	libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007fcbc78da000)
	libR.so => /opt/R/4.2.1/lib/R/lib/libR.so (0x00007fcbc73fd000)
...

However, when I call it in Julia 1.8, it throws

julia> @rlibrary("RCurl")
ERROR: REvalError: Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/opt/R/4.2.1/lib/R/library/RCurl/libs/RCurl.so':
  /opt/hostedtoolcache/julia/1.8.4/x64/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /opt/R/4.2.1/lib/R/library/RCurl/libs/RCurl.so)

If I understand correctly, it wants to use /opt/hostedtoolcache/julia/1.8.4/x64/bin/../lib/julia/libcurl.so instead of /lib/x86_64-linux-gnu/libcurl.so.4.

I have checked ldd inside the Julia session and the R-in-Julia session, both still show /lib/x86_64-linux-gnu/libcurl.so.4

julia> run(`ldd /opt/R/4.2.1/lib/R/library/RCurl/libs/RCurl.so`)
	linux-vdso.so.1 (0x00007fff9e536000)
	libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f4ba7185000)
	libR.so => /opt/R/4.2.1/lib/R/lib/libR.so (0x00007f4ba6ca8000)
...
R> system("ldd /opt/R/4.2.1/lib/R/library/RCurl/libs/RCurl.so")
	linux-vdso.so.1 (0x00007ffdae105000)
	libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f7322eb8000)
	libR.so => /opt/R/4.2.1/lib/R/lib/libR.so (0x00007f73229db000)
...

I found a workaround, set LD_PRELOAD=/lib/x86_64-linux-gnu/libcurl.so.4 before launching Julia. But a drawback later I found is that it would bring new problems when I use HTTP package, which also depends on libcurl.so, so if I pre-load the system version, it would break down the HTTP package which relied on Julia's libcurl.so

But the above issue does not occur in Julia 1.7 and even Julia 1.5.

After checking the NEWs of Julia 1.8, https://docs.julialang.org/en/v1/NEWS/
I guess it might be related to

External dependencies

• On Linux, now autodetects the system libstdc++ version, and automatically loads the system library if it is newer. The old behavior of loading the bundled libstdc++ regardless of the system version obtained by setting the environment variable JULIA_PROBE_LIBSTDCXX=0.
• Removed RPATH from the julia binary. On Linux this may break libraries that have failed to set RUNPATH.

but I am not sure. I tried to understand from the rpath viewpoint, but failed to get the logic why loading RCurl R package would be conflicted with Julia's libcurl.so.

Or maybe it is unrelated to the rpath part, but if so, why older Julia 1.7 works? Which change in Julia 1.8 caused the difference?

Hope anyone can help explain it. Many thanks!

[giordano]

But the above issue does not occur in Julia 1.7 and even Julia 1.5.

What exactly doesn't occur in Julia v1.7 or 1.5? I don't see how you could not get the same error without preloading.

[szcf-weiya]

@giordano That is quite my confusion. Since I ran the same workflow in the GitHub actions with different Julia versions. It turns out that only Julia 1.8 failed. So I thought there should be something different in Julia 1.8.

[giordano]

I asked what exactly doesn't occur, I don't think your answered the question 🙂

[szcf-weiya]

Sorry for the confusion. I mean no error when loading RCurl package in Julia

julia> @rlibrary("RCurl")

[angusmoore]

I have run into this same issue, and did a bit of digging today (though I'm well out of my depth on this, so apologies in advance for any obvious mistakes).

Here's a quick Dockerfile way to recreate the problem. You can also verify that this problem didn't occur on 1.7 and earlier by changing FROM julia:latest => FROM julia:1.7

FROM julia:latest

RUN  apt-get -y update && \
     apt-get -y install ssh build-essential r-base libcurl4-openssl-dev

# user
RUN useradd --create-home -u 999 --shell /bin/bash newuser

RUN mkdir /home/newuser/app
COPY . /home/newuser/app
WORKDIR /home/newuser/app

RUN chown newuser:newuser -R /home/newuser/app

USER newuser

RUN julia -e "using Pkg; Pkg.activate(\".\"); Pkg.add(\"RCall\"); Pkg.add(\"HTTP\");"

# Install R packages
RUN mkdir -p /home/newuser/R/site-library
ENV R_LIBS_USER "/home/newuser/R/site-library"
RUN Rscript -e 'install.packages(c("RCurl"), lib = "/home/newuser/R/site-library")'
RUN Rscript -e 'require("RCurl")'; 

ENTRYPOINT ["/bin/bash"]

Can then recreate the import failure by:

julia> using RCall, HTTP
julia> @rimport RCurl
ERROR: REvalError: Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/newuser/R/site-library/RCurl/libs/RCurl.so':
  /usr/local/julia/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /home/newuser/R/site-library/RCurl/libs/RCurl.so)

Running R by itself (not from within Julia) has no issues with library(RCurl).

I think the problem is RCurl is trying to link to the version of libcurl Julia has (ie /usr/local/julia/bin/../lib/julia/libcurl.so). But that doesn't have a libcurl.so.4:

julia> run(`ldd /usr/local/julia/bin/../lib/julia/libcurl.so`)
	linux-vdso.so.1 (0x00007fffa45cb000)
	libnghttp2.so.14 => /usr/local/julia/bin/../lib/julia/libnghttp2.so.14 (0x00007f4883400000)
	libssh2.so.1 => /usr/local/julia/bin/../lib/julia/libssh2.so.1 (0x00007f4883000000)
	libmbedtls.so.14 => /usr/local/julia/bin/../lib/julia/libmbedtls.so.14 (0x00007f4882c00000)
	libmbedx509.so.1 => /usr/local/julia/bin/../lib/julia/libmbedx509.so.1 (0x00007f4882800000)
	libmbedcrypto.so.7 => /usr/local/julia/bin/../lib/julia/libmbedcrypto.so.7 (0x00007f4882400000)
	libz.so.1 => /usr/local/julia/bin/../lib/julia/libz.so.1 (0x00007f4882000000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4883b4c000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4883b47000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f488221f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4883b5b000)
Process(`ldd /usr/local/julia/bin/../lib/julia/libcurl.so`, ProcessExited(0))

Workaround

Explicitly setting LD_LIBRARY_PATH="/lib/x86_64-linux-gnu" seems to encourage R to use the system installed version of libcurl, instead of Julia's. And that seems to solve the problem. I am guessing (though unsure) that this is what was happening on Julia 1.7 and earlier.

[simonbyrne]

What actually changed is that Julia 1.8.4 switched to using RUNPATH instead of RPATH (which was used in Julia 1.8.3 and earlier) for specifying local library paths:

├ objdump -x /central/software/julia/1.8.4/bin/julia |grep 'R.*PATH'
  RUNPATH              $ORIGIN/../lib:$ORIGIN/../lib/julia
├ objdump -x /central/software/julia/1.8.3/bin/julia |grep 'R.*PATH'
  RPATH                $ORIGIN/../lib:$ORIGIN/../lib/julia

(since libcurl.so is in <juliadir>/lib/julia)

From https://amir.rachum.com/shared-libraries/:

The only difference between rpath and runpath is the order they are searched in. Specifically, their relation to LD_LIBRARY_PATH - rpath is searched in before LD_LIBRARY_PATH while runpath is searched in after. The meaning of this is that rpath cannot be changed dynamically with environment variables while runpath can.

[szcf-weiya]

@simonbyrne Thanks for pointing out the search order difference between rpath and runpath. I also caught the difference, but failed to get the logic of the error. (And the same error also occurs on Julia1.8.1 after retesting)

In Julia 1.7.0, with rpath, we can properly call RCurl, which relies on the system's libcurl.so
In Julia 1.8.4, with runpath, calling RCurl would call Julia's libcurl.so

BTW, I just retest it in Julia 1.8.1, calling RCurl throws the same error as the one in Julia1.8.4

julia> using RCall

julia> ENV["R_HOME"]
"/usr/lib/R"

julia> @rlibrary("RCurl")
ERROR: REvalError: Error in dyn.load(file, DLLpath = DLLpath, ...) : 

in all cases, RCall.jl is based on the same system R /usr/lib/R.

[simonbyrne]

Hmm, so maybe the problem is caused by something else.

[palday]

I'm closing this as stale. If it's still a problem on Julia 1.10, then please open another issue.