-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bus error with multithreading linear algebra #298
Comments
Calling multithreaded Julia code from Python is not well supported, but see these tips: https://cjdoris.github.io/PythonCall.jl/dev/faq/#Is-PythonCall/JuliaCall-thread-safe? Some people have had success with putting |
Hi @cjdoris , wondering if you might have any further insights on why it works for some but not for others. Multi-threading in Julia code called by Python is a key feature for our use case. We are hitting this same bus error, both on Intel-based and M1 Macbooks. FWIW running it in an ubuntu docker container fails as well, though with a Segfault error rather than bus error. I have tried with Python 3.8, 3.10, 3.11 and Julia 1.9 with PythonCall/juliacall 0.9.12 and 0.9.13. I slightly modified your example here for the new from juliacall import Main as jl
jl.seval(
"""
function worker()
for i in 1:10_000_000
a = Float64[]
push!(a, 0.42)
i % 1000 == 0 && println(i)
end
end
"""
)
jl.seval(
"""
begin
PythonCall.GC.disable()
t = Threads.@spawn worker()
println("waiting")
wait(t)
PythonCall.GC.enable()
end
"""
) It succeeds with 1 thread but I hit this error for any runs with multiple threads (using
Thanks for creating and maintaining such a handy package! |
I'm afraid I don't have time to investigate multithreading issues. Until it's more reliable in JuliaCall, you may be better off running Julia in a separate process instead. |
@cjdoris thanks for the reply. Totally understandable, I will post our findings here in case anyone else hitting this issue comes across this thread. We are happy to test out any suggestions, but the issue is around memory allocation in threaded code.
So currently our workarounds are
@aditya-sengupta your issue may be entirely different, but it may be worth trying PyJulia out to see if you still hit the bus error. It's pretty easy to try, see my MWE: import os
WORKER_FN_STR = """function worker()
a = []
for i in 1:1000000
# push!(a, Float32[0.42])
push!(a, Int64[4])
i % 1000 == 0 && println(i)
end
end"""
WORKER_RUN_STR = """begin
t=Base.Threads.@spawn worker()
println("waiting")
wait(t)
println("succeeded")
end"""
def run_juliacall(n_threads: int = 2):
# pip install juliacall
os.environ["JULIA_NUM_THREADS"] = str(n_threads)
from juliacall import Main as jl
jl.seval(WORKER_FN_STR)
jl.seval("PythonCall.GC.disable()")
jl.seval(WORKER_RUN_STR)
jl.seval("PythonCall.GC.enable()")
def run_pyjulia(n_threads: int = 2):
# pip install pyjulia
os.environ["JULIA_NUM_THREADS"] = str(n_threads)
import julia
_jl = julia.Julia(compiled_modules=False)
from julia import Main
Main.eval(WORKER_FN_STR)
Main.eval(WORKER_RUN_STR)
if __name__ == "__main__":
# juliacall will cause bus error
run_juliacall()
# pyjulia runs without issue
# run_pyjulia() |
JuliaCall causes a bus error for me too, but pyjulia causes this error:
I'm sure this is easily resolved but I'm not familiar enough with pyjulia to say how. |
@aditya-sengupta I think you just need to do |
Still encountering the same error - which Python version are you using?
…On Fri, Jun 9, 2023 at 2:15 PM Brian Dellabetta ***@***.***> wrote:
@aditya-sengupta <https://github.com/aditya-sengupta> I think you just
need to do ] add PyCall in your julia env first? Also, I have not tried
in Python 3.11
—
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AIK476UFHQL5VC34XWMXHYDXKOG67ANCNFSM6AAAAAAXKIEQSM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
3.8 mostly, but 3.10 had similar results. i know a lot of low-level stuff changed in 3.11 |
⭐ Happy Update ⭐ : After several weeks of troubleshooting, I think we have a robust and minimally invasive solution that doesn't require switching away from juliacall/PythonCall towards something like PyJulia. We ran the code snippet I pasted above through Threads.@threads for i in 1:n_samples
pairs[i] = do_something(data[i, :])
end becomes PythonCall.GC.disable()
Base.GC.enable(false)
Threads.@threads for i in 1:n_samples
pairs[i] = do_something(data[i, :])
if Threads.threadid() == 1
Base.GC.gc(false)
end
end
Base.GC.enable(true)
PythonCall.GC.enable()
Disclaimer: This has only been tested on Julia 1.9.1+ and Python 3.8. Some threading logic has been updated in the julia runtime and released as part of Julia 1.9.1, see discussion here, that is likely allowing this to work. Attempts on earlier Julia versions were failing but we didn't have the exact same code. @aditya-sengupta try upgrading julia to 1.9.1 and wrap your threaded code the GC disable/enable commands. see if that does the trick for you 🤞 |
Duplicate of #219 x-ref: JuliaLang/julia#50278 |
@brian-dellabetta I've been trying to multi-thread some python routines from Julia using the PythonCall side rather than JuliaCall. Should your solution work for this too? At the moment, the following code periodically causes a segfault (I can run using PythonCall
np = pyimport("numpy")
function test(n=10)
Base.GC.enable(false)
PythonCall.GC.disable()
Threads.@threads for i in 1:n
np.zeros((i,n))
if Threads.threadid() == 1
Base.GC.gc(false)
end
end
PythonCall.GC.enable()
Base.GC.enable(true)
end I'm using Julia 1.9.1 and PythonCall v0.9.13 on Ubuntu 22.04. Thanks in advance for any help! |
@nic-barbara I didn't try any direct calls of Python code inside the multi-threaded Julia code. My guess is this in general won't work, but you could also try the solution @vchuravy suggests, it seems much more robust and less hacky than my workaround. You can try it out on this branch by setting env var |
Using multi-threading to call back into Python is unlikely to "just" work, due to Python's GIL. I am not familiar with the details of PythonCall.jl and there might be other things that could go wrong here. In any case that should be an distinct issue. |
@vchuravy is correct, you can only call Python code from Julia thread 1 (because of the GIL). This is mentioned in the FAQ. |
That's a pain, but understandable. Thanks for the help! |
This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days. |
This issue has been closed because it has been stale for 7 days. You can re-open it if it is still relevant. |
Tried this again and it worked with the disable/enable-s! Thanks all! |
Affects: JuliaCall
Describe the bug
I'm running into a bus error in trying to port and speed up a linear algebra routine from Python to Julia.
In an environment with numpy and juliacall, the following code produces the error:
When run with
python jl_seg.py
, this produces the error message[1] 73474 bus error python jl_seg.py
The number preceding the "bus error" changes each time.
Your system
Please provide detailed information about your system:
Additional context
I've figured out that this is a combination of using
Threads.@threads
, the M1 chip, and juliacall; the same code runs as expected if it's single-threaded, if I run it on my 2018 MacBook Air, or if I run it purely in the Julia REPL (withrand(10,10)
in Julia replacing thenp.random.random
call).The text was updated successfully, but these errors were encountered: