Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Openmpi bugs #31

Closed
wants to merge 63 commits into from
Closed

Openmpi bugs #31

wants to merge 63 commits into from

Commits on Mar 6, 2023

  1. Keeping only the suspected faulty test (by commenting out rest (!))

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    23b59d3 View commit details
    Browse the repository at this point in the history

  2. Removing Turing from test to isolate only one error at the time

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    40001d0 View commit details
    Browse the repository at this point in the history

  3. Check behaviour on different threadlevels

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    ff5aac9 View commit details
    Browse the repository at this point in the history

  4. Back to same threadlevel as not changing crashing behaviour

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    d00ff53 View commit details
    Browse the repository at this point in the history

  5. More conservative tag ub?

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    69bb270 View commit details
    Browse the repository at this point in the history

  6. Revert "More conservative tag ub?" 

    This reverts commit 69bb270.
    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    d94252f View commit details
    Browse the repository at this point in the history

  7. Trying on OpenMPI instead of mpich

    @alexandrebouchard
    alexandrebouchard committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    bcf016d View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2023

  1. Going back to mpich, trying threadlevel = :multiple

    @alexandrebouchard
    alexandrebouchard committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    379df4b View commit details
    Browse the repository at this point in the history

  2. Back to funneled as :multiple still crashes

    @alexandrebouchard
    alexandrebouchard committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    2cd3571 View commit details
    Browse the repository at this point in the history

  3. Further simplification

    @alexandrebouchard
    alexandrebouchard committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    c8a0261 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. Candidate fix

    @alexandrebouchard
    alexandrebouchard committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    6217650 View commit details
    Browse the repository at this point in the history

  2. Reintroducing other tests

    @alexandrebouchard
    alexandrebouchard committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    1f16936 View commit details
    Browse the repository at this point in the history

  3. purge tests from Turing, use only DynamicPPL

    @miguelbiron
    miguelbiron committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    0ca198f View commit details
    Browse the repository at this point in the history

  4. Another free

    @alexandrebouchard
    alexandrebouchard committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    ee276fc View commit details
    Browse the repository at this point in the history

  5. Trying to run tests with more rounds

    @alexandrebouchard
    alexandrebouchard committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    491c26b View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. system MPI test

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    fab7934 View commit details
    Browse the repository at this point in the history

  2. fix typos

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    2ed0aae View commit details
    Browse the repository at this point in the history

  3. remove step that does not make sense outside MPI.jl

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    c271261 View commit details
    Browse the repository at this point in the history

  4. setup MPIPreferences in the test env

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    e3f569a View commit details
    Browse the repository at this point in the history

  5. possible fix to Pkg missing

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    77824ea View commit details
    Browse the repository at this point in the history

  6. possible fix to Pkg missing

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    c69b76a View commit details
    Browse the repository at this point in the history

  7. add missing env

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    523f9c1 View commit details
    Browse the repository at this point in the history

  8. another try

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    949b628 View commit details
    Browse the repository at this point in the history

  9. test now has its own Project.toml

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    895a45f View commit details
    Browse the repository at this point in the history

  10. force Pkg.instantiate in ChildProcess

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    13c9cd2 View commit details
    Browse the repository at this point in the history

  11. dont use --project in ChildProcess

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    1772d82 View commit details
    Browse the repository at this point in the history

  12. ChildProcess inherits the exact same active project

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    bdf71ce View commit details
    Browse the repository at this point in the history

  13. new approach

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    2bcd825 View commit details
    Browse the repository at this point in the history

  14. Changing approach for Isend/isend: explicit Waitall for all requests 

    A crash seemed to occur in CI during garbage collection

julia:3364 terminated with signal 11 at PC=7f997aaec971 SP=7f9936ccb970.  Backtrace:
/opt/hostedtoolcache/julia/1.8.5/x64/bin/../lib/julia/libjulia-internal.so.1(ijl_gc_safepoint+0x11)[0x7f997aaec971]

This looks like a crash during garbage collection code (this would be consistent with the fact that it only shows up with allocation-heavy test, i.e. Turing).

So even though the finalizer checks first before calling free, maybe there is something flaky there.

#30 (comment)
    @alexandrebouchard
    alexandrebouchard committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    eadbfcd View commit details
    Browse the repository at this point in the history

  15. Is this a Turing multi-threading issue?

    @alexandrebouchard
    alexandrebouchard committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    57d0921 View commit details
    Browse the repository at this point in the history

  16. Testing single-thread for all 

    Multi-threading issue seems not specific to Turing as crash occurred
for Blang in last commit.
    @alexandrebouchard
    alexandrebouchard committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    63aad97 View commit details
    Browse the repository at this point in the history

  17. Adding a temporary "dry run" to see if problem cause by some compilation

    @alexandrebouchard
    alexandrebouchard committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    7a8de7e View commit details
    Browse the repository at this point in the history

  18. Commenting out Turing and Blang, to see behaviour on toy_mvn and swap… 

    …(0.5)
    @alexandrebouchard
    alexandrebouchard committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    5812127 View commit details
    Browse the repository at this point in the history

  19. openmpi_jll test

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    270333f View commit details
    Browse the repository at this point in the history

  20. openmpi_jll test

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    2b49008 View commit details
    Browse the repository at this point in the history

  21. openmpi_jll test

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    9c6497f View commit details
    Browse the repository at this point in the history

  22. openmpi_jll test

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    d68dc6c View commit details
    Browse the repository at this point in the history

  23. more messages from Child processes

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    38de7cf View commit details
    Browse the repository at this point in the history

  24. extra arg to OpenMPI

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    8537307 View commit details
    Browse the repository at this point in the history

  25. rm mpi version query

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    c2b732b View commit details
    Browse the repository at this point in the history

  26. use old julia_cmd + rm manifest as in MPI.jl CI

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    0e82a84 View commit details
    Browse the repository at this point in the history

  27. add --oversubscribe for OpenMPI + rethrow exception

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    82a4210 View commit details
    Browse the repository at this point in the history

  28. simplify logging

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    700c712 View commit details
    Browse the repository at this point in the history

  29. force instantiate + precompile

    @miguelbiron
    miguelbiron committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    ebcb3a6 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Add mpi_args to mpi_test

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    ad69b34 View commit details
    Browse the repository at this point in the history

  2. Temporary: trying to speed up some key tests

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    6bde8fe View commit details
    Browse the repository at this point in the history

  3. Fix

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    7b5e7d4 View commit details
    Browse the repository at this point in the history

  4. Fix the fix + reintroducing the system-MPI tests

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    fc34b85 View commit details
    Browse the repository at this point in the history

  5. Add back libmpich-dev to resume investigation on ghostbug

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    5ba839e View commit details
    Browse the repository at this point in the history

  6. Trying to simplify CI setup needed to reproduce ghostbug

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    2b37c0f View commit details
    Browse the repository at this point in the history

  7. Fix last commit

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    0847901 View commit details
    Browse the repository at this point in the history

  8. toy_mvn not enough to manifest ghostbug, trying Turing

    @alexandrebouchard
    alexandrebouchard committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    f75a280 View commit details
    Browse the repository at this point in the history

  9. test mpich+openmpi using brew

    @miguelbiron
    miguelbiron committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    93033fc View commit details
    Browse the repository at this point in the history

  10. fix wrong abi detection for mpich

    @miguelbiron
    miguelbiron committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    d46d429 View commit details
    Browse the repository at this point in the history

  11. move xtra args to MPI struct + remove prints + failsafe for empty cur… 

    …rent project
    @miguelbiron
    miguelbiron committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    7a11074 View commit details
    Browse the repository at this point in the history

  12. mpiexec args for childprocess

    @miguelbiron
    miguelbiron committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    d89911a View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2023

  1. re-introduce all CI tests + fix bug in building mpi cmd

    @miguelbiron
    miguelbiron committed Mar 11, 2023
    Configuration menu
    Copy the full SHA
    aa233e8 View commit details
    Browse the repository at this point in the history

  2. add support for using without Project.toml

    @miguelbiron
    miguelbiron committed Mar 11, 2023
    Configuration menu
    Copy the full SHA
    42993fd View commit details
    Browse the repository at this point in the history

  3. add comment explaining why we wait on Isend

    @miguelbiron
    miguelbiron committed Mar 11, 2023
    Configuration menu
    Copy the full SHA
    42bdd75 View commit details
    Browse the repository at this point in the history

  4. mpiexec_args is a Cmd now

    @miguelbiron
    miguelbiron committed Mar 11, 2023
    Configuration menu
    Copy the full SHA
    c5cb70a View commit details
    Browse the repository at this point in the history

  5. re-instate all tests

    @miguelbiron
    miguelbiron committed Mar 11, 2023
    Configuration menu
    Copy the full SHA
    9e31b49 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2023

  1. Do we have the same bug with 1 thread

    @alexandrebouchard
    alexandrebouchard committed Mar 12, 2023
    Configuration menu
    Copy the full SHA
    f93548d View commit details
    Browse the repository at this point in the history

  2. Explicitly disable GC in all pigeons runs

    @alexandrebouchard
    alexandrebouchard committed Mar 12, 2023
    Configuration menu
    Copy the full SHA
    495b4eb View commit details
    Browse the repository at this point in the history