Add support for Sierra port

[dongahn]

@lee218llnl: This PR is definitely experimental; please don't merge.

To facilitate your STAT testing, I installed this version into /usr/global/tools/launchmon/blueos_3_ppc64le_ib/default for both CZ and RZ.

TODOs:

• Verify all of the test cases. (I only verified test.launch_1 and test_attach_1. Even then there are some minor issues with the launch case.

• Add support for exec. Right now for launch mode, you should use /opt/ibm/spectrum_mpi/jsm_pmix/bin/stock/jsrun instead of /opt/ibm/spectrum_mpi/jsm_pmix/bin/jsrun because the latter does exec the former and this version still can't handle exec event yet. This is okay with attach mode.

• Add support for better handling of threads for attach case

• Test it on other key platforms

• When IBM fixes MPIR_attach_fifo issue, revert the current work around

[dongahn]

BTW, I built this with --enable-verbose=/usr/global/tools/launchmon/logs/blueos_3_ppc64le_ib so your backend logs will go there. Further, it is configured with --enable-debug.

[dongahn]

From @lee218llnl:

I had to add /collab/usr/global/tools/launchmon/blueos_3_ppc64le_ib/launchmon-1.1.0-20180716/etc/rm_ibm_spectrum.conf. Otherwise, initial testing (attach and launch) are looking OK. My STAT build in /collab/usr/global/tools/stat/blueos_3_ppc64le_ib/stat-test is using this new launchmon build.

Will add this new config into install target.

[dongahn]

From @lee218llnl:

Occasionally I get hangs with STAT, particularly after running it multiple times. It appears to be in lmon__fe.cxx on line 4601 in a pthread_cond_timedwait. I don’t know if this is an actual affect or just a correlation, but it seems like if I subsequently attach TV to the job and detach TV, then I am able to attach again with STAT.

I have also seen hang-like behavior (looping) in cobo on cobo_connect_hostname. This also appears to happen if I aggressively attach/detach/attach STAT multiple times.

I looked the (mis)behavior with Greg this morning. At first glance, this looks to me like issues with jsrun's FIFO support and co-location support. I will create a new issue for this to track. If this is IBM, we will need a few simpler test cases.

[dongahn]

@lee218llnl:

c723213: jsrun now fixes the MPIR_attach_fifo issue of expecting ASCII 1 ('1') instead of integer 1. Even after adjusting this, I think I still see the same intermittent issue you reported:

Occasionally I get hangs with STAT, particularly after running it multiple times. It appears to be in lmon__fe.cxx on line 4601 in a pthread_cond_timedwait.

As you can see from here, my initial suspicion is jsrun occasionally failing to launch the back-end daemon. I will spend some more time to look more closely.

[dongahn]

A hang occurs about once every 8 - 10 times with test.attach_1 case.

Trace from the successful runs:

<Sep 17 17:08:56> <LMON FE API> (INFO): GCRYPT shared_key: 1501848644
<Sep 17 17:08:56> <LMON FE API> (INFO): GCRYPT setkey, 31303531:36383438:3434:0
<Sep 17 17:08:56> <Launchmon> (INFO): linux_launchmon_t initialized.
<Sep 17 17:08:56> <Driver> (INFO): now creating a process object based on pid=27599, exe=/opt/ibm/spectrum_mpi/jsm_pmix/bin/stock/jsrun
<Sep 17 17:08:56> <Symtable> (INFO): reading linkage symbol table for image[=jsrun]
<Sep 17 17:08:56> <Symtable> (INFO): interpreter found in .interp section [/lib64/ld-2.17.so]
<Sep 17 17:08:56> <Symtable> (INFO): reading linkage symbol table for image[=ld-2.17.so]
<Sep 17 17:08:56> <Launchmon> (INFO): The RM process has just been trapped due to attach
<Sep 17 17:08:56> <Launchmon> (INFO):  trap after attach event handler invoked.
<Sep 17 17:08:56> <Symtable> (INFO): reading linkage symbol table for image[=libpthread.so.0]
<Sep 17 17:08:56> <Symtable> (INFO): reading linkage symbol table for image[=libc.so.6]
<Sep 17 17:08:56> <Launchmon> (INFO): trap after attach event handler completed.
<Sep 17 17:08:56> <Launchmon> (INFO): Just continued the RM process out of the first trap
<Sep 17 17:08:56> <Launchmon> (INFO): launch-breakpoint hit event handler invoked.
<Sep 17 17:08:56> <Launchmon> (INFO): launch-breakpoint hit event handler completing with MPIR_DEBUG_SPAWNED
<Sep 17 17:08:56> <Launchmon> (INFO): a proctable message shipped out
<Sep 17 17:08:56> <Launchmon> (INFO): a reshandle message shipped out
<Sep 17 17:08:56> <LMON FE API> (INFO): RPDTAB message received...
<Sep 17 17:08:56> <Launchmon> (INFO): a reshandle message shipped out
<Sep 17 17:08:56> <LMON FE API> (INFO): rid available event received...
<Sep 17 17:08:56> <LMON FE API> (INFO): rminfo event received...
<Sep 17 17:08:56> <LMON FE API> (INFO): launch_bp or first_attach done...

Trace from the hang case:

<Sep 17 17:16:53> <LMON FE API> (INFO): GCRYPT shared_key: 1592601243
<Sep 17 17:16:53> <LMON FE API> (INFO): GCRYPT setkey, 32393531:32313036:3334:0
<Sep 17 17:16:53> <Launchmon> (INFO): linux_launchmon_t initialized.
<Sep 17 17:16:53> <Driver> (INFO): now creating a process object based on pid=35414, exe=/opt/ibm/spectrum_mpi/jsm_pmix/bin/stock/jsrun
<Sep 17 17:16:53> <Symtable> (INFO): reading linkage symbol table for image[=jsrun]
<Sep 17 17:16:53> <Symtable> (INFO): interpreter found in .interp section [/lib64/ld-2.17.so]
<Sep 17 17:16:53> <Symtable> (INFO): reading linkage symbol table for image[=ld-2.17.so]
<Sep 17 17:16:53> <Launchmon> (INFO): The RM process has just been trapped due to attach
<Sep 17 17:16:53> <Launchmon> (INFO):  trap after attach event handler invoked.
<Sep 17 17:16:53> <Symtable> (INFO): reading linkage symbol table for image[=libpthread.so.0]
<Sep 17 17:16:53> <Symtable> (INFO): reading linkage symbol table for image[=libc.so.6]
<Sep 17 17:16:53> <Launchmon> (INFO): trap after attach event handler completed.
<Sep 17 17:16:53> <Launchmon> (INFO): Just continued the RM process out of the first trap

[dongahn]

It seems like the jsrun doesn't call MPIR_Breakpoint every once in a while even after being poked at MPIR_attach_fifo. I think the key is if we can reproduce this with a simple case using gdb.

[dongahn]

@lee218llnl: In the meanwhile, I did make install the version with c723213 into /collab area so that you can test STAT. You may hit the occasional hangs, but I think it shouldn't prevent you from doing some scalability benchmarking.

[lee218llnl]

OK, I confirmed that it occasionally works. Anecdotally, it appears to hang more often than not for me when running STAT.

[lee218llnl]

@dongahn is this ready to be merged? The jsrun fixes appear to resolve the hangs we previously saw.

[dongahn]

@lee218llnl: Well, I don't feel comfortable merging this in yet. Lack of CI support is making it difficult to check easily the sanity of a patch on all of the RMs we need to support. There are tons of things that we need to do but I'm resource bound.