Unable to determine path to METPLUS_BUILD_BASE for tutorial setup script

[jwraby]

I had been working on executing the tutorial setup on the HPC (bash) and I think I understand the concept of providing the paths to the METplus and MET builds in the tutorial setup script pretty well. I am running into issues on the HPC with trying to determine those paths in the METplus container. The METplus container image file is located at /p/home/jraby/MET/METplus2. I tried shelling-in to the container to locate the metplus.py script and found it at /metplus/METplus/ush. If I put that entire path in the set-up script for the METPLUS_BUILD_BASE it still can’t find the metplus.py script using “which”. Do you have any suggestions on how this can be done? The same potential issue exists for the MET_BUILD_BASE. Thanks.

[CPKalb]

Hi:

METPLUS_BUILD_BASE shouldn't include the /ush. If you omit the /ush, is it able to find the run_metplus.py script or is it still not able to locate it?

Christina

[jwraby]

Here is what I have so far. Could not find the equivalent command in Singularity to mount a data container as you did using Docker. I did find that I can "bind" a directory present in my home on the HPC to /mnt in the Singularity METplus container. I downloaded the sample_data-met_tool_wrapper-5.0.tgz from a link you provided in one of our previous discussions: (https://dtcenter.ucar.edu/dfiles/code/METplus/METplus_Data/v5.0/)
I unpacked it which created a new directory called met_tool_wrapper. I then used the singularity exec command with the option --bind and it successfully brought the data into the container. I specified the entire path to the met_tool_wrapper directory which was not collocated with the METplus container and it appears to have brought in my entire home directory with it! I guess I should have downloaded the tarball file to the same directory where the METplus container was located. Anyway, I ran the "which" commands and for point_stat it worked as expected. For the run_metplus.py script, it could not find it. I found it at /metplus/METplus/ush in the container. See the following dialog:
Singularity> which run_metplus.py
which: no run_metplus.py in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

I found the met_tool_wrapper tutorial data directory inside the container at the same exact location as it is in my home. The following is a listing of the contents:
atcf_data data extract_tiles gfdl new out scripts series_lead_filtered tc_data tc_pairs tc_stat
Will the above tutorial data work for the METplus training series?

I tested setting the environment variable you showed for MET_BUILD_BASE in your reply above and it was successfully set inside the container. The path for METPLUS_DATA is rather long, but should work.

I'm not sure why the the "which" command is not working for the run_metplus.py script.

[hankenstein2]

Hi John, at the risk of too many cooks in the kitchen, would you be willing to share what your PATH is now?
The tutorial setup script we use has

export METPLUS_BUILD_BASE=/opt/METplus
export PATH=${METPLUS_BUILD_BASE}/ush

I definitely think you want METPLUS_BUILD_BASE equal to your METplus directory and then
export PATH=${METPLUS_BUILD_BASE}/ush should set you up to run run_metplus.py without a path.

However, what you have done above also "should" work and I am not entirely sure what is going on and maybe we need to see your PATH variable and something weird might pop out.

[jwraby]

Hi Hank - On restarting the METplus container, found that the METPLUS_BUILD_BASE was not assigned the path I determined and set yesterday. I am able to find the METplus directory which contains the run_metplus.py script at /metplus/METplus/ush. The contents of this directory are:
master_metplus.py run_metplus.py ush.dox validate_config.py

Positioned at / in the container, I just tried setting the path in the METPLUS_BUILD_BASE variable to /metplus/METplus using the export command:
Singularity> export METPLUS_BUILD_BASE=/metplus/METplus
Singularity> echo $METPLUS_BUILD_BASE
/metplus/METplus
Singularity> pwd
/
After that I ran the following:
Singularity> export PATH=${METPLUS_BUILD_BASE}/ush
Singularity> echo $METPLUS_BUILD_BASE
/metplus/METplus
Singularity> echo $PATH
/metplus/METplus/ush
Singularity> which run_metplus.py
bash: which: command not found

This behavior is odd and was not what I saw yesterday, I believe it responds "command not found" when I execute the command to set PATH. To test this, I stopped the container and restarted it to zeroize the variables then ran the following:
Singularity> export METPLUS_BUILD_BASE=/metplus/METplus
Singularity> which run_metplus.py
which: no run_metplus.py in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Singularity> export PATH=${METPLUS_BUILD_BASE}/ush
Singularity> which run_metplus.py
bash: which: command not found

I'm not sure what is going on at this point.

Singularity> which run_metplus.py
which: no run_metplus.py in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

[hankenstein2]

I know we left this in a bit of an uncertain state, have you made any progress or need more assistance?

[jwraby]

Hank - Thanks for the follow-up. The situation remains the same when trying to setup for the tutorial using the METplus container. The following behavior pretty much sums up the situation which is hard to understand:
Singularity> export METPLUS_BUILD_BASE=/metplus/METplus
Singularity> which run_metplus.py
which: no run_metplus.py in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Singularity> export PATH=${METPLUS_BUILD_BASE}/ush
Singularity> which run_metplus.py
bash: which: command not found

I'm not sure what to do at this point. The Army controls the networking to the extent that I can't share my desktop outside of the DoD. Otherwise, having the ability to demo/test online to you would make debugging this easier. I can do any testing you recommend and will be glad to report back the results. I could also call you if that would be useful.

[hankenstein2]

For your second try can you do this...

export PATH=$PATH:${METPLUS_BUILD_BASE}/ush

I think the second error is that your environment is not actually finding the "which" command. I believe you reassigned your PATH to just the METPLUS_BUILD_BASE.

[jwraby]

I tried export PATH=$PATH:${METPLUS_BUILD_BASE}/ush and it worked!
Singularity> export PATH=$PATH:${METPLUS_BUILD_BASE}/ush
Singularity> which run_metplus.py
/metplus/METplus/ush/run_metplus.py
Thanks for diagnosing that error in the export command.

[hankenstein2]

Good to hear John! Keep the questions coming.