Installing Commander on OWL

[lukashergt]

Hi all,

This is intended to help newbies install Commander on the OWL cluster for the Synchrotron paper project.

Disclaimer

Note that this is targeted towards people that are new to Commander! If you already have your own Commander repository with a bunch of branches and changes, then this is not for you, because my number one recommendation for new people having trouble installing Commander is to START FROM SCRATCH! In other words, start from the beginning of this guide, don't try to skip any steps, don't use a previously used Commander repository. In fact it might be best if you removed any previously downloaded Commander repository, such that you won't get confused later on as to which one is the working repository.

Prerequisite

As a prerequisite I'm assuming that you know how to successfully log in to the OWL cluster. This setup is geared towards the nodes OWL 31 to 35. If you want to use a different node, you'll have to modify some commands accordingly. If you don't know how to do that, just use on of OWL 31 to 35.

Installing Commander

1. On the OWL cluster, go to your home directory:

   cd ~

   For me this is /mn/stornext/u3/lukashe.

2. Load the following modules:

   module load gnu git cmake intel/oneapi mpi/latest icc/latest compiler-rt/latest mkl/latest

   You will have to load these every time you log in to the OWL cluster. If you don't want to have to do that every time, then you should add this module load line to your .bashrc/.bash_login.
   (Personal recommendation: use the vim editor command. However, if you have never used vim before, make sure to run the vimtutor command first!)

3. Next, lets set up some folders to keep our stuff contained and easy to remove:

   mkdir -p GitProjects/CommanderPrj
   cd GitProjects/CommanderPrj/

4. Here, we make a fresh clone of the Commander git repository:

   git clone https://github.com/Cosmoglobe/Commander.git
   cd Commander

5. Check out the AST9240_2022 branch:

   git checkout AST9240_2022

6. Run the install script (which should hopefully take care of everything):

   ./install_ita.sh

7. Add the following line to your .bashrc/.bash_login file:

   export HEALPIX=$HOME/GitProjects/CommanderPrj/Commander/build_owl3135_oneapi/install/healpix
   export COMMANDER_PARAMS_DEFAULT="$HOME/GitProjects/CommanderPrj/Commander/commander3/parameter_files/defaults/"

This should hopefully have done the trick and installed Commander. Next up, running Commander.

Running Commander

Go to the folder from the 2022 Component Separation course:

cd /mn/stornext/d16/cmbco/AST9240/2022/

Here you should hopefully have your own directory. Go in there:

cd <your_own_directory>

Copy Lukas' base parameter file and label it with the dataset that you intend to add, e.g. replace <label> with Parkes if you intend to run with the Parkes maps (or rename it entirely if you intend to run something completely different...):

cp /mn/stornext/d16/cmbco/AST9240/2022/lthergt/commander_runs/param_AST9240_base_v1.0.0.txt ./param_AST9240_base_v1.0.0_plus_<label>.txt

Modify the parameter file as desired, make sure to also change the OUTPUT_DIRECTORY to something meaningful.

If you like you might find the following run script helpful. Copy this into a file called run_commander.sh:

#!/usr/bin/bash
source ~/.bashrc
WORK_DIR=`pwd`
PARAM_FILE_PATH=$1
DESCRIPTION=$2
PARAM_FILE_OUTPUT_INFO="`grep $PARAM_FILE_PATH -e OUTPUT_DIRECTORY`"
SPAM=(${PARAM_FILE_OUTPUT_INFO// / })
CHAINS_DIR=${SPAM[-1]}
SPAM=(${PARAM_FILE_PATH//// })
PARAM_FILE=${SPAM[-1]}
SPAM=(${PARAM_FILE/./ })
PARAM_FILE_NAME=${SPAM[0]}
export COMMANDER_PARAMS_DEFAULT=$HOME/GitProjects/CommanderPrj/Commander/commander3/parameter_files/defaults/
export OMP_NUM_THREADS=1

echo
echo "Start Commander run $PARAM_FILE_NAME"
echo "    launch directory: $WORK_DIR"
echo "    chains directory: $CHAINS_DIR"
echo "    input param file: $PARAM_FILE_PATH"

printf '%s %s %s %s : %-40s --- %-50s' `date +"%a %Y-%m-%d %T (%Z)"` "$PARAM_FILE_NAME" "$DESCRIPTION" >> commander_runs.log

mkdir $CHAINS_DIR
cp $PARAM_FILE_PATH ${CHAINS_DIR}/${PARAM_FILE_NAME}_input.txt

mpiexec -env I_MPI_FABRICS shm -n 64 $HOME/GitProjects/CommanderPrj/Commander/build_owl3135_oneapi/install/bin/commander3 $PARAM_FILE_PATH 2>&1 | tee $CHAINS_DIR/slurm.txt

printf ' --- finished: %s %s\n' `date +"%Y-%m-%d %T"` >> commander_runs.log

With that you can launch a run providing two command line arguments, the first being the parameter file to use, and the second being a descriptive string enclosed by " ", e.g. like this:

bash run_commander.sh param_AST9240_base_v1.0.0_plus_<label>.txt "short description of how you are running base plus something"

Remember, once commander is running, stay away from CTRL-C, else you might kill the node...

The run script will automatically create a directory named after OUTPUT_DIRECTORY in the parameter file and launch commander in there. It will also write to a log file called commander_runs.log, which will be helpful to keep track of the various runs that you have/will launch. All the command line output will be stored in a file called slurm.txt.