diff --git a/README.md b/README.md index 9442289..4b2bb1b 100644 --- a/README.md +++ b/README.md @@ -63,23 +63,28 @@ update-cesm-machines /path/to/cesm-x.y.z/cime/config/cesm/machines/ /path/to/ces ``` All three machine files (*machines*, *compilers* and *batch*) have to be -updated to get a working installation of CESM/CIME in Hydra or Breniac. +updated to get a working installation of CESM/CIME in the VSC clusters. ### Hydra +* The only module needed is `CESM-deps`. It has to be loaded at all times, from + cloning of the sources to case submission. + * Using `--machine hydra` is optional as long the user is in a compute node or a login node in Hydra. CESM uses `NODENAME_REGEX` in `config_machines.xml` to identify the host machine. -* The only module needed is `CESM-deps`. It has to be loaded at all times, from - cloning of the sources to case submission. +* Users have two options on creation of new cases for `--compiler`. One is + `gnu`, based on the GNU Compiler Collection. The other is `intel`, based on + Intel compilers. The versions of each compiler are described in the + [easyconfigs](#easyconfigs) of each CESM-deps module. -* There is a single configuration for the compiler that is tailored to nodes +* There is a single configuration for both compilers that is tailored to nodes with Skylake CPUs, including the login nodes. * CESM is not capable of detecting and automatically adding the required libraries to its build process. The current specification of `SLIBS` contains - just what we found so far to be required. + just what we found to be required (so far). * By design, CESM sets a specific queue with `-q queue_name`, otherwise it fails to even create the case. In Hydra we use the partition `skylake_mpi` as @@ -108,6 +113,32 @@ updated to get a working installation of CESM/CIME in Hydra or Breniac. will derive the job to `q1h`, `q24h` or `q72h` depending on the walltime requested. +### Hortense + +* The only module needed to create, setup and build cases is `CESM-deps`. + It has to be loaded at all times, from cloning of the sources to case + submission. CESM will also load vsc-mympirun at runtime to be able to + use MPI. + +* Using `--machine hortense` is optional as long the user is in a non-GPU + compute node or a login node in Hortense. CESM uses `NODENAME_REGEX` in + `config_machines.xml` to identify the host machine. + +* Cases are submitted with Slurm's sbatch as CESM is not compatible with + [jobcli](https://github.com/hpcugent/jobcli). + +* By default all cases are run in the `cpu_rome` partition. Optionally, cases + can also be submitted to `cpu_rome_512` with the high memory nodes. + +* Downloading input data from the default FTP servers of UCAR is not possible. + Input data has to be manually downloaded until an alternative method is set + up for Hortense. + +* The recommended workflow is to create the case as usual, setup and build the + case in the compute nodes with the job script + [case.setupbuild.slurm](scripts/case.setupbuild.slurm) and then submit the + case as usual with `case.submit`. + ## File structure All paths defined in `config_machines.xml` are below `$VSC_SCRATCH`. These @@ -218,32 +249,51 @@ each job. This can cause problems in a heterogeneous environment as not all nodes might provide the same hardware features as the login nodes. The example job scripts in [cesm-config/scripts](scripts) solve this problem -by executing all steps in the compute nodes of the cluster. In this way, the +by executing these steps in the compute nodes of the cluster. In this way, the compilation can be optimized to the host machine, simplifying the configuration, and the user does not have to worry about where the case is build and where it is executed. +* [case.slurm](scripts/case.slurm): performs setup, build and execution of the + case + +* [case.setupbuild.slurm](scripts/case.setupbuild.slurm): performs setup and + build of the case, then the user can use `case.submit` as usual + ## Easyconfigs -* CESM-deps loads all dependencies to build and run CESM cases +### CESM-deps + +Loads all dependencies to build and run CESM cases. + +* [CESM-deps-2-intel-2019b.eb](easyconfigs/CESM-deps/CESM-deps-2-intel-2019b.eb): - * `CESM-deps-2-intel-2019b.eb` is used in Breniac + * only option in the two Breniac clusters - * `CESM-deps-2-intel-2019b.eb` is used in Hydra + * used in Hydra with `--compiler=intel` -* CESM-tools loads software commonly used to analyse the results of the - simulations +* [CESM-deps-2-foss-2021b.eb](easyconfigs/CESM-deps/CESM-deps-2-foss-2021b.eb): - * `CESM-tools-2-foss-2019a.eb` is available in Hydra + * only option in Hortense + + * used in Hydra with `--compiler=gnu` Our easyconfigs of CESM-deps are based on those available in [EasyBuild](https://github.com/easybuilders/easybuild-easyconfigs/tree/master/easybuild/easyconfigs/c/CESM-deps). However, the CESM-deps module in Hydra and Breniac also contains the configuration files and scripts from this repository, which are located in the installation directory (`$EBROOTCESMMINDEPS`). Hence, our users have direct -access to these files once `CESM-dep/2-intel-2019b` is loaded. The usage -instructions of our CESM-deps modules also provide a minimum set of -instructions to create cases with this configuration files. +access to these files once `CESM-deps` is loaded. The usage instructions of our +CESM-deps modules also provide a minimum set of instructions to create cases +with this configuration files. + +### CESM-tools + +Loads software commonly used to analyse the results of the simulations. + +* [CESM-tools-2-foss-2019a.eb](easyconfigs/CESM-tools/CESM-tools-2-foss-2019a.eb): + + * available in Hydra ## CPRNC diff --git a/easyconfigs/CESM-deps/CESM-deps-2-foss-2021b.eb b/easyconfigs/CESM-deps/CESM-deps-2-foss-2021b.eb new file mode 100644 index 0000000..df9213a --- /dev/null +++ b/easyconfigs/CESM-deps/CESM-deps-2-foss-2021b.eb @@ -0,0 +1,50 @@ +easyblock = 'Bundle' + +name = 'CESM-deps' +version = '2' + +homepage = 'https://www.cesm.ucar.edu/models/cesm2/' +description = """CESM is a fully-coupled, community, global climate model that +provides state-of-the-art computer simulations of the Earth's past, present, +and future climate states.""" + +toolchain = {'name': 'foss', 'version': '2021b'} + +dependencies = [ + ('CMake', '3.22.1'), + ('Python', '3.9.6'), + ('lxml', '4.6.3'), + ('Perl', '5.34.0'), + ('XML-LibXML', '2.0207'), + ('ESMF', '8.2.0'), + ('netCDF', '4.8.1'), + ('netCDF-Fortran', '4.5.3'), + ('netCDF-C++4', '4.3.1'), + ('PnetCDF', '1.12.3'), + ('Subversion', '1.14.1'), + ('git', '2.33.1', '-nodocs'), + ('git-lfs', '3.2.0', '', True), +] + +components = [ + # install extra configuration tools for CESM + ('cesm-config', '1.6.0', { + 'easyblock': 'Tarball', + 'source_urls': ['https://github.com/vub-hpc/%(name)s/archive'], + 'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': SOURCE_TAR_GZ}], + 'start_dir': '%(name)s-%(version)s', + }), +] + +sanity_check_paths = { + 'files': ['bin/update-cesm-machines', 'scripts/case.pbs', 'scripts/case.slurm'], + 'dirs': ['machines', 'irods'], +} + +usage = """Environment to build and run CESM v2 simulations + 1. Download a release of CESM v2: `git clone -b release-cesm2.2.0 https://github.com/ESCOMP/cesm.git cesm-2.2.0` + 2. Add external programs for CESM: `cd cesm-2.2.0; ./manage_externals/checkout_externals` + 3. Update config files: `update-cesm-machines cime/config/cesm/machines/ $EBROOTCESMMINDEPS/machines/` + 4. Create case: `cd cime/scripts && ./create_newcase --machine ...`""" + +moduleclass = 'geo' diff --git a/machines/config_batch.xml b/machines/config_batch.xml index 930b5f1..af9431a 100644 --- a/machines/config_batch.xml +++ b/machines/config_batch.xml @@ -49,6 +49,20 @@ Author: Alex Domingo (Vrije Universiteit Brussel) + + sbatch + job (\d+) + + + + + + + cpu_rome + cpu_rome_512 + + + -l nodes={{ num_nodes }}:ppn={{ tasks_per_node }} diff --git a/machines/config_compilers.xml b/machines/config_compilers.xml index c7333bd..4c1b8c5 100644 --- a/machines/config_compilers.xml +++ b/machines/config_compilers.xml @@ -37,6 +37,32 @@ Author: Alex Domingo (Vrije Universiteit Brussel) + + $ENV{EBROOTESMF}/lib + $ENV{EBROOTNETCDF} + $ENV{EBROOTNETCDFMINCPLUSPLUS4} + $ENV{EBROOTNETCDFMINFORTRAN} + $ENV{EBROOTPNETCDF} + gpfs + + + mpicc + mpicxx + mpif90 + gcc + g++ + gfortran + + -O2 + + + -O2 -fallow-argument-mismatch -fallow-invalid-boz + + + -L$ENV{EBROOTSCALAPACK}/lib -lscalapack -L$ENV{EBROOTOPENBLAS}/lib -lopenblas + -L$ENV{EBROOTNETCDFMINFORTRAN}/lib -lnetcdff -L$ENV{EBROOTNETCDF}/lib -lnetcdf + + mpiicc mpiicpc @@ -51,14 +77,35 @@ Author: Alex Domingo (Vrije Universiteit Brussel) -xCOMMON-AVX512 -no-fma - -L$ENV{EBROOTNETCDFMINFORTRAN}/lib -lnetcdff -L$ENV{EBROOTNETCDF}/lib64 -lnetcdf + -L$ENV{EBROOTNETCDFMINFORTRAN}/lib -lnetcdff -L$ENV{EBROOTNETCDF}/lib -lnetcdf + + + $ENV{EBROOTESMF}/lib $ENV{EBROOTNETCDF} $ENV{EBROOTNETCDFMINCPLUSPLUS4} $ENV{EBROOTNETCDFMINFORTRAN} $ENV{EBROOTPNETCDF} - gpfs + lustre + + + mpicc + mpicxx + mpif90 + gcc + g++ + gfortran + + -O2 + + + -O2 -fallow-argument-mismatch -fallow-invalid-boz + + + -L$ENV{EBROOTSCALAPACK}/lib -lscalapack -L$ENV{EBROOTOPENBLAS}/lib -lopenblas + -L$ENV{EBROOTNETCDFMINFORTRAN}/lib -lnetcdff -L$ENV{EBROOTNETCDF}/lib -lnetcdf + diff --git a/machines/config_machines.xml b/machines/config_machines.xml index f927307..e1d09c4 100644 --- a/machines/config_machines.xml +++ b/machines/config_machines.xml @@ -38,11 +38,12 @@ Author: Alex Domingo (Vrije Universiteit Brussel) - Hydra - heterogeneous cluster at VUB, batch system is PBS + Hydra - heterogeneous cluster at VUB, batch system is Slurm (node3.*\.hydra\.(os|brussel\.vsc)|login.*\.cerberus\.os) LINUX - intel - impi + intel,gnu + impi + openmpi $ENV{VSC_SCRATCH}/cesm/output $ENV{VSC_SCRATCH}/cesm/inputdata $ENV{VSC_SCRATCH}/cesm/inputdata/atm/datm7 @@ -55,11 +56,10 @@ Author: Alex Domingo (Vrije Universiteit Brussel) 40 40 FALSE - + srun --ntasks={{ total_tasks }} - --cpu_bind=sockets --cpu_bind=verbose --kill-on-bad-exit @@ -74,14 +74,56 @@ Author: Alex Domingo (Vrije Universiteit Brussel) module -q - XML-LibXML/2.0201-GCCcore-8.3.0 - Python/2.7.16-GCCcore-8.3.0 - CMake/3.15.3-GCCcore-8.3.0 - git/2.23.0-GCCcore-8.3.0-nodocs CESM-deps/2-intel-2019b + + CESM-deps/2-foss-2021b + + + + -1 + + + + + Hortense - Tier-1 cluster at UGent, batch system is Slurm + (node|login)\d+\.dodrio\.os + LINUX + gnu + openmpi + $ENV{VSC_SCRATCH}/cesm/output + $ENV{VSC_SCRATCH}/cesm/inputdata + $ENV{VSC_SCRATCH}/cesm/inputdata/atm/datm7 + $CIME_OUTPUT_ROOT/archive/$CASE + $ENV{VSC_SCRATCH}/cesm/cesm_baselines + $ENV{VSC_SCRATCH}/cesm/tools/cprnc/cprnc + 16 + slurm + HPC-Ugent + 128 + 128 + TRUE + + mympirun + + + /usr/share/lmod/lmod/init/perl + /usr/share/lmod/lmod/init/env_modules_python.py + /usr/share/lmod/lmod/init/csh + /usr/share/lmod/lmod/init/sh + /usr/share/lmod/lmod/libexec/lmod perl -q + /usr/share/lmod/lmod/libexec/lmod python -q + module -q + module -q + + + + + CESM-deps/2-foss-2021b + vsc-mympirun + -1 diff --git a/scripts/case.setupbuild.slurm b/scripts/case.setupbuild.slurm new file mode 100644 index 0000000..a9d0f00 --- /dev/null +++ b/scripts/case.setupbuild.slurm @@ -0,0 +1,17 @@ +#!/bin/bash +#SBATCH --job-name=CESM-case-setupbuild +#SBATCH --output="%x-%j.out" +#SBATCH --time=4:00:00 +#SBATCH --nodes 1 +#SBATCH --ntasks 16 + +module load CESM-deps/2-foss-2021b + +# CESM case setup +echo +./case.setup --reset +./preview_run + +# CESM case build +echo +./case.build