Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up kokkos/tril situation #6473

Closed
wants to merge 1 commit into from
Closed

Clean up kokkos/tril situation #6473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions cime_config/machines/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,6 @@
<env name="BLA_VENDOR">Generic</env>
<env name="Albany_ROOT">$SHELL{if [ -z "$Albany_ROOT" ]; then echo /global/common/software/e3sm/mali_tpls/albany-e3sm-serial-release-gcc-cmake-fix; else echo "$Albany_ROOT"; fi}</env>
<env name="Trilinos_ROOT">$SHELL{if [ -z "$Trilinos_ROOT" ]; then echo /global/common/software/e3sm/mali_tpls/trilinos-e3sm-serial-release-gcc; else echo "$Trilinos_ROOT"; fi}</env>
<env name="Kokkos_ROOT">$SHELL{if [ -z "$Kokkos_ROOT" ]; then echo /global/common/software/e3sm/mali_tpls/trilinos-e3sm-serial-release-gcc; else echo "$Kokkos_ROOT"; fi}</env>
</environment_variables>
<environment_variables compiler="nvidia" mpilib="mpich">
<env name="ADIOS2_ROOT">$SHELL{if [ -z "$ADIOS2_ROOT" ]; then echo /global/cfs/cdirs/e3sm/3rdparty/adios2/2.9.1/cray-mpich-8.1.25/nvidia-22.7; else echo "$ADIOS2_ROOT"; fi}</env>
Expand Down Expand Up @@ -2656,7 +2655,6 @@
<environment_variables compiler="gnu">
<env name="Albany_ROOT">$SHELL{if [ -z "$Albany_ROOT" ]; then echo /lcrc/group/e3sm/ac.jwatkins/LandIce/AlbanyBuilds/build-gcc-sfad12-e3sm/install; else echo "$Albany_ROOT"; fi}</env>
<env name="Trilinos_ROOT">$SHELL{if [ -z "$Trilinos_ROOT" ]; then echo /lcrc/group/e3sm/ac.jwatkins/LandIce/TrilinosBuilds/build-gcc-e3sm/install; else echo "$Trilinos_ROOT"; fi}</env>
<env name="Kokkos_ROOT">$SHELL{if [ -z "$Kokkos_ROOT" ]; then echo /lcrc/group/e3sm/ac.jwatkins/LandIce/TrilinosBuilds/build-gcc-e3sm/install; else echo "$Kokkos_ROOT"; fi}</env>
</environment_variables>
</machine>

Expand Down
10 changes: 7 additions & 3 deletions components/cmake/find_dep_packages.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,14 @@
# errors.
if (USE_KOKKOS)

# Kokkos will be built in the sharedlibs if Kokkos_ROOT is
# unset.
if (NOT DEFINED ENV{Kokkos_ROOT})
set(ENV{Kokkos_ROOT} ${INSTALL_SHAREDPATH})
# We should use the kokkos from Trilinos if we are using Trilinos
if (USE_TRILINOS)
set (ENV{Kokkos_ROOT} ${Trilinos_ROOT})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One potential issue with this is that when we switch to the preinstalled kokkos, we'll need a new path. But maybe I can change this then by creating something like Trilinos_Kokkos_ROOT.

Copy link
Contributor

@bartgol bartgol Jun 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at first Kokkos_ROOT is only used to decide whether CIME needs to build kokkos during sharedlib phase. Then, in the CMake logic of the modelbuild phase, we could look for Kokkos only after handling Albany/Trilinos (if needed), so that we may skip find_package(Kokkos) altogether. Something like

if (USE_ALBANY)
  find_package(Albany REQUIRED)  # this also finds trilinos and kokkos, as they are deps
endif()
if (USE_TRILINOS)
  find_package(Trilinos REQUIRED)  # this also finds kokkos, as it is a dep
endif()
if (USE_KOKKOS)
  find_package(Kokkos REQUIRED)
endif()

In this version, if USE_ALBANY=TRUE, the calls to find_pacakge for trilinos and kokkos return immediately, since the package was alrady found.

Right now, instead, we first look for kokkos, then for trilinos, and finally for albany. We should do it the other way around, since cmake pulls in deps anyways... If we do that, then Kokkos_ROOT becomes unused, since Kokkos is already found when find_package(Kokkos REQUIRED) is parsed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bartgol , those latter two blocks should be elseifs , right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't matter. The calls to find_package return immediately if the pkg was already found anyways...

else()
# Kokkos will be built in the sharedlibs if Kokkos_ROOT is unset.
set(ENV{Kokkos_ROOT} ${INSTALL_SHAREDPATH})
endif()
endif()

find_package(Kokkos REQUIRED)
Expand Down
8 changes: 7 additions & 1 deletion share/build/buildlib.kokkos
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ OR
###############################################################################
def buildlib(bldroot, installpath, case):
###############################################################################
# If we are using Trilinos, we must use the Kokkos that comes with it
if case.get_value("USE_TRILINOS"):
print ("You case is using Trilinos and will use its Kokkos")
return

installed_kokkos_dir = os.environ.get("Kokkos_ROOT")
if installed_kokkos_dir is not None:
# We are trying to use a pre-installed kokkos. Look for the relevant folders/libs,
Expand All @@ -65,9 +70,10 @@ def buildlib(bldroot, installpath, case):
# often arch dependent (e.g., $prefix/lib or $prefix/lib64)... The best thing
# would be to run a small cmake script, that calls find_package. But E3SM's
# cmake build system will do that soon enough, so any error will be caught there.
print (f"Using pre-installed Kokkos_ROOT: {kokkos_root}")
return
else:
print ("no value foudn in env for Kokkos_ROOT. building from scratch")
print ("no value found in env for Kokkos_ROOT. building from scratch")

srcroot = case.get_value("SRCROOT")
ekat_dir = os.path.join(srcroot, "externals", "ekat")
Expand Down