Installing E3SM with MALI

These build instructions are for compiling Albany Land Ice (ALI) as a preinstalled library for E3SM+MALI. These instructions assume you have already ported E3SM to your machine (you have it defined in config_machines.xml and can build and run cases.) Currently supported machines are:

Perlmutter at NERSC
Chrysalis at ANL

Clone repos and load environment

First, create a dedicated directory and download the appropriate versions of Kokkos, Trilinos and Albany:

export E3SM_MALI=${HOME}/e3sm_mali
mkdir ${HOME}/e3sm_mali
cd ${E3SM_MALI}
git clone https://github.com/E3SM-Project/kokkos -b e3sm-kokkos-4.2.00
git clone https://github.com/E3SM-Project/Trilinos -b e3sm-trilinos-15.1.1
git clone https://github.com/sandialabs/Albany.git -b e3sm-albany-2024-03-26

Then, ensure your shell environment matches what is used for E3SM by using CIME. Here, we assume E3SM is located in the same directory:

python ${E3SM_MALI}/E3SM/cime/CIME/Tools/get_case_env >& env.sh
source env.sh

Alternatively, you can use the file generated in any case directory:

source .env_mach_specific.sh

Example: Albany/pmcpu_gnu_modules.sh at e3sm-albany-2024-03-26 · sandialabs/Albany

Building Kokkos

Normally, Kokkos is installed while building E3SM using CIME case.build. Here, we need to install Kokkos as a tpl outside case.build in order for E3SM, Trilinos and Albany to use the same Kokkos. Kokkos options for the internal build are specified in E3SM/cime_config/machines/cmake_macros at master · E3SM-Project/E3SM and can be used to ensure E3SM is configured properly.

Make a build directory for Kokkos and copy over an example configuration file:

mkdir ${E3SM_MALI}/kokkos/build-kokkos
cd ${E3SM_MALI}/kokkos/build-kokkos
cp ${E3SM_MALI}/kokkos/do-cmake-kokkos-pmcpu-gnu .

Modify the following if needed:

SOURCE_DIR=${E3SM_MALI}/kokkos
INSTALL_DIR=${E3SM_MALI}/kokkos/build-kokkos/install
...
-D CMAKE_CXX_COMPILER=${MPICXX} \ # Path to mpi compiler
...
-D Kokkos_ARCH_ZEN2:BOOL=ON \ # Depends on your machine architecture. Remove this line if unknown

Then, configure, build and install:

source do-cmake-kokkos-pmcpu-gnu
make -j4  # use more if available
make install

Building Boost for Trilinos

Trilinos requires many tpls that are also needed in E3SM. The one exception is boost. A script is provided as an example for installing boost.

mkdir ${E3SM_MALI}/boost
cd ${E3SM_MALI}/boost
cp ${E3SM_MALI}/Trilinos/run-boost-1.83.0-pmcpu-gnu.sh .

Modify the following:

BOOST_PATH=${E3SM_MALI}/boost/install

Then run:

./run-boost-1.83.0-pmcpu-gnu.sh

Building Trilinos

Make a build directory for Trilinos and copy over an example configuration file:

mkdir ${E3SM_MALI}/Trilinos/build-trilinos
cd ${E3SM_MALI}/Trilinos/build-trilinos
cp ${E3SM_MALI}/Trilinos/do-cmake-trilinos-pmcpu-gnu .

Modify the following if needed:

SOURCE_DIR=${E3SM_MALI}/Trilinos
INSTALL_DIR=${E3SM_MALI}/Trilinos/build-trilinos/install
BOOST_DIR=${E3SM_MALI}/boost/install
KOKKOS_DIR=${E3SM_MALI}/kokkos/build-kokkos/install
...
  -D CMAKE_C_COMPILER=cc \ # path to mpi compilers
  -D CMAKE_CXX_COMPILER=CC \
  -D CMAKE_Fortran_COMPILER=ftn \
...
  -D TPL_BLAS_LIBRARIES:STRING="${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.so" \ # path to blas/lapack
  -D TPL_LAPACK_LIBRARIES:STRING="${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.so" \
...
  -D Netcdf_INCLUDE_DIRS:PATH=${NETCDF_DIR}/include \ # path to netcdf
  -D Netcdf_LIBRARY_DIRS:PATH=${NETCDF_DIR}/lib \
...
  -D Pnetcdf_INCLUDE_DIRS:PATH=${PNETCDF_DIR}/include \ # path to parallel-netcdf
  -D Pnetcdf_LIBRARY_DIRS:PATH=${PNETCDF_DIR}/lib \
...
  -D HDF5_INCLUDE_DIRS:PATH=${HDF5_DIR}/include \ # path to hdf5
  -D HDF5_LIBRARY_DIRS:PATH=${HDF5_DIR}/lib\

Then, configure, build and install:

source do-cmake-trilinos-pmcpu-gnu
make -j4  # use more if available
make install

Building Albany

Make a build directory for Albany and copy over an example configuration file:

mkdir ${E3SM_MALI}/Albany/build-albany
cd ${E3SM_MALI}/Albany/build-albany
cp ${E3SM_MALI}/Albany/do-cmake-albany-pmcpu-gnu .

Modify the following if needed:

SOURCE_DIR=${E3SM_MALI}/Albany
INSTALL_DIR=${E3SM_MALI}/Albany/build-albany/install
TRILINOS_DIR=${E3SM_MALI}/Trilinos/build-trilinos/install
...
 -D CMAKE_CXX_COMPILER=CC \ # path to mpi compiler

Then, configure, build and install:

source do-cmake-albany-pmcpu-gnu
make -j4  # use more if available
make install

Build/Run E3SM+MALI

Modify ${E3SM_MALI}/E3SM/cime_config/machines/config_machines.xml to point to your Albany/Trilinos installations:

<env name="Albany_ROOT">$SHELL{if [ -z "$Albany_ROOT" ]; then echo $E3SM_MALI/Albany/build-albany/install; else echo "$Albany_ROOT"; fi}</env>
<env name="Trilinos_ROOT">$SHELL{if [ -z "$Trilinos_ROOT" ]; then echo $E3SM_MALI/Trilinos/build-trilinos/install; else echo "$Trilinos_ROOT"; fi}</env>

Note: $E3SM_MALI should be replaced with the full path to ensure it finds the installation with E3SM_MALI defined. Alternatively, the Albany and Trilinos environment variables can be specified explicitly before the E3SM build phase:

export Albany_ROOT=${E3SM_MALI}/Albany/build-albany/install
export Trilinos_ROOT=${E3SM_MALI}/Trilinos/build-trilinos/install

Next, build/run a simple test to ensure everything is properly configured:

${E3SM_MALI}/E3SM/cime/scripts/create_test SMS.ne30pg2_r05_IcoswISC30E3r5_gis20.IGELM_MLI.mali-gis20km

Support

For support, contact the team via the E3SM slack in the #land-ice channel or post an issue here: https://github.com/sandialabs/Albany/issues