Skip to content

Installing AMG2023 with Spack

AMG2023 is a parallel algebraic multigrid solver for linear systems arising from problems on unstructured grids. The source code and documentation can be found here. AMG2023 can be installed with Spack, but it requires a specific configuration to be built successfully.

Step 1: Setting up the Prerequisites

  1. Ensure you have access to Tuolumne and/or Dane
  2. Follow the instructions under the "Installing and Configuring Spack" section of the Bootcamp Intro to Spack article to set up Spack.
  3. Follow the instructions under the "Additional COMPASS-Specific Instructions" section of the Bootcamp Intro to Spack article to add external system packages and COMPASS-specific packages (including the AMG2023 package with a CMake fix missing from Spack's original AMG2023 package).

Important

Be sure to run the setup script with the "-E" flag (and then re-login) so that the system specific spack package files are set in your environment before Spack attempts to build AMG.

Step 2: Create the Spack environment for AMG2023

Create a new spack environment using one of the following methods:

  • Create an environment saved inside the Spack repository
  • Run the command spack env create <your_environment_name>, and then activate it using spack env activate <environment_name>.
  • Open the environment configuration file (called spack.yaml) using spack config edit, and add the specs in one of the code blocks below, depending on your system, for AMG2023 and hypre.
  • Create an environment saved in a directory of your choice
  • Create a directory where you want your Spack environment to be saved, and create a file called spack.yaml in that directory.
  • Copy the contents of one of the code blocks below, depending on your system, into your spack.yaml file.
  • Activate the environment using spack env activate /path/to/directory/with/spack.yaml. 

# AMG2023 Spack environment for Tuolumne
spack:
  packages:
    c:
      require: [cce@20.0.0]
    cxx:
      require: [cce@20.0.0]
    fortran:
      require: [cce@20.0.0]
  specs:
  - matrix:
    - ["^cray-mpich@9.0.1"]
    - ["ldflags=-L/opt/cray/pe/mpich/9.0.1/gtl/lib ldlibs=-lmpi_gtl_hsa"]
    - ["hypre@2.33.0+caliper+rocm+gpu-aware-mpi+shared", "amg2023+rocm+caliper", "caliper cppflags=-pthread"]


# AMG2023 Spack environment for Dane
spack:
  packages:
    c:
      require: [gcc@13.3.1]
    cxx:
      require: [gcc@13.3.1]
    fortran:
      require: [gcc@13.3.1]
  specs:
  - matrix:
    - ["^mvapich2@2.3.7"]
    - ["hypre@2.33.0+shared", "amg2023+caliper", "caliper cppflags=-pthread"]

Step 3: Concretize the Spack environment for AMG2023

With your environment activated, run spack concretize. This command tells package to build the dependency chain needed to install AMG2023. Packages prefixed by [e] are external packages Spack found in packages.yaml from step 1. Packages prefixed with [-] need to be installed, and packages prefixed with [+] are already installed.

Important

If you are on Tuolumne, hypre must be linked with libmpi_cray.so.12 and libmpi_gtl_hsa.so.0 to run with GPU-aware MPI. If not, you will get the runtime error GPU_SUPPORT_ENABLED is requested, but GTL library is not linked. To ensure these libraries are being linked, check that the following exists in your concretization output for hypre and AMG2023:

amg2023@develop ldflags=-L/opt/cray/pe/mpich/9.0.1/gtl/lib ldlibs=-lmpi_gtl_hsa ...
hypre@2.33.0 ldflags=-L/opt/cray/pe/mpich/9.0.1/gtl/lib ldlibs=-lmpi_gtl_hsa ...

Step 4: Install AMG2023 and all dependencies

  1. Run spack install to install all packages prefixed by a [-] in the concretization output.
  2. After the install is complete, add the amg binary to your path with the command spack load amg2023. Now the benchmark can be executed by using the command amg.

Important

If you are on Tuolumne, check again that the libmpi libraries are being linked. Run the following command and check that the libraries are found:

bash$ ldd $(which amg) | grep "mpi"
libmpi_cray.so.12 => /opt/cray/pe/mpich/9.0.1/ofi/crayclang/20.0/lib/libmpi_cray.so.12 (0x00001554ebb47000)
libmpi_gtl_hsa.so.0 => /opt/cray/pe/mpich/9.0.1/ofi/crayclang/20.0/lib/libmpi_gtl_hsa.so.0 (0x00001554eb681000)

Step 5: Run AMG2023

Below are sample run scripts for Tuolumne (Flux) and Dane (Slurm). Copy the script you need, complete the paths to your Spack installation and AMG2023 Spack environment, then run the script using flux batch <script> (Flux) or sbatch <script> (Slurm).

#!/bin/bash

# AMG2023 Flux job script

#flux: --nodes=2
#flux: --nslots=8
#flux: --time-limit=5m
#flux: --queue=pdebug
#flux: --exclusive
#flux: --env=FLUX_TO_BASH={{nnodes}}

# Variable Setup
NODES=$FLUX_TO_BASH
PPN=4

#Stop if any command fails
set -e

# Setup Spack
PATH_TO_SPACK= #/path/to/spack
source $PATH_TO_SPACK/share/spack/setup-env.sh

# Setup Spack Environment
ENV_FOR_SPACK_DIR= #/path/to/amg2023/spack_env
spack env activate -d $ENV_FOR_SPACK_DIR

# Load the Application
spack load amg2023

# Turn on GPU-AWARE Cray MPICH
export MPICH_GPU_SUPPORT_ENABLED=1

# Run the program
flux run --nodes=$NODES --tasks-per-node=$PPN amg -problem 2 -n 128 128 128 -P 2 2 2 -memory_device -exec_device

#!/bin/bash

# AMG2023 Slurm job script

#SBATCH --job-name=amg2023
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --time=0:05:00
#SBATCH --partition=pdebug
#SBATCH --exclusive
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

# Stop if any command fails
set -e

# Variable setup (matches your Slurm intent)
NODES=${SLURM_JOB_NUM_NODES}
PPN=4

# Setup Spack
PATH_TO_SPACK= #/path/to/spack
source ${PATH_TO_SPACK}/share/spack/setup-env.sh

# Setup Spack Environment
ENV_FOR_SPACK_DIR= #/path/to/amg2023/spack_env
spack env activate -d $ENV_FOR_SPACK_DIR

# Load the Application
spack load amg2023

# Run the program (Slurm equivalent of `flux run ...`)
srun --nodes=$NODES --ntasks-per-node=$PPN amg -problem 2 -n 128 128 128 -P 2 2 2

Simple Run Parameters

Single Node, 4 GPUs

  • Try each problem
  • Matrix sizes to test:
    -n 128 128 128
-n 256 256 256
-n 512 512 512

Scaling Study

Important bottlenecks: at the strong scaling limit, e.g. the maximum number of processes that you can run on before run times start to increase. This is important, because this is where you can solve the problem fastest. If you can improve any bottleneck at the strong scaling limit, you can solve the problem faster.

  • To test the strong scaling limit, increase process counts while total problem size stays constant. By default, AMG 2023 does weak scaling (-n specifies per-process size).
  • Calculate the total problem size on one node (e.g. 512x512x512xPPN) and choose -n values that will keep this constant. Formula: cube_root(nx * ny * nz * PPN / process_count)
  • Continue increasing process count, keeping global matrix size constant, until total time increases (where total time is returned by hypre as setup time + solve time).
  • Note: you may reach a point where further increasing process count results in increases in solve phase time , but setup phase time decreases. If the sum of the two decreases, continue scaling.