Skip to content

Benchpark

Benchpark is a application framework to generate and organize benchmark testing on different systems. It is capable of leveraging spack and ramble to generate and run experiments. The application is well documented on the official website. and has a online YouTube tutorial.

Goals of Module

This document will walk through a few of the more confusing stages as well as some of the options not fully explored in the documentation. By following this tutorial, you should have basic knowledge on how to activate Benchpark, create a workspace, and run an experiment.

The latter half walks through how to modify the system and configuration files to create new variants of experiments.

Notes on building using the RealVCN client.

If using the RealVCN client to connect to LLNL systems, the terminal inside the desktop does not have access to a portion of the cray-mpich library. You will need to ssh into one of the clusters.

Installation

Make sure you have the prerequisites:

  • Python 3.8
  • pip

Clone the repo and build dependencies:

git clone https://github.com/LLNL/benchpark.git
cd benchpark
. setup-env.sh
python3 -m venv my-env
. my-env/bin/activate
pip install -r requirements.txt

Tip

Depending on your system pip may not install everything. If this occurs you may have to install the problem packages manually.

Setup-env.sh adds the core Benchpark functions to your path. Most Benchpark functions follow the schema benchpark \<subsystem> \<command> \<arg>

python3 -m venv my-env creates a python virtual environment to contain the required python libraries. If you already have a virtual environment, you may activate that environment instead.

By default, these changes are temporarily and the setup script and python environment must be activated each time you open a new shell. You can automate this process by modifying and sourcing ~/.bashrc or similar file depending on the system configuration.

Important folders and files in top level of Benchpark Repo

  • setup-env.sh
  • my-env (your python environment)
  • repo (local copy of spack)
  • systems (current build definitions for testing systems)
  • experiments (current definitions of existing experiments)
  • modifiers (additional supported options you can add to an experiment)
  • lib (core Benchpark library)

Making an experiment

Setting up and running an experiment takes part in four stages:

  • Configure settings for the target system
  • Configure the experiment to run on that system
  • Create/select an experiment root directory
  • Create the ramble workspace.

Initialize System Configuration

benchpark system init --dest=<mySystemName> <SystemName> <Option=value>
  • mySystemName - Name of the folder to be created containing the generated system configuration files. The folder must not already exist.
  • SystemName - the system on which the test is to be run. The system must have a file inside the systems folder.
  • Options - Some systems have additional configurations for particular clusters. In which case it is noted in the system.py. The additional configurations can be set by name=value.
  • Example: benchpark system init --dest=test_system llnl-elcapitan cluster=tuolumne compiler=cce

The above command creates a test_system directory and populates it with the configuration files needed to run on Tuolumne while using the cce compiler.

Experiment Initialization Setup

benchpark experiment init <mySystemName> <Benchmark> +/~<Boolean Variant> <String Variant>=<value>
  • mySystemName - folder containing generated system configuration files. Generated by previous step.
  • benchmark - base experiment program, must be inside experiments folder.
  • Boolean_variants - boolean variants can be added to the experiment can be activated or deactivated by using '+' or '~' respectively.
  • String_variants - modifiers and variables used by the experiment. Set by name=value

Example: benchpark experiment init test_system amg2023+weak caliper=mpi

The above example creates a folder within the test_system directory containing a yaml for running a weak-scaling study of amg2023 with caliper recording mpi calls.

After running experiment init, Benchpark will print additional instructions for how to setup the workspace. These instructions are covered in more detail in the following sections.

The system and experiment directories from this and the previous step are linked to during the next steps, deleting them will cause the Benchpark to break when you attempt to run the experiment.

If you are running multiple variants of the same benchmark for the same system. You will need to change the name of the folder generated afterwards. Benchpark will NOT give you the option to overwrite a system or experiment configuration yaml.

Available Experiments and Modifiers:

A list of experiments and modifiers can be found by the following commands or by looking in the appropriate folders in the top level of the repo. 

benchpark -list experiments
benchpark -list modifiers

Experiment Workspace setup

Benchpark uses ramble to create and run the execution environment for experiments. The core of this is the experiment_root folder. The experiment root folder contains an isolated copy of spack and ramble, which are used by the ramble workspaces. 

benchpark setup mySystemName/Benchmark <experiments_root>
. <experiments_root>/setup.sh

Benchpark setup creates the root directory for your tests (unlike other cases, Benchpark will use an existing experiment_root directory and create sub-directory inside as long as the particular combination of system and benchmark does not have a existing sub-directory.)

The experiment root directory contains a copy of spack and ramble for the experiments inside that root experiments. Both Spack and Ramble can be initialized by running the setup.sh inside the experiment_root. 

ramble --workspace-dir <experiments_root>/<mySystemName>/<Benchmark>/workspace/ setup

This sets up and configures the workspace for the experiment.

Running the Experiment

Once your workspace is setup, you can run all experiments in the workspace experiment by running the all_experiments scripts in the workspace or by using the ramble on command. 

ramble --workspace-dir <experiments_root>/<test_system>/<benchmark>/workspace on

Individual tests can be run by navigating to the workspace and going into the experiments folder, inside is a 'execute_experiment' script that will run that experiment.

If you have a spack problem, ramble uses the spack environment within workspace/config, so be sure to activate the proper environment that before debugging.

The results of each experiment are contained with experiments. The flux.out for each job is created in the directory where the batch command was launched.

Rerunning an experiment will overwrite previous runs of that same experiment. If you want to preserve your previous run data, rename the files or copy them elsewhere.

Modifying Experiments

Replicating an Benchpark experiment requires one of two things, either the system.py,the experiment.py and the list of arguments used on the command line, or the system and experiment configurations.

System.py

The system.py controls build-time environmental variables as well as system dependent options including mount points and globally installed software. Supported variant vary slightly by system, but typically include the following options.

As a base in Benchpark v0.1 the following variants are available on El-Capitan systems:

  • compiler
  • gtl
  • rocm
  • qpumode
  • queue
  • mount_point
  • blas
  • bank

Experiment.py

Adding Variant Flags

Variants behave similar to spack in that they act as flags for triggering behavior when the experiment.py is read.

Variants in Benchpark experiment.py appear in the following format:

variant(
   "variant_name",
   default="default_value",
   values=("value1", "value2"),
   description="description",
)

If a variable is a boolean, it can be triggered by +/~ otherwise the variable_name must be typed with a valid value on the command line.

Inside the experiment.py, the value of a variant is checked using self.spec.satisfies(variant), with variant_value being the value as if assigned on the command line.

Example: self.spec.satisfies("workload=problem1")

Adding Environment variables

You can add environmental variables by using the set_environmental_variable command inside the experiment.pyThe variable will be added and exported by the generated flux/slurm script.
Example: self.set_environment_variable("HSA_XNACK", 1) will add export HSA_XNACK=1 to your flux/slurm script.

Modifying spack

To modify the spack environment, create a new experiment_root folder and run the setup.sh inside it. Add the appropriate repos containing the packages you want to change with spack repo add.

If you modify an existing experiment_root the change will propagate into the contained experiments if they are rebuilt.