Profiling with Kokkos Tools

The Kokkos tools repository provides profiling tools that link with Kokkos. While not as robust as tools like Caliper, profiling with Kokkos tools takes little effort to get set up and running. It is good for "quick-and-dirty" profiling of codes that use Kokkos. Here is an example output:

 (Type)   Total Time, Call Count, Avg. Time per Call, %Total Time in Kernels, %Total Program Time
-------------------------------------------------------------------------

Regions: 

- Canopy::Solver::multipole_to_local
 (REGION)   1.029476 2 0.514738 117.446669 56.635965
- Canopy::SolverLayer::multipole_to_local
 (REGION)   0.669102 8 0.083638 76.333814 36.810232
- Canopy::Solver::create_multipoles
 (REGION)   0.632773 2 0.316386 72.189227 34.811601
-------------------------------------------------------------------------

Kernels: 

- fill_locals_cells
 (ParFor)   0.310426 6 0.051738 35.414620 17.077889
- Canopy::SolverLayer::multipole_to_local team
 (ParFor)   0.301326 8 0.037666 34.376461 16.577261
- set_multipoles
 (ParFor)   0.158637 8 0.019830 18.097912 8.727303
- Kokkos::ViewFill-1D
 (ParFor)   0.018871 155 0.000122 2.152913 1.038193
- Kokkos::ZeroMemset via parallel_for
 (ParFor)   0.017777 1217 0.000015 2.028039 0.977975

-------------------------------------------------------------------------
Summary:

Total Execution Time (incl. Kokkos + non-Kokkos):                   1.81771 seconds
Total Time in Kokkos kernels:                                       0.87655 seconds
   -> Time outside Kokkos kernels:                                  0.94116 seconds
   -> Percentage in Kokkos kernels:                                   48.22 %
Total Calls to Kokkos Kernels:                                         4050

-------------------------------------------------------------------------

Regions are user-defined sections of the code to profile with Kokkos::Profiling calls. Usage examples:

Kokkos::Profiling::pushRegion("Canopy::Solver::multipole_to_local");
// Region code here
Kokkos::popRegion()

// Or define a region that persist as long as the "region" object is in scope
Kokkos::Profiling::ScopedRegion region("Canopy::Solver::multipole_to_local");

Kernels are parallel loops, such as Kokkos::parallel_for or Kokkos::parallel_reduce. For example, a parallel for loop from the example output is declared as follows:

Kokkos::parallel_for("fill_locals_cells",
    Kokkos::RangePolicy<execution_space>(0, view.size()),
    KOKKOS_LAMBDA(const int i)
    { 
        // Do work
    });

This page explains how to use the SimpleKernelTimer to profile time spent in Kokkos scoped or parallel regions. There are usage instructions in the Kokkos tools repository, but the instructions have parts that are out of date.

Kokkos Tools Installation Instructions (Tested on Dane and Tuolumne)

Clone the Kokkos tools repository. No need to use Spack!
If on an HPC system, no modules other than the default need to be loaded. Run the following commands:
```
mkdir build
cd build
cmake ..
make
```

Profile with SimpleKernelTimer

Run the following command, setting the path to where you cloned your kokkos-tools repository and the name of your build directory:
```
export KOKKOS_TOOLS_LIBS=/path/to/kokkos-tools/build/profiling/simple-kernel-timer/libkp_kernel_timer.so
```
Run your application as you normally would. If running via a job script, add the export command before the command which launches your application.
After your application run completes, navigate to the directory is was launched in.
You will notice a collection of *.dat files, one per process spawned. These are binary files. You must use the kp_reader program in kokkos tools to convert it to human-readable format.

Run the following command to create a text file from the data in a *.dat file:

/path/to/kokkos-tools/build/profiling/simple-kernel-timer/kp_reader <filename>.dat &> <filename>.txt

Now you can view the results of the profiling in the .txt file. An example output is shown at the beginning of this guide.