Skip to content

Lesson 1: Getting Started with HPC

This module is designed to make sure all new members of the COMPASS PSAAP project have reasonable sense of familiarity with the basic tools and tasks encountered on HPC systems. This particular module will be geared towards LLNL machines, but analogous variants of the tasks can be completed on any HPC cluster.

If you have any trouble completing any of the tasks below, these other Bootcamp articles may be of help:
1. Computing At LLNL
2. GitHub Overview

Step 0: Verify access to LLNL systems

Before proceeding with the rest of the module, it's a good idea to test your RSA token to make sure it's working and you understand how to generate the One-Time Password. To do that, attempt to log in to this LLNL token test page (and bookmark it in case you have access issues in the future!)

Important

Most web-based LLNL pages that can be accessed externally use your "LLNL Official User Name (OUN)", while accessing the clusters uses your "LC Username". For most people, these should be the same, but double check your initial paperwork just in case they are not the same.

Once you have succeeded, proceed to the next the step. If you are unable to get test it to work, try the "Token Diagnostics" on that page, and if that fails, call the number on the help page to work with LLNL support to fix your token.

Tip

LLNL regularly does maintenance on the clusters, so be sure to double check your emails for any "LC Notices" if you ever find you can't login to a specific machine. Trying a different machine (Dane, Tuolumne, Tioga, or the VNC server) or even the OTP Token Test page are a good way of troubleshooting if it is the network or your token.

Step 1: Access HPC Machines (Using SSH)

The first step to accessing an HPC system is figuring out which application you want to use to connect to the cluster. The simplest is using some sort of SSH client. Mac OS and most Linux distributions ship with an ssh client you can use in the terminal. Windows 10/11 machines usually have ssh capabilities from both Powershell and the Command Prompt, though older versions may not have them automatically enabled. Windows machines also have the the ability to use WSL2 with a Linux distribution that has an SSH client. However, WSL2 can be involved to get set up correctly, so unless you need a local linux environment, it may be better to stick to the Command Prompt or a third-party application.

There are also third-party applications, such as PuTTY, MobaXterm, and VSCode, to name just a few. The RealVNC Viewer can also be used to get access to a LLNL Linux machine with ssh support. This module can be carried out completely from the Command Line, but feel free to take a different tool out for a test run using the tasks in this module.

Once you have an ssh client ready, try to carry out the following tasks:

  1. SSH to an LLNL machine. Remember that the address for LLNL machines should be <LC username>@<server>.llnl.gov
  2. Once on an LLNL machine, attempt to ssh to a different LLNL machine. As a bonus challenge, try to figure out which parts of the address are needed to ssh to a different LLNL machine.
  3. Check your storage quota (Hint: Check the LLNL PSAAP Quickstart Guide)
  4. Access the PSAAP/UNM shared workspace. In order to have access to this, you will need to be manually added. If you have not asked for access, do that first, and while waiting, access your "workspace" location.

Step 2: Moving Files

Another key skill needed is moving files to and from HPC clusters. While most files are "moved" via your work with Git-based projects, there are still times when one-off files and artifacts may need to be moved to/from your local machine.

Note

While you can carry out the following tasks going between LLNL machines (or the VNC server and a cluster), since they have a shared filesystem, we encourage you to use a machine that is not on the LLNL network.

Additionally, these tasks are designed around using the "scp" command. If you are using a tool that visualizes the remote filesystem, consider repeating the steps with the command line after you figure it out with your tool of choice.

Try out the following tasks:

  1. Move a file from your local machine to an LLNL machine
  2. Move a file from an LLNL machine to your local machine

Tip

You can still do this command from your local machine instead of trying to figure out what address to use from the LLNL machine.

  1. On an LLNL machine, create a folder with multiple files, and copy them all over to your computer.
  2. If you did the last step by compressing the folder, try it again without compressing the folder and with sticking to a single SCP command. If you did move all of the files in one go, try compressing the folder, moving it, and uncompressing it on your local machine.

Now that you can move your own files, try to move files to other users on the LC systems by using the give and take commands:

Tip

Try running man give or man take in the command line if you are unfamiliar with these tools!

You can also give/take files to/from yourself if you do not have another person to test on.

  1. Give a file to another user on an LLNL system
  2. Take a file from another user on an LLNL system.

Step 3: Flux & Slurm

Now that you have access to a cluster, another key component is to being able to submit jobs to the scheduling system on the cluster. On LLNL's Dane, that would be with Slurm; On Tuolumne that would be with Flux.

If this is your first time using either of the tools, here are a few links that can provide you with some examples to get started with:
1. LLNL Building a Job Script (Slurm)
2. LLNL Flux Job Script Example

Tip

Both LLNL systems have custom wrappers that can support Slurm scripts on Flux and Flux scripts on Slurm. For this module, it is advised that you try to get a basic version of the native script version before using the wrappers. That way, if there ends up being an incompatibility between the wrappers and the native tool, you are aware of the general features of the tool.

  1. Create an interactive allocation and from that terminal, print out which host you are on using the hostname command.
  2. With that allocation still active, open another terminal and figure out which nodes your job is running on.

Tip

The command that shows all of your jobs will provide this information (as well as other helpful information)

  1. With the same allocation still open, the second terminal session to SSH into one of the node(s) you have access to for the job.
  2. Close the interactive session from the terminal that created it. You should notice that your second session was also removed.
  3. Create a simple job script that can carry out some sort of command across all nodes & cores you ask for.
  4. Experiment with various flags to add to your script, such as a time limit, the debug queue, and changing the output file.
  5. Do some simple experiments to answer these question: "Do HPC jobs carry over the loaded modules?" and "Do HPC jobs carry over the environment variables set when the job was launched?"
  6. Bonus: If feeling adventurous, attempt to repeat all of the steps on both Dane and Tuolumne/Tioga, and in both interactive and script modes.

Step 4: Other Helpful Commands

This step focus on offering some simple commands that can help make various tasks less tedious on HPC clusters, and only scratch the scratch the surface of what you can do from the command line.

Searching Previous Commands

  1. Using the history command, figure out what your last ten (and only those 10) commands where.
  2. Using both history and grep, do a search for something.
  3. Hit the ctrl + r keys and do the same search you just did.
  4. Figure out how many big your history can be on the cluster of your choice.
  5. Use the ! version of "history" to run the last ls command you ran
  6. Use the "up" arrow key to select a previous command, and then edit it to do something different
  7. Use ctrl + r to find a previous command, and then edit it to do something different

Using Modules

  1. Find the documentation for the module command to use for the rest of these tasks. Your options include: man module (the man pages), module help (a custom command provided by the tool), or other official documentation websites (either from LLNL or the link provided by man module).
  2. Figure out the module subcommand that shows what modules you currently have loaded.
  3. Figure out the module subcommand that shows you all versions of a tool/library that can currently be loaded.
  4. Figure out the module subcommand that shows you all versions of a tool/library that are installed on the system.
  5. Figure out the module subcommand that shows what modules are needed to load a module that shows up by using the command from Task 3, but not the command in Task 2.
  6. Use the which command to figure out where a specific compiler of your choice is.

Tip

The which command is handy for making sure that a specific tool is on your PATH and can be found, and can be a great tool to help troubleshoot job scripts that are having trouble finding certain tools.

Step 5: Using Git

Regardless of what project you end up on, you will find yourself working with some sort of Git repository. This section focuses on making sure you can do more than just clone a repo on an HPC machine as a well as carrying out a few GitHub features.

  1. Before hoping on your machine, make sure you can access the COMPASS Bootcamp GitHub repository. Once you have access to it, make a fork of it.
  2. On an HPC machine, clone your fork of this repository.
  3. Once you have the fork cloned, make a branch and then add a tip or trick to the "Community Notes" at bottom of this page. Your tip/tick does not have to be super complex; Your contribution can be anything you've learned, any application you've found handy, even an acronym that you didn't know, etc. If you can't think of anything, look up an common terminal command and find a new flag to add a note about.
  4. Commit and push your changes to your branch.
  5. On the original COMPASS Bootcamp repository, create a pull request that will merge your fork's new branch into the main branch of COMPASS Bootcamp repository.
  6. Wait for a review on the Pull Request and resolve any comments made by a reviewer.

Note

While this project has the ability to build the documentation locally, do not do this on the head node of a cluster. If you really need to preview your Markdown changes to the Community Notes section, use a different viewer (such as the GitHub render of it).

Tip

We recommend also checking out the COMPASS GitHub guidelines, which has information on how we generally like to setup COMPASS related repositories.

Community Notes

Below are all of the tips and tricks that others have added. Once you've finished looking over them, consider attempting Lesson 2: Compiling on LLNL Systems!