Skip to content

Flux Tips and Tricks

The goal of this article is not to cover the basics of how to use the Flux job launcher, as LLNL already offers a tutorial for learning the basics with Flux on their website. Instead, this article will focus on various tips and tricks learned for more advanced features that are less obvious, or are deviations from Slurm.

The documentation of all Flux commands can be a bit tricky to get, as there are many parts to Flux that can clog up the search. The top level site we recommend for digging into the basic Flux commands can be found here. For specific details on running jobs, click the Section 1 link, and then navigate to the specific page you want to read more about.

Influencing the Job Output File

Like Slurm, Flux offers the ability to change the output of a job through a flag (in this case, with --output). However, the default behavior for Flux is to overwrite this file. That means if the same output file is used for multiple jobs, or even multiple steps within the same job, the file will be overwritten each time and you'll be left with the output from the last step. While this behavior is also possible in Slurm, it is not the default behavior there, so scripts porting to Flux will need to be aware of this change.

To get around this, there are two straight-forward options:

  1. Update the script to use a unique output per job/run, set by the script itself through its own variables or Flux's "Mustache Templates"
  2. Use the --setopt=output.mode=append command. This switches from the default mode of truncate(overwrite) to appending to the file.

Adjusting the Final Timeout

When debugging with Flux on LLNL systems, you may encounter a situation where there is a large time gap between ranks exiting on Flux. This could be because some ranks are stuck, some ranks are writing a lot of output, or just generally lagging processes. By default, flux will kill all ranks after 30 seconds have passed since the first process exited (not Rank 0, but the first process to return). While this is handy to kill a job early if it fails, this can be annoying in other times when it kills other ranks still creating output. Luckily, this is also an option you can influence when running a job. If doing a batch job (e.g., you run flux batch my_script.sh that then contains a flux run inside it), you can use the --setopt=exit-timeout option on any job you start inside your allocation. From testing, it seems that setting this option at the top of your script (e.g., #flux: --setopt=exit-timeout=) does not seem to set a global standard for all runs in the script. Instead, you need to put it on any flux run commands you want to control the exit-timeout for. The value for acceptable timeouts uses the flux time format, so values like 30s or 2m are usable here. To remove the timeout entirely, use --setopt=exit-timeout=none. When using no timeout, be sure to set an overall job time limit in case your jobs is truly stuck.

Chaining Jobs in Flux

When running experiments, you may find yourself needing to carry out parameter/value sweep that requires you to make independent scripts and job allocations. While the exact means to create this setup is not covered here, there is a Flux flag that allows many jobs to be chained together without requesting the entire cluster all at once for your sweep. If you've ever run a Flux job on an LLNL system, you'll notice that the output of flux batch is the job id. If you capture this in a script, you can then turn around and pass it to Flux's --dependency flag to allow to be blocked by another job. Here is an example of chaining two flux batch submissions together:

# Assuming the .sh files are in the the same directory
# Save the output of the first job ID
prev_job_id=flux batch my_first_script.sh
# Tell flux to launch this job after the previous 
flux batch --dependency=afterany:$prev_job_id my_second_script.sh

In addition to the job ID to wait on, this flag also requires you to set the rules for which state the previous job needs to accomplish before this new job can be considered eligible to run. In the example above, afterany means our second job can be run only when the first job has completed (either successfully or unsuccessfully). For a full list of available dependency schemes, see this section on the Flux documentation.