Tutorial: Simulation via SLURM

Tutorial: Simulation via SLURM#

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

This tutorial implements the same workflow introduced in Tutorial 04: Simulation, but executes the simulation with the SLURM workload manager.

References#

Environment#

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

  1. Create the tutorials environment if it doesn’t exist

    $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

  2. Activate the environment

    $ conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Directory Structure#

  1. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite --tutorial 4 && mv tutorial_04_simulation_SConstruct SConstruct
Destination directory: '/home/roppenheimer/waves-tutorials'

  1. Download and copy the tutorial_04_simulation file to a new file named tutorial_sbatch with the WAVES Command-Line Utility fetch subcommand.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite tutorials/tutorial_04_simulation && cp tutorial_04_simulation tutorial_sbatch
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

SConscript#

A diff against the tutorial_04_simulation file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_sbatch

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_sbatch
@@ -56,7 +56,7 @@
 # Comment used in tutorial code snippets: marker-3
 
 # SolverPrep
-copy_source_list = [
+abaqus_source_list = [
     "#/modsim_package/abaqus/rectangle_compression.inp",
     "#/modsim_package/abaqus/assembly.inp",
     "#/modsim_package/abaqus/boundary.inp",
@@ -65,39 +65,19 @@
     "#/modsim_package/abaqus/parts.inp",
     "#/modsim_package/abaqus/history_output.inp",
 ]
-workflow.extend(
-    env.CopySubstfile(
-        copy_source_list,
-    )
-)
+abaqus_source_list = [pathlib.Path(source_file) for source_file in abaqus_source_list]
+workflow.extend(env.CopySubstfile(abaqus_source_list))
 
 # Comment used in tutorial code snippets: marker-4
 
 # Abaqus Solve
-solve_source_list = [
-    "rectangle_compression.inp",
-    "assembly.inp",
-    "boundary.inp",
-    "field_output.inp",
-    "materials.inp",
-    "parts.inp",
-    "history_output.inp",
-    "rectangle_mesh.inp",
-]
-
+solve_source_list = [source_file.name.rstrip(".in") for source_file in abaqus_source_list]
+solve_source_list.append("rectangle_mesh.inp")
+common_suffixes = ("odb", "dat", "msg", "com", "prt")
+datacheck_suffixes = ("023", "mdl", "sim", "stt")
 datacheck.extend(
     env.AbaqusSolver(
-        target=[
-            "rectangle_compression_DATACHECK.odb",
-            "rectangle_compression_DATACHECK.dat",
-            "rectangle_compression_DATACHECK.msg",
-            "rectangle_compression_DATACHECK.com",
-            "rectangle_compression_DATACHECK.prt",
-            "rectangle_compression_DATACHECK.023",
-            "rectangle_compression_DATACHECK.mdl",
-            "rectangle_compression_DATACHECK.sim",
-            "rectangle_compression_DATACHECK.stt",
-        ],
+        target=[f"rectangle_compression_DATACHECK.{suffix}" for suffix in common_suffixes + datacheck_suffixes],
         source=solve_source_list,
         job="rectangle_compression_DATACHECK",
         program_options="-double both -datacheck",
@@ -106,16 +86,10 @@
 
 # Comment used in tutorial code snippets: marker-5
 
+abaqus_implicit_extensions = ("sta",)
 workflow.extend(
     env.AbaqusSolver(
-        target=[
-            "rectangle_compression.odb",
-            "rectangle_compression.dat",
-            "rectangle_compression.msg",
-            "rectangle_compression.com",
-            "rectangle_compression.prt",
-            "rectangle_compression.sta",
-        ],
+        target=[f"rectangle_compression.{suffix}" for suffix in common_suffixes + abaqus_implicit_extensions],
         source=solve_source_list,
         job="rectangle_compression",
         program_options="-double both",

Note that the new AbaqusSolver builder will be conditionally defined in the SConstruct file according to the availability of the sbatch command. If sbatch is not available, the slurm_job variable will go unused by the waves.scons_extensions.abaqus_solver_builder_factory`() builder. Since SCons builders don’t throw errors for unused keyword arguments, we do not need to define the task twice in the SConscript file.

SConstruct#

A diff against the SConstruct file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_sbatch_SConstruct
@@ -69,6 +69,7 @@
 env["ABAQUS_PROGRAM"] = env.AddProgram(
     env["abaqus_commands"] if env["abaqus_commands"] is not None else default_abaqus_commands
 )
+env["SBATCH_PROGRAM"] = env.AddProgram(["sbatch"])
 
 # Comments used in tutorial code snippets: marker-4
 
@@ -89,8 +90,21 @@
 
 # Comments used in tutorial code snippets: marker-5
 
+
 # Add builders and pseudo-builders
-env.Append(BUILDERS={})
+def solve_builder_select():
+    """Run the SLURM sbatch builder if available. Fall back to AbaqusSolver builder in not."""
+    if env["SBATCH_PROGRAM"]:
+        return waves.scons_extensions.sbatch_abaqus_solver_builder_factory(program="${ABAQUS_PROGRAM}")
+    else:
+        return waves.scons_extensions.abaqus_solver_builder_factory(program="${ABAQUS_PROGRAM}")
+
+
+env.Append(
+    BUILDERS={
+        "AbaqusSolver": solve_builder_select(),
+    }
+)
 
 # Comments used in tutorial code snippets: marker-6
 
@@ -100,6 +114,7 @@
     "tutorial_02_partition_mesh",
     "tutorial_03_solverprep",
     "tutorial_04_simulation",
+    "tutorial_sbatch",
 ]
 for workflow in workflow_configurations:
     build_dir = env["variant_dir_base"] / workflow

Build Targets#

  1. Build the new targets

$ pwd
/home/roppenheimer/waves-tutorials
$ scons tutorial_sbatch

Output Files#

Explore the contents of the build directory using the tree command against the build directory, as shown below. Note that the output files from the previous tutorials also exist in the build directory, but the directory is specified by name to reduce clutter in the ouptut shown.

$ pwd
/home/roppenheimer/waves-tutorials
$ tree build/tutorial_sbatch/
Contents