Tutorial: Task Definition Reuse

Tutorial: Task Definition Reuse#

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

This tutorial demonstrates one solution to re-using task definitions in multiple workflows. While the solution can be implemented with per-task granularity, the task definitions in this tutorial are split into blocks to suite the parameterization workflows of Tutorial 07: Cartesian Product and Tutorial: Mesh Convergence.

The presented solution comes with significant restrictions to directory organization and challenges to target node identification. First, due to the way SCons constructs variant (build) directory directed graphs, the re-usable portions of the workflow must be co-located with the calling file or found in a subdirectory. The choice of organization has implications for the build directory organization options, too. For example, the two possible directory organization may look like one of

project directory/
|-- SConscruct
|-- workflow1
|-- workflow2
|-- common
`-- build directory/
    |-- workflow1 directory/
    |   |-- workflow1 output
    |   |-- common1 output
    `-- workflow2 directory*
        |-- workflow2 output
        `-- common2 output

project directory/
|-- SConscruct
|-- workflow1
|-- workflow2
|-- common directory/
|   `-- common
`-- build directory/
    |-- workflow1 directory/
    |   |-- workflow1 output
    |   `-- common directory/
    |       `-- common1 output
    `-- workflow2 directory/
        |-- workflow2 output
        `-- common directory/
            `-- common2 output

Second, if a target from the parent file is used as a source in the common file, SCons will recognize the build path correctly when provided with the target file base name, but it may not associate the source and target nodes correctly. This results in an incorrectly assembled directed graph and race conditions in the task execution. The target/source association may be made explicit with the SCons FindFile function. While this works, it is not as straightforward as simply specifying the file basename, as implemented in the core tutorials. This second restriction does not affect the current tutorial, but will affect workflows similar to the Tutorial: Mesh Convergence.

References#

Environment#

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

  1. Create the tutorials environment if it doesn’t exist

    $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

  2. Activate the environment

    $ conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Directory Structure#

  1. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite --tutorial 12 && mv tutorial_12_archival_SConstruct SConstruct
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

  1. Download and copy the tutorial_12_archival file to a new file named tutorial_task_reuse with the WAVES Command-Line Utility fetch subcommand.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite tutorials/tutorial_12_archival && cp tutorial_12_archival tutorial_task_reuse
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

SConscript#

A diff against the tutorial_12_archival file from Tutorial 12: Data Archival is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_task_reuse

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_12_archival
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_task_reuse
@@ -27,8 +27,10 @@
 # Simulation variables
 build_directory = pathlib.Path(Dir(".").abspath)
 workflow_name = build_directory.name
-workflow_configuration = [env["project_configuration"], workflow_name]
 parameter_study_file = build_directory / "parameter_study.h5"
+geometry_configuration = "rectangle_geometry_partition.scons"
+simulation_configuration = "rectangle_mesh_solverprep_solve_extract.scons"
+workflow_configuration = [env["project_configuration"], workflow_name, geometry_configuration, simulation_configuration]
 
 # Collect the target nodes to build a concise alias for all targets
 workflow = []
@@ -53,115 +55,33 @@
 
     # Comment used in tutorial code snippets: marker-4
 
-    # Geometry
-    workflow.extend(
-        env.AbaqusJournal(
-            target=[set_name / "rectangle_geometry.cae", set_name / "rectangle_geometry.jnl"],
-            source=["#/modsim_package/abaqus/rectangle_geometry.py"],
-            subcommand_options="--width ${width} --height ${height}",
-            **simulation_variables,
-        )
+    # Geometry, Partition
+    workflow, datacheck = env.SConscript(
+        geometry_configuration,
+        variant_dir=set_name.name,
+        exports={
+            "env": env,
+            "simulation_variables": simulation_variables,
+            "workflow": workflow,
+            "datacheck": datacheck,
+        },
+        duplicate=False,
     )
 
-    # Partition
-    workflow.extend(
-        env.AbaqusJournal(
-            target=[set_name / "rectangle_partition.cae", set_name / "rectangle_partition.jnl"],
-            source=["#/modsim_package/abaqus/rectangle_partition.py", set_name / "rectangle_geometry.cae"],
-            subcommand_options="--width ${width} --height ${height}",
-            **simulation_variables,
-        )
+    # Mesh, SolverPrep, Abaqus Solve, Extract Abaqus
+    workflow, datacheck = env.SConscript(
+        simulation_configuration,
+        variant_dir=set_name.name,
+        exports={
+            "env": env,
+            "simulation_variables": simulation_variables,
+            "workflow": workflow,
+            "datacheck": datacheck,
+        },
+        duplicate=False,
     )
 
-    # Mesh
-    workflow.extend(
-        env.AbaqusJournal(
-            target=[
-                set_name / "rectangle_mesh.inp",
-                set_name / "rectangle_mesh.cae",
-                set_name / "rectangle_mesh.jnl",
-            ],
-            source=["#/modsim_package/abaqus/rectangle_mesh.py", set_name / "rectangle_partition.cae"],
-            subcommand_options="--global-seed ${global_seed}",
-            **simulation_variables,
-        )
-    )
-
-    # SolverPrep
-    copy_source_list = [
-        "#/modsim_package/abaqus/rectangle_compression.inp.in",
-        "#/modsim_package/abaqus/assembly.inp",
-        "#/modsim_package/abaqus/boundary.inp",
-        "#/modsim_package/abaqus/field_output.inp",
-        "#/modsim_package/abaqus/materials.inp",
-        "#/modsim_package/abaqus/parts.inp",
-        "#/modsim_package/abaqus/history_output.inp",
-    ]
-    workflow.extend(
-        env.CopySubstfile(
-            copy_source_list,
-            substitution_dictionary=env.SubstitutionSyntax(simulation_variables),
-            build_subdirectory=set_name,
-        )
-    )
-
-    # Comment used in tutorial code snippets: marker-5
-
-    # Abaqus Solve
-    solve_source_list = [
-        set_name / "rectangle_compression.inp",
-        set_name / "assembly.inp",
-        set_name / "boundary.inp",
-        set_name / "field_output.inp",
-        set_name / "materials.inp",
-        set_name / "parts.inp",
-        set_name / "history_output.inp",
-        set_name / "rectangle_mesh.inp",
-    ]
-
-    datacheck.extend(
-        env.AbaqusSolver(
-            target=[
-                set_name / "rectangle_compression_DATACHECK.odb",
-                set_name / "rectangle_compression_DATACHECK.dat",
-                set_name / "rectangle_compression_DATACHECK.msg",
-                set_name / "rectangle_compression_DATACHECK.com",
-                set_name / "rectangle_compression_DATACHECK.prt",
-                set_name / "rectangle_compression_DATACHECK.023",
-                set_name / "rectangle_compression_DATACHECK.mdl",
-                set_name / "rectangle_compression_DATACHECK.sim",
-                set_name / "rectangle_compression_DATACHECK.stt",
-            ],
-            source=solve_source_list,
-            job="rectangle_compression_DATACHECK",
-            program_options="-double both -datacheck",
-        )
-    )
-
-    workflow.extend(
-        env.AbaqusSolver(
-            target=[
-                set_name / "rectangle_compression.odb",
-                set_name / "rectangle_compression.dat",
-                set_name / "rectangle_compression.msg",
-                set_name / "rectangle_compression.com",
-                set_name / "rectangle_compression.prt",
-                set_name / "rectangle_compression.sta",
-            ],
-            source=solve_source_list,
-            job="rectangle_compression",
-            program_options="-double both",
-        )
-    )
-
-    # Extract Abaqus
-    extract_source_list = [set_name / "rectangle_compression.odb"]
-    workflow.extend(
-        env.AbaqusExtract(
-            target=[set_name / "rectangle_compression.h5"],
-            source=extract_source_list,
-        )
-    )
+# Comment used in tutorial code snippets: marker-5
 
 # Comment used in tutorial code snippets: marker-6

  1. Create a new file named rectangle_geometry_partition.scons from the contents below

waves-tutorials/rectangle_geometry_partition.scons

 1#! /usr/bin/env python
 2"""Rectangle model's Geometry and Partition task definitions
 3
 4Requires the following ``SConscript(..., exports={})``
 5
 6* ``env`` - The SCons construction environment with the following required keys
 7
 8
 9* ``simulation_variables`` - The dictionary of simulation variables with the following required keys
10
11  * ``'width'`` - The rectangle model width
12  * ``'height'`` - The rectangle model height
13
14* ``workflow`` - list of targets in the main workflow
15* ``datacheck`` - list of targets in the datacheck workflow
16
17Returns the following variables
18
19* ``workflow`` - updated list of targets in the main workflow
20* ``datacheck`` - updated list of targets in the datacheck workflow
21
22Example usage:
23
24.. code-block::
25
26   import pathlib
27
28   env = Environment()
29   project_dir = pathlib.Path(Dir(".").abspath)
30
31   workflow = []
32   datacheck = []
33
34   nominal_workflow, nominal_datacheck = SConscript(
35       "rectangle_geometry_partition.scons",
36       exports={
37           "env": env,
38           "simulation_variables": simulation_variables,
39           "workflow": workflow,
40           "datacheck": datacheck
41       },
42       duplicate=False)
43
44   for set_name in [f"parmameter_set{number}" for number in range(4)]
45       parameters_workflow, parameter_datacheck = SConscript(
46           "rectangle_geometry_partition.scons",
47           variant_dir=set_name,
48           exports={
49               "env": env,
50               "simulation_variables": simulation_variables,
51               "workflow": workflow,
52               "datacheck": datacheck
53           },
54           duplicate=False)
55"""
56
57import pathlib
58
59# Inherit the parent construction environment
60Import(["env", "simulation_variables", "workflow", "datacheck"])
61
62# Simulation variables
63build_directory = pathlib.Path(Dir(".").abspath)
64
65# Geometry
66workflow.extend(
67    env.AbaqusJournal(
68        target=["rectangle_geometry.cae", "rectangle_geometry.jnl"],
69        source=["#/modsim_package/abaqus/rectangle_geometry.py"],
70        subcommand_options="--width ${width} --height ${height}",
71        **simulation_variables,
72    )
73)
74
75# Partition
76workflow.extend(
77    env.AbaqusJournal(
78        target=["rectangle_partition.cae", "rectangle_partition.jnl"],
79        source=["#/modsim_package/abaqus/rectangle_partition.py", "rectangle_geometry.cae"],
80        subcommand_options="--width ${width} --height ${height}",
81        **simulation_variables,
82    )
83)
84
85Return(["workflow", "datacheck"])

  1. Create a new file named rectangle_mesh_solverprep_solve_extract.scons from the contents below

waves-tutorials/rectangle_mesh_solverprep_solve_extract.scons

  1#! /usr/bin/env python
  2"""Rectangle model's Geometry and Partition task definitions
  3
  4Requires the following ``SConscript(..., exports={})``
  5
  6* ``env`` - The SCons construction environment with the following required keys
  7
  8
  9* ``simulation_variables`` - The dictionary of simulation variables with the following required keys
 10
 11  * ``'global_seed'`` - The rectangle model global seed size
 12  * ``'displacement'`` - The rectangle model uniaxial displacement
 13
 14* ``workflow`` - list of targets in the main workflow
 15* ``datacheck`` - list of targets in the datacheck workflow
 16
 17Returns the following variables
 18
 19* ``workflow`` - updated list of targets in the main workflow
 20* ``datacheck`` - updated list of targets in the datacheck workflow
 21
 22Example usage:
 23
 24.. code-block::
 25
 26   import pathlib
 27
 28   env = waves.scons_extensions.WAVESEnvironment()
 29   project_dir = pathlib.Path(Dir(".").abspath)
 30
 31   workflow = []
 32   datacheck = []
 33
 34   nominal_workflow, nominal_datacheck = SConscript(
 35       "rectangle_mesh_solverpep_solve_extract.scons",
 36       exports={
 37           "env": env,
 38           "simulation_variables": simulation_variables,
 39           "workflow": workflow,
 40           "datacheck": datacheck
 41       },
 42       duplicate=False)
 43
 44   for set_name in [f"parmameter_set{number}" for number in range(4)]
 45       parameters_workflow, parameter_datacheck = SConscript(
 46           "rectangle_mesh_solverpep_solve_extract.scons",
 47           variant_dir=set_name,
 48           exports=["env", "simulation_variables", "workflow", "datacheck")
 49           duplicate=False)
 50"""
 51import pathlib
 52
 53
 54# Inherit the parent construction environment
 55Import(["env", "simulation_variables", "workflow", "datacheck"])
 56
 57# Simulation variables
 58build_directory = pathlib.Path(Dir(".").abspath)
 59
 60# Mesh
 61workflow.extend(
 62    env.AbaqusJournal(
 63        target=["rectangle_mesh.inp", "rectangle_mesh.cae", "rectangle_mesh.jnl"],
 64        source=["#/modsim_package/abaqus/rectangle_mesh.py", "rectangle_partition.cae"],
 65        subcommand_options="--global-seed ${global_seed}",
 66        **simulation_variables,
 67    )
 68)
 69
 70# SolverPrep
 71copy_source_list = [
 72    "#/modsim_package/abaqus/rectangle_compression.inp.in",
 73    "#/modsim_package/abaqus/assembly.inp",
 74    "#/modsim_package/abaqus/boundary.inp",
 75    "#/modsim_package/abaqus/field_output.inp",
 76    "#/modsim_package/abaqus/materials.inp",
 77    "#/modsim_package/abaqus/parts.inp",
 78    "#/modsim_package/abaqus/history_output.inp",
 79]
 80workflow.extend(
 81    env.CopySubstfile(
 82        copy_source_list,
 83        substitution_dictionary=env.SubstitutionSyntax(simulation_variables),
 84    )
 85)
 86
 87# Abaqus Solve
 88solve_source_list = [
 89    "rectangle_compression.inp",
 90    "assembly.inp",
 91    "boundary.inp",
 92    "field_output.inp",
 93    "materials.inp",
 94    "parts.inp",
 95    "history_output.inp",
 96    "rectangle_mesh.inp",
 97]
 98
 99datacheck.extend(
100    env.AbaqusSolver(
101        target=[
102            "rectangle_compression_DATACHECK.odb",
103            "rectangle_compression_DATACHECK.dat",
104            "rectangle_compression_DATACHECK.msg",
105            "rectangle_compression_DATACHECK.com",
106            "rectangle_compression_DATACHECK.prt",
107            "rectangle_compression_DATACHECK.023",
108            "rectangle_compression_DATACHECK.mdl",
109            "rectangle_compression_DATACHECK.sim",
110            "rectangle_compression_DATACHECK.stt",
111        ],
112        source=solve_source_list,
113        job="rectangle_compression_DATACHECK",
114        program_options="-double both -datacheck",
115    )
116)
117
118workflow.extend(
119    env.AbaqusSolver(
120        target=[
121            "rectangle_compression.odb",
122            "rectangle_compression.dat",
123            "rectangle_compression.msg",
124            "rectangle_compression.com",
125            "rectangle_compression.prt",
126            "rectangle_compression.sta",
127        ],
128        source=solve_source_list,
129        job="rectangle_compression",
130        program_options="-double both",
131    )
132)
133
134# Extract Abaqus
135extract_source_list = ["rectangle_compression.odb"]
136workflow.extend(
137    env.AbaqusExtract(
138        target=["rectangle_compression.h5"],
139        source=extract_source_list,
140    )
141)
142
143Return(["workflow", "datacheck"])

SConstruct#

A diff against the SConstruct file from Tutorial 12: Data Archival is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_12_archival_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_task_reuse_SConstruct
@@ -61,6 +61,7 @@
     print_build_failures=GetOption("print_build_failures"),
     abaqus_commands=GetOption("abaqus_command"),
     TARFLAGS="-c -j",
+    TARSUFFIX=".tar.bz2",
 )
 
 # Conditionally print failed task *.stdout files
@@ -121,6 +122,7 @@
     "tutorial_09_post_processing",
     "tutorial_11_regression_testing",
     "tutorial_12_archival",
+    "tutorial_task_reuse",
 ]
 for workflow in workflow_configurations:
     build_dir = env["variant_dir_base"] / workflow

Build Targets#

  1. Build the new targets

$ pwd
/home/roppenheimer/waves-tutorials
$ scons tutorial_task_reuse --jobs=4

Output Files#

$ pwd
/home/roppenheimer/waves-tutorials
$ tree build/tutorial_task_reuse/parameter_set0/
build/tutorial_task_reuse/parameter_set0/
|-- abaqus.rpy
|-- abaqus.rpy.1
|-- abaqus.rpy.2
|-- assembly.inp
|-- boundary.inp
|-- field_output.inp
|-- history_output.inp
|-- materials.inp
|-- parts.inp
|-- rectangle_compression.inp
|-- rectangle_compression.inp.in
|-- rectangle_compression_DATACHECK.023
|-- rectangle_compression_DATACHECK.com
|-- rectangle_compression_DATACHECK.dat
|-- rectangle_compression_DATACHECK.mdl
|-- rectangle_compression_DATACHECK.msg
|-- rectangle_compression_DATACHECK.odb
|-- rectangle_compression_DATACHECK.prt
|-- rectangle_compression_DATACHECK.sim
|-- rectangle_compression_DATACHECK.stdout
|-- rectangle_compression_DATACHECK.stt
|-- rectangle_geometry.cae
|-- rectangle_geometry.jnl
|-- rectangle_geometry.stdout
|-- rectangle_mesh.cae
|-- rectangle_mesh.inp
|-- rectangle_mesh.jnl
|-- rectangle_mesh.stdout
|-- rectangle_partition.cae
|-- rectangle_partition.jnl
`-- rectangle_partition.stdout

0 directories, 30 files
Contents