Tutorial 00: SConstruct

Tutorial 00: SConstruct#

Prerequisites#

  1. Computational Practices - Build System

  2. Software Carpentry: GNU Make - https://swcarpentry.github.io/make-novice/ [1, 9, 30]

  3. Software Carpentry: Python - https://swcarpentry.github.io/python-novice-inflammation/ [1, 8]

References#

Environment#

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

  1. Create the tutorials environment if it doesn’t exist

    $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

  2. Activate the environment

    $ conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Warning

STOP! Before continuing, check that the documentation version matches your installed package version.

  1. You can find the documentation version in the upper-left corner of the webpage.

  2. You can find the installed WAVES version with waves --version.

If they don’t match, you can launch identically matched documentation with the WAVES Command-Line Utility docs subcommand as waves docs.

Description#

WAVES is a suite of build wrappers and command-line utilities for the build system SCons. Build systems help automate the execution of complex computing workflows by constructing component task execution order. In the WAVES tutorials there are two components to the SCons project configuration: project configuration and simulation workflow configurations. This tutorial introduces the SCons project configuration file for the WAVES tutorials and ModSim Templates.

The command-line utilities provided by WAVES are those utilities required to implement engineering workflows in traditional software build systems. For most engineering simulation workflows, software build systems will work out-of-the-box. However, it is difficult to implement engineering parameter studies in software build systems, which are designed to produce a single program, not many nearly identical configurations of the same program. The WAVES parameter generator command-line interface(s) is designed to work with most build systems, but were originally developed with the requirements of CMake in mind.

For production-engineering analysis, WAVES focuses on extending the build system SCons because SCons configuration files use Python as a fully featured scripting language. This choice is primarily driven by the familiarity of the engineering community with Python as a programming language, but also because the parameter generation utility can be integrated more closely with the build system via a Python API, Parameter Generators.

This tutorial will build out the project configuration file, named SConstruct by default. First a code snippet will be introduced followed by a description of what that snippet does. After sufficient code is added to the file, it can be executed with the scons command. Later tutorials will continue to expand the SConstruct file and add workflow configuration files.

SCons Project Configuration#

  1. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

  1. Create a new file named SConstruct in the waves-tutorials directory and add the contents listed below. Note that the filename is case sensitive and does not use a file extension.

waves-tutorials/SConstruct

1#! /usr/bin/env python
2
3import os
4import pathlib
5
6import waves

By convention, the SCons root project file is named SConstruct. Because this is a Python file, we can import Python libraries to help define project settings. The shebang in the first line is included to help text editors identify the file as a Python file for syntax highlighting. Using the PEP-8 formatting, the Python built-in imports are listed in the first block and third-party imports are listed in the second block, including the WAVES package.

  1. Add the content below to the SConstruct file to add the project’s command-line build options to the project configuration.

Note

The line numbers may jump between code snippets. The skipped lines are not important to the tutorial content as you create the tutorial files. They are a side effect of methods used to robustly create these code snippets from the original source code.

waves-tutorials/SConstruct

 9# Accept command line options with fall back default values
10AddOption(
11    "--build-dir",
12    dest="variant_dir_base",
13    default="build",
14    nargs=1,
15    type="string",
16    action="store",
17    metavar="DIR",
18    help="SCons build (variant) root directory. Relative or absolute path. (default: '%default')",
19)
20AddOption(
21    "--unconditional-build",
22    dest="unconditional_build",
23    default=False,
24    action="store_true",
25    help="Boolean flag to force building of conditionally ignored targets. (default: '%default')",
26)
27AddOption(
28    "--print-build-failures",
29    dest="print_build_failures",
30    default=False,
31    action="store_true",
32    help="Print task *.stdout target file(s) on build failures. (default: '%default')",
33)
34# Python optparse appends to the default list instead of overriding. Must implement default/override ourselves.
35default_abaqus_commands = [
36    "/apps/abaqus/Commands/abq2024",
37    "/usr/projects/ea/abaqus/Commands/abq2024",
38    "abq2024",
39    "abaqus",
40]
41AddOption(
42    "--abaqus-command",
43    dest="abaqus_command",
44    nargs=1,
45    type="string",
46    action="append",
47    metavar="COMMAND",
48    help=f"Override for the Abaqus command. Repeat to specify more than one (default: {default_abaqus_commands})",
49)

The SCons command-line build options are specific to the project configuration that you are currently creating. SCons projects may add or remove command-line options to aid in build behavior control. The most relevant option to modsim projects will be --build-dir and --abaqus-command, which allow project developers to change the build directory location and Abaqus executable path from the command-line without modifying the SConstruct file source code.

For example, when there are workflow tasks defined in the the SConstruct file, a call to scons would create the default build directory named build and a call to scons --build-dir=non_default_build would create a build directory named non_default_build. The current SConstruct file will not produce these directories yet because there are no workflow tasks defined. The demonstration below shows how the build directory option will behave when workflow tasks are defined.

Warning

The equals sign separator is important to the internal implementation of options defined by AddOption in SConstruct files. When whitespace is used to delimit the option from the argument, the argument handling is undefined and may result in unexpected behavior. Interested readers can read more in the SCons documentation: https://scons.org/doc/production/HTML/scons-user.html#app-functions

$ scons
$ ls
build SConstruct
$ scons --build-dir=non_default_build
$ ls
build non_default_build SConstruct

The --abaqus-command option will be useful if the tutorial files struggle to find your Abaqus installation. You can also edit the default_abaqus_commands list to include your Abaqus installation to avoid having to provide it on the command line every time you call scons.

$ scons --abaqus-command=/path/to/installed/abaqus

The --unconditional-build option is mostly useful for Regression Testing and continuous integration. It is used in the tutorial workflows to force a workflow to execute, even if the required programs are not found. This is useful for system testing software availability in the construction environment itself.

By default, WAVES builders redirect task STDOUT and STDERR to a *.stdout file to avoid cluttering the task execution reported by SCons. The --print-build-failures will print the associated *.stdout file of all failed tasks at the end of SCons execution.

At the end of this tutorial, you will see how to explore the project specific command-line options help and usage.

  1. Add the content below to the SConstruct file to initialize the waves.scons_extensions.WAVESEnvironment SCons construction environment.

waves-tutorials/SConstruct

53# Inherit user's full environment and set project variables
54env = waves.scons_extensions.WAVESEnvironment(
55    ENV=os.environ.copy(),
56    variant_dir_base=pathlib.Path(GetOption("variant_dir_base")),
57    unconditional_build=GetOption("unconditional_build"),
58    print_build_failures=GetOption("print_build_failures"),
59    abaqus_commands=GetOption("abaqus_command"),
60)
61
62# Conditionally print failed task *.stdout files
63env.PrintBuildFailures(print_stdout=env["print_build_failures"])

Most build systems inherit the user’s active shell environment at build configuration time, referred to as the “external” environment in the SCons documentation. This means that (most) of the environment must be identical for all build tasks in the project. SCons differs from most build systems by managing the construction environment for each task separately from the external environment. SCons projects do not inherit the user’s shell environment at build configuration by default. Instead, projects define one or more construction environments that is used to define per-task environment configuration.

While this is a powerful feature for large, complex projects, most modsim projects will benefit from maintaining a single construction environment inherited from the active shell environment at build configuration time. In addition to copying the active external environment, the above code adds the project command-line build options to the construction environment for reuse throughout the project configuration files, SConstruct and SConscript, for build control.

  1. Add the content below to the SConstruct file to add the third-party software dependency checks to the project configuration.

waves-tutorials/SConstruct

67# Find required programs for conditional target ignoring and absolute path for use in target actions
68env["ABAQUS_PROGRAM"] = env.AddProgram(
69    env["abaqus_commands"] if env["abaqus_commands"] is not None else default_abaqus_commands
70)

These checks are not strictly required for an SCons WAVES modsim project; however, they provide valuable build control options for developers. Most of the WAVES tutorials compute environment dependencies are Python packages managed with Conda as described in the Environment section of this tutorial. Many modsim repositories will also depend on proprietary or commercial software that is not included in a package manager such as Conda. Instead, the project configuration can check the construction environment for software installation(s) and provide an environment variable to conditionally skip any tasks that depend on missing software.

In WAVES and WAVES tutorials, this approach is primarily used to allow developers to perform development work on local computers without cluttering their test builds with tasks that cannot succeed on their local computer. Tutorial 01: Geometry will introduce the use of these variables for build control.

The SCons native solution for finding a program is the CheckProg configuration method. The waves.scons_extensions.add_program() method wraps CheckProg like behavior to search for a list of possible program names. This is most useful when multiple versions of a program can be used to build the project and the various servers where the project is built may have different versions available. The project will build with the first available program name without changes to the project configuration. If a program is found, it will also be added to the SCons environment PATH.

The env.AddProgram was added to the construction environment with waves.scons_extensions.WAVESEnvironment for convenience. Anywhere the construction environment is available, we can use env.AddProgram. It is also possible to use the waves.scons_extensions.add_program() function directly. Most projects will search for more than one third-party software, so the shortened env.AddProgram function call is also less to type.

Note

The Abaqus program naming convention used here is specific to the naming convention used on the WAVES continuous integration server. Users may need to update the abaqus program name to match their system installation.

  1. Add the content below to the SConstruct file to add the project meta data to the construction environment.

waves-tutorials/SConstruct

74# Set project internal variables and variable substitution dictionaries
75project_name = "WAVES-TUTORIAL"
76version = "0.1.0"
77project_dir = pathlib.Path(Dir(".").abspath)
78project_variables = {
79    "project_name": project_name,
80    "project_dir": project_dir,
81    "version": version,
82}
83for key, value in project_variables.items():
84    env[key] = value

The WAVES tutorials make use of the SCons hierarchical build feature to separate simulation output in the build directory. This is valuable for modsim repositories that include a suite of simulations. To avoid hardcoded duplication of project meta data, the project meta data variables are added to the construction environment, which will be passed around to all SCons configuration files. The implementation that passes the construction environment around is introduced in Tutorial 01: Geometry.

  1. Add the content below to the SConstruct file to add a placeholder for WAVES builders to the project configuration.

waves-tutorials/SConstruct

88# Add builders and pseudo-builders
89env.Append(BUILDERS={})

Although it is possible to re-create the WAVES tutorials entirely in native SCons code, the builder extensions provided by WAVES reduce the requisite background knowledge to begin creating modsim repositories. While the waves.scons_extensions.WAVESEnvironment construction environment provides some useful default builders, it is possible and often valuable to add project custom-tailored builders, as well. The construction environment BUILDERS variable must be updated to include these custom SCons builders and make them available to the simulation configuration starting in Tutorial 04: Simulation.

The WAVES SCons Extensions API describes the available builder factories and their usage. As WAVES matures, more software will be supported with build wrappers. Prior to a WAVES builder, modsim developers can create their own SCons custom builders or start from a WAVES template builder factory in Tutorial: Writing Builders.

  1. Add the content below to the SConstruct file to create a placeholder call to the hierarchical simulation configuration files.

waves-tutorials/SConstruct

93# Add simulation targets
94workflow_configurations = []
95for workflow in workflow_configurations:
96    build_dir = env["variant_dir_base"] / workflow
97    SConscript(workflow, variant_dir=build_dir, exports={"env": env}, duplicate=False)

The for loop in this code-snippet is the method for implementing an SCons hierarchical build. The exports keyword argument allows the project configuration file to pass the env construction environment variable with the SCons sharing environments feature. The first simulation configuration will be added to the workflow_configurations list in Tutorial 01: Geometry.

  1. Add the content below to the SConstruct file to add an empty default target list and to modify the project help message.

waves-tutorials/SConstruct

101# Add default target list to help message
102env.Default()  # Empty defaults list to avoid building all simulation targets by default
103# Add aliases to help message so users know what build target options are available
104# This must come *after* all expected Alias definitions and SConscript files.
105env.ProjectHelp()

Because the WAVES tutorials contain a suite of simulations, it is useful to limit what SCons will build by default. To protect against running all simulations by default, create an empty default list. This will require that simulation targets are specified by name in the SCons build command. In addition to limiting the default target list, it is useful to print the list of default targets in the project help to remind developers what targets will build when no target is specified in the call to scons.

Simulation build workflows will typically involve many targets and tasks in a non-trivial execution order. The target file names may also be cumbersome to type when explicitly listing build targets in the SCons build command. For convenience, the WAVES tutorials simulation configurations will add a collector alias for the list of simulation targets with the SCons Alias feature. By convention, the WAVES tutorials match the alias name to the simulation subdirectory name. Tutorial 01: Geometry will introduce the first target alias, which will then populate the project help message displayed by the scons -h command option.

We previously added the ProjectHelp method to the construction environment using waves.scons_extensions.WAVESEnvironment. This points to waves.scons_extensions.project_help() which wraps two common calls to SCons Help that will append the following to the project help message accessed by scons -h:

  • the command-line build options

  • the default target list

  • the project alias list

The help message recovers targets and aliases from the construction environment, so the env.ProjectHelp call must come after all SConscript and Alias method calls. Generally, it’s best to simply call this method as the final line in your project configuration.

Note

The project help message uses the conventional lowercase help option, -h. Most bash commands use this option to display the command’s help message corresponding to the software options. Due to this behavior, the SCons command options are displayed with the unconventional capitalized option, -H, as scons -H. The WAVES tutorials and documentation will not discuss the full range of SCons command options, so modsim developers are encouraged to read the SCons usage summary and SCons manpage to learn more about available build control options.

  1. Explore the project help message

$ pwd
/home/roppenheimer/waves-tutorials
$ scons -h
scons: Reading SConscript files ...
Checking whether /apps/abaqus/Commands/abq2024 program exists.../apps/abaqus/Commands/abq2024
Checking whether abq2024 program exists.../apps/abaqus/Commands/abq2024
scons: done reading SConscript files.
Local Options:
  --build-dir=DIR         SCons build (variant) root directory. Relative or absolute path. (default: 'build')
  --unconditional-build   Boolean flag to force building of conditionally ignored targets, e.g. if the target's
                            action program is missing and it would normally be ignored. (default: 'False')
  --print-build-failures  Print task *.stdout target file(s) on build failures. (default: 'False')

Default Targets:

Target Aliases:

Use scons -H for help about SCons built-in command-line options.

The text shown in the sample code block above is the project specific help message(s) added in the previous step. The Default Targets: and Target Aliases: lists will begin to populate in the following tutorial.

Contents