Tutorial 11: Regression Testing

Regression testing is the practice of running a verification test suite after making changes to a repository or codebase. For modsim repositories, there may not be many unit or integration tests if there is no software or scripting library specific to the project. Instead, regression testing a modsim repository may look more like regular execution of system tests that verify the simulation workflow still executes as expected.

Ideally, this verification suite of system tests would perform the complete simulation workflow from start to finish. However, modsim repositories often contain simulations that are computationally expensive or produce large amounts of data on disk. In these cases, it may be too expensive to run the full simulation suite at any regular interval. It is still desirable to provide early warning of breaking changes in the simulation workflow, so as much of the workflow that can be tested should be tested as regularly as possible given compute resource constraints.

This tutorial introduces a project wide alias to allow convenient execution of the simulation workflow through the simulation datacheck task introduced in Tutorial 04: Simulation. From that tutorial onward, each tutorial has propagated a tutorial specific datacheck alias. This tutorial will add a project wide datacheck alias and apply it to a copy of the Tutorial 09: Post-Processing configuration files. The user may also go back to previous tutorials to include the full suite of datacheck tasks in the project wide datacheck regression test alias.

In addition to the datachecks, this tutorial will introduce a full simulation results regression test script and task. The regression test task will be added to the regular workflow alias to run everytime the full workflow is run. This test compares the actual simulation results within a float tolerance. Comprehensive results regression testing is valuable to evaluate changes to important quantities of interest when software versions change, e.g. when installing a new version of Abaqus.

After this tutorial, the workflow will have three sets of tests: fast running unit tests introduced in Tutorial 10: Unit Testing, relatively fast running simulation preparation checks with the datacheck alias, and full simulation results regression tests. The tutorial simulations run fast enough that performing the full suite of tests for every change in the project is tractable. However, in practice, projects may choose to run only the unit tests and datachecks on a per-change basis and reserve the comprehensive results testing for a scheduled regression suite.

References

• Regression Testing

• SCons Alias [32]

• Continuous Integration software

  • GitHub Actions: https://docs.github.com/en/actions [52]

  • Gitlab CI: https://docs.gitlab.com/ee/ci/ [53]

  • Bitbucket Pipelines: https://bitbucket.org/product/features/pipelines [54]

Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest
   

2. Activate the environment

   $ conda activate waves-tutorial-env
   

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Warning

STOP! Before continuing, check that the documentation version matches your installed package version.

1. You can find the documentation version in the upper-left corner of the webpage.

2. You can find the installed WAVES version with waves --version.

If they don’t match, you can launch identically matched documentation with the WAVES Command-Line Utility docs subcommand as waves docs.

Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite --tutorial 10 && mv tutorial_10_unit_testing_SConstruct SConstruct
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

4. Download and copy the tutorial_09_post_processing file to a new file named tutorial_11_regression_testing with the WAVES Command-Line Utility fetch subcommand.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite tutorials/tutorial_09_post_processing && cp tutorial_09_post_processing tutorial_11_regression_testing
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

Regression Script

5. In the waves-tutorials/modsim_package/python directory, review the file named regression.py introduced in Tutorial 10: Unit Testing.

waves-tutorials/modsim_package/python/regression.py

#!/usr/bin/env python
import sys
import pathlib
import argparse

import pandas
import yaml


def sort_dataframe(dataframe, index_column="time", sort_columns=["time", "set_name"]):
    """Return a sorted dataframe and set an index

    1. sort columns by column name
    2. sort rows by column values ``sort_columns``
    3. set an index

    :returns: sorted and indexed dataframe
    :rtype: pandas.DataFrame
    """
    return dataframe.reindex(sorted(dataframe.columns), axis=1).sort_values(sort_columns).set_index(index_column)


def csv_files_match(current_csv, expected_csv, index_column="time", sort_columns=["time", "set_name"]):
    """Compare two pandas DataFrame objects and determine if they match.

    :param pandas.DataFrame current_csv: Current CSV data of generated plot.
    :param pandas.DataFrame expected_csv: Expected CSV data.

    :returns: True if the CSV files match, False otherwise.
    :rtype: bool
    """
    current = sort_dataframe(current_csv, index_column=index_column, sort_columns=sort_columns)
    expected = sort_dataframe(expected_csv, index_column=index_column, sort_columns=sort_columns)
    try:
        pandas.testing.assert_frame_equal(current, expected)
    except AssertionError as err:
        print(
            f"The CSV regression test failed. Data in expected CSV file and current CSV file do not match.\n{err}",
            file=sys.stderr,
        )
        equal = False
    else:
        equal = True
    return equal


def main(
    first_file: pathlib.Path,
    second_file: pathlib.Path,
    output_file: pathlib.Path,
) -> None:
    """Compare CSV files and return an error code if they differ

    :param first_file: path-like or file-like object containing the first CSV dataset
    :param second_file: path-like or file-like object containing the second CSV dataset
    """
    regression_results = {}

    # CSV regression file comparison
    first_data = pandas.read_csv(first_file)
    second_data = pandas.read_csv(second_file)
    regression_results.update({"CSV comparison": csv_files_match(first_data, second_data)})

    with open(output_file, "w") as output:
        output.write(yaml.safe_dump(regression_results))

    if len(regression_results.values()) < 1 or not all(regression_results.values()):
        sys.exit("One or more regression tests failed")


def get_parser() -> argparse.ArgumentParser:
    """Return parser for CLI options

    All options should use the double-hyphen ``--option VALUE`` syntax to avoid clashes with the Abaqus option syntax,
    including flag style arguments ``--flag``. Single hyphen ``-f`` flag syntax often clashes with the Abaqus command
    line options and should be avoided.

    :returns: parser
    :rtype:
    """
    script_name = pathlib.Path(__file__)
    default_output_file = f"{script_name.stem}.yaml"

    prog = f"python {script_name.name} "
    cli_description = "Compare CSV files and return an error code if they differ"
    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
    parser.add_argument(
        "FIRST_FILE",
        type=pathlib.Path,
        help="First CSV file for comparison",
    )
    parser.add_argument(
        "SECOND_FILE",
        type=pathlib.Path,
        help="Second CSV file for comparison",
    )
    parser.add_argument(
        "--output-file",
        type=pathlib.Path,
        default=default_output_file,
        help="Regression test pass/fail list",
    )
    return parser


if __name__ == "__main__":
    parser = get_parser()
    args = parser.parse_args()
    main(
        args.FIRST_FILE,
        args.SECOND_FILE,
        args.output_file,
    )

CSV file

6. In the waves-tutorials/modsim_package/python directory, create a new file named rectangle_compression_cartesian_product.csv from the contents below

waves-tutorials/modsim_package/python/rectangle_compression_cartesian_product.csv

time,set_name,step,elements,integrationPoint,E values,E,S values,S,width,height,global_seed,displacement,set_hash
0.0175000000745058,parameter_set0,Step-1,1,1.0,E22,-0.000175,S22,-0.0175,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
0.0175000000745058,parameter_set1,Step-1,1,1.0,E22,-0.000159091,S22,-0.0159091,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
0.0175000000745058,parameter_set2,Step-1,1,1.0,E22,-0.000175,S22,-0.0175,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
0.0175000000745058,parameter_set3,Step-1,1,1.0,E22,-0.000159091,S22,-0.0159091,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
0.0709374994039536,parameter_set0,Step-1,1,1.0,E22,-0.000709375,S22,-0.0709375,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
0.0709374994039536,parameter_set1,Step-1,1,1.0,E22,-0.000644886,S22,-0.0644886,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
0.0709374994039536,parameter_set2,Step-1,1,1.0,E22,-0.000709375,S22,-0.0709375,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
0.0709374994039536,parameter_set3,Step-1,1,1.0,E22,-0.000644886,S22,-0.0644886,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
0.251289069652557,parameter_set0,Step-1,1,1.0,E22,-0.00251289,S22,-0.251289,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
0.251289069652557,parameter_set1,Step-1,1,1.0,E22,-0.00228445,S22,-0.228445,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
0.251289069652557,parameter_set2,Step-1,1,1.0,E22,-0.00251289,S22,-0.251289,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
0.251289069652557,parameter_set3,Step-1,1,1.0,E22,-0.00228445,S22,-0.228445,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
0.859975576400757,parameter_set0,Step-1,1,1.0,E22,-0.00859976,S22,-0.859976,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
0.859975576400757,parameter_set1,Step-1,1,1.0,E22,-0.00781796,S22,-0.781796,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
0.859975576400757,parameter_set2,Step-1,1,1.0,E22,-0.00859976,S22,-0.859976,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
0.859975576400757,parameter_set3,Step-1,1,1.0,E22,-0.00781796,S22,-0.781796,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
1.0,parameter_set0,Step-1,1,1.0,E22,-0.01,S22,-1.0,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
1.0,parameter_set1,Step-1,1,1.0,E22,-0.00909091,S22,-0.909091,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
1.0,parameter_set2,Step-1,1,1.0,E22,-0.01,S22,-1.0,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
1.0,parameter_set3,Step-1,1,1.0,E22,-0.00909091,S22,-0.909091,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f

This file represents a copy of previous simulation results that the project has stored as the reviewed and approved simulation results. The regression task will compare these “past” results with the current simulation results produced during the post-processing task introduced in Tutorial 09: Post-Processing rectangle_compression_cartesian_product.csv using the regression.py CLI.

SConscript

A diff against the tutorial_09_post_processing file from Tutorial 09: Post-Processing is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_11_regression_testing

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_09_post_processing
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_11_regression_testing
@@ -5,6 +5,8 @@
 
 * ``env`` - The SCons construction environment with the following required keys
 
+  * ``datacheck_alias`` - String for the alias collecting the datacheck workflow targets
+  * ``regression_alias`` - String for the alias collecting the regression test suite targets
   * ``unconditional_build`` - Boolean flag to force building of conditionally ignored targets
   * ``abaqus`` - String path for the Abaqus executable
 """
@@ -176,11 +178,27 @@
     )
 )
 
+# Regression test
+workflow.extend(
+    env.PythonScript(
+        target=["regression.yaml"],
+        source=[
+            "#/modsim_package/python/regression.py",
+            "stress_strain_comparison.csv",
+            "#/modsim_package/python/rectangle_compression_cartesian_product.csv",
+        ],
+        subcommand_options="${SOURCES[1:].abspath} --output-file ${TARGET.abspath}",
+    )
+)
+
 # Collector alias based on parent directory name
 env.Alias(workflow_name, workflow)
 env.Alias(f"{workflow_name}_datacheck", datacheck)
+env.Alias(env["datacheck_alias"], datacheck)
+env.Alias(env["regression_alias"], datacheck)
 
 if not env["unconditional_build"] and not env["ABAQUS_PROGRAM"]:
     print(f"Program 'abaqus' was not found in construction environment. Ignoring '{workflow_name}' target(s)")
     Ignore([".", workflow_name], workflow)
     Ignore([".", f"{workflow_name}_datacheck"], datacheck)
+    Ignore([".", env["datacheck_alias"], env["regression_alias"]], datacheck)

There are two changes made in this tutorial. The first is to compare the expected simulation results to the current simulation’s output. A new task compares the expected results as the CSV file created above against the current simulation output with the new Python script, regression.py. See the regression.py CLI documentation for a description of the post-processing script’s behavior.

The second change adds a dedicated alias for the datacheck targets to allow partial workflow execution. This is useful when a full simulation may take a long time, but the simulation preparation is worth testing on a regular basis. We’ve also added the regression alias introduced briefly in Tutorial 10: Unit Testing. Previously, this alias was a duplicate of the unit_testing workflow alias. Now this alias can be used as a collector alias for running a regression suite with a single command, while preserving the ability to run the unit tests as a standalone workflow.

Here we add the datacheck targets as an example of running a partial workflow as part of the regression test suite. For fast running simulations, it would be valuable to run the full simulation and post-processing with CSV results testing as part of the regular regression suite. For large projects with long running simulations, several regression aliases may be added to facilitate testing at different intervals. For instance, the datachecks might be run everytime the project changes, but the simulations might be run on a weekly schedule with a regression_weekly alias that includes the full simulations in addition to the unit tests and datachecks.

SConstruct

A diff against the SConstruct file from Tutorial 10: Unit Testing is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_10_unit_testing_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_11_regression_testing_SConstruct
@@ -82,6 +82,7 @@
     "project_dir": project_dir,
     "version": version,
     "regression_alias": "regression",
+    "datacheck_alias": "datacheck",
 }
 for key, value in project_variables.items():
     env[key] = value
@@ -112,6 +113,7 @@
     "tutorial_07_cartesian_product",
     "tutorial_08_data_extraction",
     "tutorial_09_post_processing",
+    "tutorial_11_regression_testing",
 ]
 for workflow in workflow_configurations:
     build_dir = env["variant_dir_base"] / workflow

Build Targets

7. Build the datacheck targets without executing the full simulation workflow

$ pwd
/home/roppenheimer/waves-tutorials
$ time scons datacheck --jobs=4
<output truncated>
scons: done building targets.

real 0m9.952s
user 0m21.537s
sys  0m15.664s

8. Run the full workflow and verify that the CSV regression test passes

$ pwd
/home/roppenheimer/waves-tutorials
$ scons datacheck --clean
$ time scons tutorial_11_regression_testing --jobs=4
<output truncated>
scons: done building targets.

real 0m29.031s
user 0m25.712s
sys  0m25.622s
$ cat build/tutorial_11_regression_testing/regression.yaml
CSV comparison: true

If you haven’t added the project-wide datacheck alias to the previous tutorials, you should expect the datacheck alias to run faster than the tutorial_11_regression_testing alias because the datacheck excludes the solve, extract, and post-processing tasks. In these tutorials, the difference in execution time is not large. However, in many production modsim projects, the simulations may require hours or even days to complete. In that case, the relatively fast running solverprep verification may be tractable for regular testing where the full simulations and post-processing are not.

To approximate the time savings of the new project-wide datacheck alias for a (slightly) larger modsim project, you can go back through the previous tutorials and add each tutorial’s datacheck task to the new alias. For a fair comparison, you will also need to add a comparable alias to collect the full workflow for each tutorial, e.g. full_workflows. You can then repeat the time commands above with a more comprehensive datacheck and full_workflows aliases.

Output Files

$ pwd
/home/roppenheimer/waves-tutorials
$ tree build/tutorial_11_regression_testing/parameter_set0/
build/tutorial_11_regression_testing/parameter_set0/
|-- abaqus.rpy
|-- abaqus.rpy.1
|-- abaqus.rpy.2
|-- assembly.inp
|-- boundary.inp
|-- field_output.inp
|-- history_output.inp
|-- materials.inp
|-- parts.inp
|-- rectangle_compression.inp
|-- rectangle_compression.inp.in
|-- rectangle_compression_DATACHECK.023
|-- rectangle_compression_DATACHECK.com
|-- rectangle_compression_DATACHECK.dat
|-- rectangle_compression_DATACHECK.mdl
|-- rectangle_compression_DATACHECK.msg
|-- rectangle_compression_DATACHECK.odb
|-- rectangle_compression_DATACHECK.prt
|-- rectangle_compression_DATACHECK.sim
|-- rectangle_compression_DATACHECK.stdout
|-- rectangle_compression_DATACHECK.stt
|-- rectangle_geometry.cae
|-- rectangle_geometry.jnl
|-- rectangle_geometry.stdout
|-- rectangle_mesh.cae
|-- rectangle_mesh.inp
|-- rectangle_mesh.jnl
|-- rectangle_mesh.stdout
|-- rectangle_partition.cae
|-- rectangle_partition.jnl
`-- rectangle_partition.stdout

0 directories, 31 files
$ tree build/tutorial_11_regression_testing/ -L 1
build/tutorial_11_regression_testing/
|-- parameter_set0
|-- parameter_set1
|-- parameter_set2
|-- parameter_set3
|-- parameter_study.h5
|-- regression.yaml
|-- regression.yaml.stdout
|-- stress_strain_comparison.csv
|-- stress_strain_comparison.pdf
`-- stress_strain_comparison.stdout

4 directories, 6 files

Workflow Visualization

View the workflow directed graph by running the following command and opening the image in your preferred image viewer. Plot the workflow with only the first set, set0.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves visualize datacheck --output-file tutorial_11_datacheck_set0.png --width=42 --height=8 --exclude-list /usr/bin .stdout .jnl .prt .com .msg .dat .sta --exclude-regex "set[1-9]"

The output should look similar to the figure below.

This tutorial’s datacheck directed graph should look different from the graph in Tutorial 09: Post-Processing. Here we have plotted the datacheck alias output, which does not execute the full simulation workflow. This partial directed graph may run faster than the full simulation workflow for frequent regression tests.

Automation

There are many tools that can help automate the execution of the modsim project regression tests. With the collector alias, those tools need only execute a single SCons command to perform the selected, lower cost tasks for simulation workflow verification, scons datacheck. If git [21] is used as the version control system, developer operations software such as GitHub [24], Gitlab [25], and Atlassian’s Bitbucket [26] provide continuous integration software that can automate verification tests on triggers, such as merge requests, or on a regular schedule.