Tutorial 10: Unit Testing

Unit testing is a software development practice that allows developers to verify the functionality of individual units or components of their codebase. In modsim repositories, unit tests play a vital role in verifying custom scripting libraries tailored to the project. This tutorial introduces a project-wide alias, streamlining the execution of unit tests using the pytest [54] framework.

References

• pytest [54]

• Regression Testing

Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   Linux/MacOS

   $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest
   

   Windows

   PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest
   

2. Activate the environment

   Linux/MacOS

   $ conda activate waves-tutorial-env
   

   Windows

   PS > conda activate waves-tutorial-env
   

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Warning

STOP! Before continuing, check that the documentation version matches your installed package version.

1. You can find the documentation version in the upper-left corner of the webpage.

2. You can find the installed WAVES version with waves --version.

If they don’t match, you can launch identically matched documentation with the WAVES Command-Line Utility docs subcommand as waves docs.

Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

Linux/MacOS

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

Windows

PS > New-Item $HOME\waves-tutorials -ItemType "Directory"
PS > Set-Location $HOME\waves-tutorials
PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

Linux/MacOS

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite --tutorial 9 && mv tutorial_09_post_processing_SConstruct SConstruct
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

Windows

PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

PS > waves fetch --overwrite --tutorial 9 && Move-Item tutorial_09_post_processing_SConstruct SConstruct -Force
WAVES fetch
Destination directory: 'C:\Users\roppenheimer\waves-tutorials'

Regression Script

4. In the waves-tutorials/modsim_package/python directory, create a new file named regression.py from the contents below

waves-tutorials/modsim_package/python/regression.py

#!/usr/bin/env python
"""Perform regression testing on simulation output."""

import argparse
import pathlib
import sys

import pandas
import yaml


def sort_dataframe(
    dataframe: pandas.DataFrame,
    index_column: str = "time",
    sort_columns: list[str] | tuple[str, ...] = ("time", "set_name"),
) -> pandas.DataFrame:
    """Return a sorted dataframe and set an index.

    1. sort columns by column name
    2. sort rows by column values ``sort_columns``
    3. set an index

    :param dataframe: dataframe to sort
    :param index_column: name of the column to use an index
    :param sort_columns: name of the column(s) to sort by

    :returns: sorted and indexed dataframe
    """
    return dataframe.reindex(sorted(dataframe.columns), axis=1).sort_values(list(sort_columns)).set_index(index_column)


def csv_files_match(
    current_csv: pandas.DataFrame,
    expected_csv: pandas.DataFrame,
    index_column: str = "time",
    sort_columns: list[str] | tuple[str, ...] = ("time", "set_name"),
) -> bool:
    """Compare two pandas DataFrame objects and determine if they match.

    :param current_csv: Current CSV data of generated plot.
    :param expected_csv: Expected CSV data.
    :param index_column: name of the column to use an index
    :param sort_columns: name of the column(s) to sort by. Defaults to ``["time", "set_name"]``

    :returns: True if the CSV files match, False otherwise.
    """
    current = sort_dataframe(current_csv, index_column=index_column, sort_columns=sort_columns)
    expected = sort_dataframe(expected_csv, index_column=index_column, sort_columns=sort_columns)
    try:
        pandas.testing.assert_frame_equal(current, expected)
    except AssertionError as err:
        print(
            f"The CSV regression test failed. Data in expected CSV file and current CSV file do not match.\n{err}",
            file=sys.stderr,
        )
        equal = False
    else:
        equal = True
    return equal


def main(
    first_file: pathlib.Path,
    second_file: pathlib.Path,
    output_file: pathlib.Path,
) -> None:
    """Compare CSV files and return an error code if they differ.

    :param first_file: path-like or file-like object containing the first CSV dataset
    :param second_file: path-like or file-like object containing the second CSV dataset
    """
    regression_results = {}

    # CSV regression file comparison
    first_data = pandas.read_csv(first_file)
    second_data = pandas.read_csv(second_file)
    regression_results.update({"CSV comparison": csv_files_match(first_data, second_data)})

    with output_file.open(mode="w") as output:
        output.write(yaml.safe_dump(regression_results))

    if len(regression_results.values()) < 1 or not all(regression_results.values()):
        sys.exit("One or more regression tests failed")


def get_parser() -> argparse.ArgumentParser:
    """Return parser for CLI options.

    All options should use the double-hyphen ``--option VALUE`` syntax to avoid clashes with the Abaqus option syntax,
    including flag style arguments ``--flag``. Single hyphen ``-f`` flag syntax often clashes with the Abaqus command
    line options and should be avoided.

    :returns: parser
    :rtype:
    """
    script_name = pathlib.Path(__file__)
    default_output_file = f"{script_name.stem}.yaml"

    prog = f"python {script_name.name} "
    cli_description = "Compare CSV files and return an error code if they differ"
    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
    parser.add_argument(
        "FIRST_FILE",
        type=pathlib.Path,
        help="First CSV file for comparison",
    )
    parser.add_argument(
        "SECOND_FILE",
        type=pathlib.Path,
        help="Second CSV file for comparison",
    )
    parser.add_argument(
        "--output-file",
        type=pathlib.Path,
        default=default_output_file,
        help="Regression test pass/fail list",
    )
    return parser


if __name__ == "__main__":
    parser = get_parser()
    args = parser.parse_args()
    main(
        args.FIRST_FILE,
        args.SECOND_FILE,
        args.output_file,
    )

This script is introduced early for Tutorial 11: Regression Testing because unit testing the post_processing.py functions requires advanced testing techniques. The functions of regression.py can be tested more directly as an introduction to Python unit testing.

Unit test file

5. In the waves-tutorials/modsim_package/python/tests directory, Create a new file named test_regression.py from the contents below

waves-tutorials/modsim_package/python/tests/test_regression.py

"""Test the regression module."""

import pandas

from modsim_package.python import regression


def test_sort_dataframe() -> None:
    """Test :func:`regression.sort_dataframe`."""
    data = {
        "time": [0.0, 0.5, 1.0],
        "Column1": [1, 2, 3],
        "Column2": [4, 5, 6],
    }
    control = pandas.DataFrame.from_dict(data)
    unsorted_copy = control[["Column2", "Column1", "time"]]

    sorted_control = regression.sort_dataframe(control, sort_columns=["time"])
    sorted_copy = regression.sort_dataframe(unsorted_copy, sort_columns=["time"])

    pandas.testing.assert_frame_equal(sorted_control, sorted_copy)


def test_csv_files_match() -> None:
    """Test :func:`regression.csv_files_match`."""
    data = {
        "time": [0.0, 0.5, 1.0],
        "Column1": [1, 2, 3],
        "Column2": [4, 5, 6],
    }
    # Control DataFrame
    control = pandas.DataFrame.from_dict(data)

    # Identical DataFrame
    identical_copy = control.copy()
    unsorted_copy = control[["time", "Column2", "Column1"]]

    # Different DataFrame
    different_copy = control.copy()
    different_copy.loc[0, "Column1"] = 999

    # Assert that the function returns False when the DataFrames differ
    assert regression.csv_files_match(control, different_copy, sort_columns=["time"]) is False

    # Assert that the function returns True when the DataFrames are identical
    assert regression.csv_files_match(control, identical_copy, sort_columns=["time"]) is True

    # Assert that the function returns True when the sorted DataFrames are identical
    assert regression.csv_files_match(control, unsorted_copy, sort_columns=["time"]) is True

In the test_regression.py file, you’ll find a test implementation of two simple functions within the regression.py module. However, the remaining functions delve into more complex territory which need advanced techniques such as mocking. These aspects are intentionally left as exercises for you, the reader, to explore and master. For a deeper understanding of how mocking operates in Python, refer to Unittest Mock [55]. A more complete suite of unit tests may be found in the ModSim Templates.

SConscript

6. In the waves-tutorials directory, create a new file named unit_testing from the contents below

waves-tutorials/unit_testing.scons

#! /usr/bin/env python
"""Rectangle compression workflow.

Requires the following ``SConscript(..., exports={})``

* ``env`` - The SCons construction environment with the following required keys

  * ``regression_alias`` - String for the alias collecting the regression test suite targets
"""

import pathlib

# Inherit the parent construction environment
Import("env")

# Set unit test workflow variables
build_directory = pathlib.Path(Dir(".").abspath)
workflow_name = build_directory.name

# Collect the target nodes to build a concise alias for all targets
workflow = []

# Unit test target
workflow.extend(
    env.Command(
        target=[f"{workflow_name}_results.xml"],
        source=["#/modsim_package/python/tests/test_regression.py"],
        action="pytest --junitxml=${TARGETS[0]}",
    )
)

env.Alias(workflow_name, workflow)
env.Alias(env["regression_alias"], workflow)

For this SCons task, the primary purpose of the pytest JUnit XML output report is to provide SCons with a build target to track. If the project uses a continuous integration server, the output may be used for automated test reporting [56].

SConstruct

7. Update the SConstruct file. A diff against the SConstruct file from Tutorial 09: Post-Processing is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_09_post_processing_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_10_unit_testing_SConstruct
@@ -1,5 +1,5 @@
 #! /usr/bin/env python
-"""Configure the WAVES post-processing tutorial."""
+"""Configure the WAVES unit testing tutorial."""
 
 import os
 import pathlib
@@ -82,6 +82,7 @@
     "project_name": project_name,
     "project_dir": project_dir,
     "version": version,
+    "regression_alias": "regression",
 }
 for key, value in project_variables.items():
     env[key] = value
@@ -117,6 +118,11 @@
     build_dir = env["variant_dir_base"] / pathlib.Path(workflow).stem
     SConscript(workflow, variant_dir=build_dir, exports={"env": env}, duplicate=False)
 
+# Add unit test target
+test_workflow = pathlib.Path("unit_testing.scons")
+test_build_dir = env["variant_dir_base"] / test_workflow.stem
+SConscript(test_workflow, variant_dir=test_build_dir, exports={"env": env}, duplicate=False)
+
 # Comments used in tutorial code snippets: marker-7
 
 # Add default target list to help message

Our test alias is initialized similarly to that of the workflow aliases. In order to clarify that the tests are not part of a modsim workflow, the unit_testing call is made separately from the workflow loop. Additionally, a regression test alias is added as a collector alias for future expansion beyond the unit tests in Tutorial 11: Regression Testing.

Build Targets

8. Build the test results

Linux/MacOS

$ pwd
/home/roppenheimer/waves-tutorials
$ scons unit_testing
<output truncated>

Windows

PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

PS > scons unit_testing
<output truncated>

Output Files

Explore the contents of the build directory using the tree command against the build directory, as shown below. Note that the output files from the previous tutorials may also exist in the build directory, but the directory is specified by name to reduce clutter in the output shown.

Linux/MacOS

$ pwd
/home/roppenheimer/waves-tutorials
$ tree build/unit_testing/
build/unit_testing/
└── unit_testing_results.xml

0 directories, 1 file

Windows

PS > Get-Location

   Path
   ----
   C:\Users\roppenheimer\waves-tutorials

PS > tree build\unit_testing\ /F
C:\USERS\ROPPENHEIMER\WAVES-TUTORIALS\BUILD\UNIT_TESTING
    unit_testing_results.xml

No subfolders exist