Developer Manual

Developer Manual#

Clone#

GitHub

$ git clone git@github.com:lanl-aea/waves.git

LANL

$ git clone ssh://git@re-git.lanl.gov:10022/aea/python-projects/waves.git

Dependencies#

The development environment dependencies are found in the project’s environment.yml file. For convenience, the file is reproduced here

channels:
  - fierromechanics
  - conda-forge

dependencies:
  - black
  - boa
  - calculix
  - ccx2paraview >=3.2
  - fierro-fe-cpu
  # Remove trilinos-cpu explicit requirement when Fierro package(s) resolve the missing library issue
  # https://re-git.lanl.gov/aea/python-projects/waves/-/issues/865
  - trilinos-cpu <=15
  - flake8
  - glab >=1.52
  - h5netcdf
  - h5py
  - ipython
  - lazy_loader >=0.4
  - matplotlib-base
  - meshio
  - mypy
  - networkx
  # Required to satisfy undocumented SALib dependencies on numpy version
  - numpy >=2
  - pandas
  - pip
  - pytest>=6
  - pytest-cov
  - pytest-xdist
  - python-build
  - python-gmsh
  - pyyaml
  # Required to satisfy undocumented SALib dependencies on numpy version
  - salib >=1.5.1
  - scipy >=1
  - scons >=4.6
  - seaborn
  - setuptools >=64
  - setuptools_scm >=8
  - sphinx >=7.1
  - sphinx-argparse
  - sphinx-book-theme
  - sphinx-copybutton >=0.5.1
  - sphinx-design
  - sphinxcontrib-bibtex
  - twine >=5
  # Remove explicit dependency on VTK and rely on ccx2paraview when a VTK >=9.4 version fixes the macOS crash
  # https://re-git.lanl.gov/aea/python-projects/waves/-/issues/901
  - vtk <=9.3
  - xarray

In addition to the Conda environment managed packages, this project also requires a separately managed installation of TeXLive to build the PDF documentation.

The tutorials in this project may also require:

Abaqus
Cubit
Matlab

Activate Environment#

Local development environments#

SCons can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Create the environment if it doesn’t exist

$ pwd
path/to/local/git/clone/waves
$ conda env create --name waves-env --file environment.yml

Activate the environment
```
$ conda activate waves-env
```

In addition to the primary development environment file environment.yml, several other environment files are maintained for CI jobs. For Windows developers, environment-win.yml removes packages that are not available for Windows and packages that are only necessary for deployment jobs. The conda-build.yml and pip-build.yml are stripped down to the bare essentials for building Conda and pip packages, respectively.

AEA CI server environment#

A full development environment for the project Gitlab-CI pipelines is maintained on AEA servers.

Add the AEA modulefiles directory

$ module use /projects/aea_compute/modulefiles

Load the project specific modulefile
```
$ module load waves-env
```

The Conda packages found in environment.yml are reproduced in the Dependencies section.

HPC CI server environment#

For computing policy reasons, HPC CI pipelines are owned by the launching user and launched with the user’s account. The HPC CI server environment must be created in the launching user’s scratch space. For merge request pipelines, this means a project development environment will be created in the submitting developer’s scratch space, e.g. ${system_scratch}/$USER/waves-env.

Because the CI pipeline runs as the launching user, Conda will create a package and environment cache according to the user’s HPC Conda configuration. By default, Conda creates ~/.conda/pkgs for the package cache, which can grow quite large. If a user’s home directory space is limited, developers are highly encouraged to configure Conda to use their scratch space for the package cache, e.g. with a ~/.condarc file using the template below, where the text ${user_scratch} is replaced by the absolute path to the user’s scratch directory.

envs_dirs:
  - ${user_scratch}/conda/envs

pkgs_dirs:
  - ${user_scratch}/conda/pkgs

You can read more about managing the Conda package and environment cache configuration here: https://conda.io/projects/conda/en/latest/user-guide/configuration/custom-env-and-pkg-locations.html

Windows CI environment#

The Windows CI server must have a system (or Gitlab-Runner user) installation of the following software separately from the Conda environment.

Git
Git-LFS
TeXLive
Anaconda/Miniconda/Miniforge

The Windows CI server environment is created under the default Gitlab-Runner user in the Gitlab-Runner build directory and is not be available for developers. The Windows CI jobs use a mix of the environment-win.yml, conda-build.yml, and pip-build.yml files. Windows developers may use environment-win.yml to create a local development environment that closely mirrors the linux CI environment.

MacOS CI environment#

The MacOS CI server must have a system (or Gitlab-Runner user) installation of the following software separately from the Conda environment.

Git
Git-LFS
TeXLive
Anaconda/Miniconda/Miniforge

The MacOS CI server environment is created under the default Gitlab-Runner user in the Gitlab-Runner build directory and is not available for developers. The MacOS CI jobs use a mix of the environment.yml, conda-build.yml, and pip-build.yml files. MacOS developers may use the full environment.yml file to create a local development environment that closely mirrors the linux CI environment.

Contribution Guidelines#

The WAVES project uses the Gitlab Flow [25] workflow model.

Git Commit Message#

Begin Git commit messages with one of the following headings:

BUG: bug fix
DOC: documentation
FEAT: feature
MAINT: maintenance
TST: tests
REL: release
WIP: work-in-progress

For example:

git commit -m "FEAT: short intent of new feature"
git commit -m "BUG: fixes nasty bug"
git commit -m "DOC: adds documentation for feature"

reStructured Text#

Sphinx reads in docstrings and other special portions of the code as reStructured text. Developers should follow styles in this Sphinx style guide.

Style Guide#

This project does not yet have a full style guide. Generally, wherever a style can’t be inferred from surrounding code this project falls back to PEP-8-like styles. There are two notable exceptions to the notional PEP-8 fall back:

Build#

This project uses the SCons build system. This section will discuss some common build operations. For a full list of SCons command-line options and target build behavior, see the SCons manpage. The SCons manpage is also installed with Scons in the environment and can be opened from the command-line as man scons in the AEA Compute environment.

View project specific command-line options, default targets, and aliases
```
$ scons -h
```
Build all default targets
```
$ scons
```
Build a specific target
```
$ scons <target name>
```
Remove the default targets’ artifacts
```
$ scons --clean
```
Remove all targets’ artifacts
```
$ scons . --clean
```

This project may be built as a Conda or pip package. The Conda package is the preferred distribution and takes the default distribution name waves. The Conda recipe(s) will call the pip build commands directly, so it is not necessary to build the pip package first.

pip#

To build the pip package, use the build alias. The package artifacts will live in build/dist.

$ scons build

The package distribution name may be changed from the default waves to something else. For instance, on PyPI the distribution name waves is taken by another project, so the PyPI distribution name of this project is waves-workflows.

$ scons build --distribution-name=waves-workflows

For this reason the pyproject.toml file has been templated in the source tree as pyproject.toml.in. The necessary pyproject.toml file will be generated as necessary with the specified distribution name by the SCons build, install, style, and test aliases.

The install alias is provided for packaging in ecosystems which support data file installation outside the package directory, e.g. man pages. By default, this alias installs to a local install/ directory, but the prefix may be set to match the recipe’s installation prefix. For instance, the Conda recipe must install to the ${PREFIX} variable managed by conda-build.

$ scons install --prefix=${PREFIX}

If a pyproject.toml file has been generated, the package may be built with the usual py-build and pip commands, as well. Such an installation will not include the man pages or HTML documentation and the waves docs subcommand will not work to launch locally installed documentation. Direct pip builds should be avoided as general practice, but may be desirable for package maintainers that require more direct control over the build and install options.

$ scons pyproject.toml
$ python -m build .
$ python -m pip install dist/waves-*.tar.gz

Conda#

To build the Conda package activate the development environment and run the conda build command found in the CI configuration file. The current command may be found as

$ sed -n '/output_folder=/,/VERSION/p' .gitlab-ci.yml .pipeline*.yml
...

$ output_folder='conda-bld'
$ mkdir ${output_folder}
$ VERSION=$(python -m setuptools_scm) conda build recipe --channel conda-forge --no-anaconda-upload --output-folder ${output_folder}

A second recipe that bundles the LANL internally linked documentation is found in waves/recipe-internal and can be built similarly by replacing recipe with recipe-internal in the above command.

Tutorials#

This project contains several, separate SCons project configurations, where the SConstruct file name indicates an SCons project by convention. The WAVES package and documentation are defined in the waves/SConstruct file. The modsim template and regression tests are defined in a separate waves/modsim_template/Sconstruct file. The WAVES tutorials each have a tutorial specific configuration file waves/tutorials/*SConstruct. The following build commands apply to each, but must be run from their respective project configuration parent directories, waves, waves/modsim_template and waves/tutorials. The available targets and aliases differ accordingly.

When executing the tutorials or modsim template build commands directly in the repository, the WAVES project root repository must be put on PYTHONPATH. In personal (but not shared) virtual environments, the preferred method is to run conda develop . once from the project root directory. See the Conda documentation for more information about “development” mode installs. For shared environments, the preferred solution is to prefix the build commands with PYTHONPATH=.. ``, where it is assumed that the ``PWD is the tutorial or modsim template root directory.

$ pwd
/home/roppenheimer/repos/waves/waves/modsim_template
$ PYTHONPATH=/home/roppenheimer/repos/waves scons -h

Test#

The project regression suite is collected under the regression alias and can be run as

$ scons regression

By default, the system tests are configured to run with permissive skips for missing third-party software. This allows developers to run as much of the system test suite as possible on local systems with incomplete third-party software installations. To force the full execution of the system tests, a pass-through --unconditional-build flag may be added. This forces the documentation build, even if sphinx-build is missing, and passes through pytest to the system tests and tutorials as an SCons CLI option of the same name. The AEA RHEL CI server contains all necessary third-party software, so the CI tests on this server require the full build as

$ scons regression --unconditional-build

The continuous integration server also performs a separate style guide check using flake8 and black with associated aliases or the collector alias style.

$ scons style

If black reports files that should be formatted, the following alias will format files using the same command line options as the check alias. black can also be run directly, but this may miss files without the .py extension, such as SConstruct and SConscript files. After formatting, the developer must review the changes and commit them to their branch.

$ scons black-format

WAVES has dedicated target aliases for the unit and system tests. To run the unit tests, activate a conda environment and run

$ scons pytest

XML and HTML coverage reports are always generated for the pytest alias. The XML output is used to provide coverage visualization on merge requests. The HTML output can be opened with a browser to explore coverage interactively and is uploaded to the internally hosted Gitlab-Pages documentation at: https://aea.re-pages.lanl.gov/python-projects/waves/coverage/index.html.

$ scons pytest
$ ls build/pytest/coverage.xml build/pytest/coverage/index.html
build/pytest/coverage.xml  build/pytest/coverage/index.html
$ find build/pytest -name coverage.xml -o -name index.html
build/pytest/coverage.xml
build/pytest/coverage/index.html

Warning

Since a v0.12.9 pre-release, the pyproject.toml file is templated from pyproject.toml.in to allow distribution name changes during packaging with scons build. The SCons aliases which depend on pyproject.toml will automatically create this templated file as necessary. However, if this file has not yet been generated, it is not possible to run the pytest executable directly. Run scons pyproject.toml or any of the testing aliases before running the test suite directly with the pytest command.

$ scons pyproject.toml
$ pytest -v -n 4 waves/_tests/test_scons_extensions.py

The tutorials and modsim template are run as system tests and require third-party software not available on conda-forge. To run the system tests, install the third-party software and make them available in your PATH, activate a conda environment and run

$ scons systemtest

The full list of continuous integration test commands can be found in the Gitlab-CI files.

$ pwd
path/to/local/git/clone/waves/
$ sed -n '/developer-test/,/tags/p' .gitlab-ci.yml .pipeline-*.yml

The WAVES unit and system tests may also be executed from the installation directory. These tests use pytest directly and may require non-default pytest options and markers to execute the system tests serially. The full list of CI commands may be found in the internal recipe file, recipe-internal/meta.yaml, under the test keyword. The following grep command will show the pytest commands and options.

$ grep "pytest " recipe-internal/*

Test Local Package#

When testing CLI changes locally, the waves module must be run as a script. We must also set the PYTHONPATH in order to include the current waves module when operating on a configuration that imports waves.

Below is an example of a visualization test of an SConstruct file using the local waves module.

$ pwd
path/to/local/git/clone/waves/
$ PYTHONPATH=$PWD python -m waves._main visualize . --sconstruct /path/to/local/SConstruct

Documentation#

The documentation build is automated with SCons as the documentation target. The HTML documentation builds to waves/build/docs/html/index.html

Build the WAVES documentation

$ pwd
path/to/local/git/clone/waves/
$ scons documentation

Build the MODSIM-TEMPLATE documentation. The WAVES package must be on PYTHONPATH. For developers, the least disruptive solution is a per-command modification of PYTHONPATH.
```
$ pwd
path/to/local/git/clone/waves/modsim_template
$ PYTHONPATH=/path/to/local/git/clone/waves:$PYTHONPATH scons documentation
```

Windows users#

This project uses symbolic links to minimize duplication of files where possible. Some files, such as the tutorial and modsim template package files, can not be shared in common due to their applications and the necessary directory structure for each. However, if the file content is identical, a symbolic link is used to avoid duplicating the entire file.

Symbolic links require special handling on Windows computers. If contributors are developing from a Windows machine, they are encouraged to

Read about mklink and “developer mode” for Windows 10/11
Use an up-to-date version of git

Use one of the following git configurations

# Global configuration. Run from anywhere.
> git config --global core.symlinks true

# Local configuration. Run from repository root directory after cloning.
> git config core.symlinks true

Use unix line endings with one of the following git configurations

# Global configuration. Run from anywhere.
> git config --global core.autocrlf true

# Local configuration. Run from repository root directory after cloning.
> git config core.autocrlf true

Gitlab-CI#

AEA RHEL#

There are several known AEA RHEL CI fragile false-negative failure mechanisms. If one of these failures is observed, try re-running the CI job without change.

Conda environment creation fails in conda-build jobs. The shared conda command sometimes interferes with itself while running simultaneous conda-build jobs.

HPC#

While the system tests are run with an HPC CI pipeline, the machine and runner uptime and system test runtime are considered too fragile for reliable use. The AEA RHEL system tests pass in less than 8 minutes, but the HPC system test job has been observed to complete in under 15 minutes or time out after an hour. The HPC CI system tests are always run for merge-requests and scheduled pipelines, but the pass/fail status is not used to mark CI tests as failing. Developers are encouraged to check the HPC system test results, but not required to address failing HPC CI pipelines.

The HPC CI jobs usually run in 15-25 minutes when it runs successfully.

There are several known HPC CI fragile false-negative failure mechanisms. If one of these failures is observed, try re-running the CI job without change.

Timeout at 1 hour

Windows#

While the system tests are run with a Windows CI pipeline, the Windows CI server does not have the full suite of third-party software required to run the full system tests. The pipeline is always run for merge-requests and scheduled pipelines, but it is considered experimental and the pass/fail status is not used to mark CI tests as failing. Developers are encouraged to check the Windows test results, but not required to address failing Windows CI pipelines.

The Windows CI jobs usually run in under 6 minutes.

There are several known Windows CI fragile false-negative failure mechanisms. If one of these failures is observed, try re-running the CI job without change.

Incorrect or imcomplete CI environment build: Script file 'C:\ProgramData\anaconda3\envs\waves-dev\Scripts\sphinx-build-script.py' is not present.

MacOS#

While the system tests are run with a MacOS CI pipeline, the MacOS CI server does not have the full suite of third-party software required to run the full system tests. The pipeline is always run for merge-requests and scheduled pipelines, but it is considered experimental and the pass/fail status is not used to mark CI tests as failing. Developers are encouraged to check the MacOS test results, but not required to address failing MacOS CI pipelines.

The MacOS CI jobs usually run in under 3 minutes.