Tutorial: Quinoa

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

Warning

Most WAVES tutorials are used as system tests in the regression test suite to ensure that the tutorial files are up-to-date and functional. While the Quinoa team refactors around a breaking change in the input file parser, the Quiona tutorial is not part of the regression suite, but the Quinoa builder unit tests are part of the verification suite. If you run into problems running this tutorial, please contact the Quiona development team for the current status of the input file parser.

Warning

Most WAVES tutorials are used as system tests in the regression test suite to ensure that the tutorial files are up-to-date and functional. The quinoa-local alias without sbatch submission is part of the regression suite, but neither the sbatch submission nor the quinoa-remote alias is regression tested. The SSH remote excution and sbatch behavior was tested when the tutorial was written and is expected to run correctly. If you run into problems running this tutorial, please contact the WAVES development team.

References

• Cubit: Python Interface [57]

• Cubit: Importing Cubit into Python [57]

• Quinoa: Quinoa documentation [70, 71]

• Charm++ [72]

Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest
   

2. Activate the environment

   $ conda activate waves-tutorial-env
   

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

4. Create a new tutorial_quinoa directory with the waves fetch command below

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --destination tutorial_quinoa tutorials/tutorial_quinoa
$ ls tutorial_quinoa
box.py flow.lua SConstruct SConscript

5. Make the new tutorial_quinoa directory the current working directory

$ pwd
/home/roppenheimer/waves-tutorials
$ cd tutorial_quinoa
$ pwd
/home/roppenheimer/waves-tutorials/tutorial_quinoa
$ ls
box.py flow.lua SConstruct SConscript

SConscript

6. Review the SConscript file.

The structure has changed enough from the core tutorials that a diff view is not as useful. Instead the contents of the new SConscript files are duplicated below.

waves-tutorials/tutorial_quinoa/SConscript

#! /usr/bin/env python
import pathlib

import waves


Import("env", "envQuinoa")

# TODO: move parameters to a dedicated dictionary file read
simulation_variables = {
    "xlength": 1.0,
    "ylength": 0.5,
    "zlength": 0.5,
}

workflow = []
remote_directory_prefix = env["remote_directory"]

# Geometry/partition/mesh
# TODO: separate the local and remote workflows into dedicated SConscript files that re-use the cubit workflow as in the
# Cubit+Abaqus and Cubit+Sierra tutorials
mesh = env.PythonScript(
    target=["box.cub", "box.exo"],
    source=["box.py"],
    subcommand_options="--output-file ${TARGET.abspath} --xlength ${xlength} --ylength ${ylength} --zlength ${zlength}",
    **simulation_variables,
)
workflow.extend(mesh)
env.Alias("mesh", mesh)

# SSH remote build on HPC
remote_directory = remote_directory_prefix / "remote"
remote_target = env.SSHQuinoaSolver(
    target=["remote.stdout"],
    source=["flow.lua", "box.exo"],
    charmrun="charmrun",
    inciter="inciter",
    remote_directory=remote_directory,
)
env.Alias("quinoa-remote", remote_target)

# Local build on local server
local_target = envQuinoa.QuinoaSolver(
    target=["local.stdout"],
    source=["flow.lua", "box.exo"],
    charmrun=envQuinoa["charmrun"],
    inciter=envQuinoa["inciter"],
)
envQuinoa.Alias("quinoa-local", local_target)

if not env["unconditional_build"] and not envQuinoa["inciter"]:
    print(f"Program 'inciter' was not found in construction environment. Ignoring 'quinoa-local' target(s)")
    Ignore([".", "quinoa-local"], local_target)

Cubit Journal Files

7. Review the box.py Cubit journal file

The Cubit journal file includes a similar CLI introduced in Tutorial 02: Partition and Mesh for the Abaqus journal files. Besides the differences in Abaqus and Cubit commands, the major difference between the Abaqus and Cubit journal files is the opportunity to use Python 3 with Cubit, where Abaqus journal files must use the Abaqus controlled installation of Python 2. Unlike the core tutorials, this journal file only parameterizes the geometry, so intermediate partition and mesh journal files are not as useful and the journal file performs all operations at once.

waves-tutorials/tutorial_quinoa/box.py

import sys
import argparse
import pathlib

import cubit


def main(output_file, xlength, ylength, zlength):
    """Write brick geometry with specified dimensions

    :param str output_file: The output file for the Cubit model. Will be stripped of the extension and ``.cub`` and
        ``.exo`` will be used for the model and mesh files, respectively.
    :param xlength: box edge length on global x axis
    :param ylength: box edge length on global y axis
    :param zlength: box edge length on global z axis

    :returns: writes ``output_file``.cub and ``output_file``.exo
    """
    cubit.init(["cubit", "-noecho", "-nojournal", "-nographics", "-batch"])
    cubit.cmd("new")
    cubit.cmd("reset")

    cubit_file = pathlib.Path(output_file).with_suffix(".cub")
    exodus_file = cubit_file.with_suffix(".exo")

    # Geometry
    cubit.cmd(f"brick x {xlength} y {ylength} z {zlength}")
    cubit.cmd(f"move volume 1 x {xlength * 0.5} y {ylength * 0.5} z {zlength * 0.5}")
    cubit.cmd(f"webcut volume 1 with plane xplane offset {xlength * 0.5} noimprint nomerge")
    cubit.cmd(f"merge volume 1 2")

    # Sets (partition)
    cubit.cmd("sideset 1 add surface 8 15")
    cubit.cmd("sideset 2 add surface 9 16")
    cubit.cmd("sideset 3 add surface 10 14")
    cubit.cmd("sideset 4 add surface 4")
    cubit.cmd("sideset 5 add surface 11 13")
    cubit.cmd("sideset 6 add surface 6")

    # Mesh
    cubit.cmd("Trimesher surface gradation 1.05")
    cubit.cmd("Trimesher volume gradation 1.05")

    cubit.cmd("surface 6 7 8 9 10 11 13 14 15 16 Scheme TriMesh geometry approximation angle 10")
    cubit.cmd("surface 13 14 15 16  size 0.065")
    cubit.cmd("surface 4 size 0.09")
    cubit.cmd("surface 7 size .09")
    cubit.cmd("surface 8 9 10 11 size 0.45")
    cubit.cmd("surface 6 size 0.09")

    cubit.cmd("mesh surface 6 7 8 9 10 11 13 14 15 16")

    cubit.cmd("volume 1 2 Scheme Tetmesh proximity layers off geometry approximation angle 10")
    cubit.cmd("volume 1 2 Tetmesh growth_factor 1.001")
    cubit.cmd("volume 1 size 0.065")
    cubit.cmd("volume 2 size 0.065")

    cubit.cmd("mesh volume 1 2")
    cubit.cmd("block 1 add volume 1 2")

    cubit.cmd("set exodus netcdf4 off")
    cubit.cmd("set large exodus file on")
    cubit.cmd(f"export Genesis '{exodus_file}' dimension 3 block 1  overwrite")
    cubit.cmd(f"save as '{cubit_file}' overwrite")


def get_parser():
    """Return the command line parser"""
    script_name = pathlib.Path(__file__)
    prog = f"python {script_name.name} "
    cli_description = "Write brick geometry ``output_file``.cub and ``output_file``.exo with specified dimensions"
    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
    parser.add_argument(
        "--output-file",
        type=str,
        required=True,
        # fmt: off
        help="The output file for the Cubit model. Will be stripped of the extension and ``.cub`` and ``.exo`` will "
             "be used for the model and mesh files, respectively.",
        # fmt: on
    )
    parser.add_argument("--xlength", type=float, required=True, help="box edge length on global x axis")
    parser.add_argument("--ylength", type=float, required=True, help="box edge length on global y axis")
    parser.add_argument("--zlength", type=float, required=True, help="box edge length on global z axis")
    return parser


if __name__ == "__main__":
    parser = get_parser()
    args = parser.parse_args()
    main(
        output_file=args.output_file,
        xlength=args.xlength,
        ylength=args.ylength,
        zlength=args.zlength,
    )

Quinoa Input File(s)

8. Create or review the Quinoa input file from the contents below

waves-tutorials/tutorial_quinoa/flow.lua

inciter = {

  title = "Flow in a simple box" ,

  term = 0.11,  -- Max physical time, s
  ttyi= 10,  -- TTY output interval
  cfl =0.8,
  scheme = "alecg",

  partitioning = "rcb",

  compflow = {
    physics ="euler", 
  },
    -- Units: m,kg,s,
  ic = {
    density = 0.912,  -- kg/m^3
    velocity={ 0.0, 0.0, 0.0 },  -- m/s
    pressure = 77816.5,  -- Initiate linear
  },

  material = {
    {
      gamma = { 1.4 },
      cv ={ 717.5 },
    }
  },

  bc = { 
    {
      farfield = {6},  -- Outlet
      pressure = 77816.5,
      density = 0.912,
      velocity = {0.0, 0.0, 0.0},
      timedep = { 
	{
          sideset = {4}, --Inlet
	  fn = {  -- t  p  rho  xvel yvel zvel
            0.0, 77816.5, 0.912, 0.000, 0.000, 0.000,
            0.1, 77816.5, 0.912, 5.000, 0.000, 0.000,
           10.0, 77816.5, 0.912, 5.000, 0.000, 0.000,
	  }
	}
      },
      symmetry = {1,2,3,5},
    }
  },


  history_output = {
    interval = 1,
    point = {
      {id = "p1", coord = { 0.5, 0.25, 0.25 }},
    }
  },


  field_output = {
    time_interval = 0.001,
    nodevar = {
      "density",
      "x-velocity",
      "y-velocity",
      "z-velocity",
      "specific_total_energy",
      "pressure"
    }
  },
 

  diagnostics = {
    interval= 250,
    format  = "scientific",
    error   =  "l2"
  }

}

SConstruct

Note that Quinoa requires a separate construction environment from the launching Conda environment. This is because the Quinoa compiler/linker and openmpi library conflict with the launching Conda environment. You may need to update the Quinoa activation shell command according to the instructions on your local system.

A diff against the SConstruct file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_quinoa/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorial_quinoa_SConstruct
@@ -1,7 +1,9 @@
 #! /usr/bin/env python
 
 import os
+import getpass
 import pathlib
+import subprocess
 
 import waves
 
@@ -33,20 +35,19 @@
     help="Print task *.stdout target file(s) on build failures. (default: '%default')",
 )
 # Python optparse appends to the default list instead of overriding. Must implement default/override ourselves.
-default_abaqus_commands = [
-    "/apps/abaqus/Commands/abq2024",
-    "/usr/projects/ea/abaqus/Commands/abq2024",
-    "abq2024",
-    "abaqus",
+default_cubit_commands = [
+    "/apps/Cubit-16.12/cubit",
+    "/usr/projects/ea/Cubit/Cubit-16.12/cubit",
+    "cubit",
 ]
 AddOption(
-    "--abaqus-command",
-    dest="abaqus_command",
+    "--cubit-command",
+    dest="cubit_command",
     nargs=1,
     type="string",
     action="append",
     metavar="COMMAND",
-    help=f"Override for the Abaqus command. Repeat to specify more than one (default: {default_abaqus_commands})",
+    help=f"Override for the Cubit command. Repeat to specify more than one (default: {default_cubit_commands})",
 )
 
 # Comments used in tutorial code snippets: marker-2
@@ -57,18 +58,31 @@
     variant_dir_base=pathlib.Path(GetOption("variant_dir_base")),
     unconditional_build=GetOption("unconditional_build"),
     print_build_failures=GetOption("print_build_failures"),
-    abaqus_commands=GetOption("abaqus_command"),
+    cubit_commands=GetOption("cubit_command"),
 )
 
 # Conditionally print failed task *.stdout files
 env.PrintBuildFailures(print_stdout=env["print_build_failures"])
 
+# Always copy (no sym-links) when duplicating
+env.SetOption("duplicate", "copy")
+
 # Comments used in tutorial code snippets: marker-3
 
 # Find required programs for conditional target ignoring and absolute path for use in target actions
-env["ABAQUS_PROGRAM"] = env.AddProgram(
-    env["abaqus_commands"] if env["abaqus_commands"] is not None else default_abaqus_commands
+env["CUBIT_PROGRAM"] = env.AddCubit(
+    env["cubit_commands"] if env["cubit_commands"] is not None else default_cubit_commands
 )
+env["SBATCH_PROGRAM"] = env.AddProgram(["sbatch"])
+env["user"] = getpass.getuser()
+
+# Try a preference ordered list of project server-specific environment modulefiles
+quinoa_hpc_modulefile = pathlib.Path("/usr/projects/ea/Quinoa/modules/modulefiles/quinoa")
+quinoa_modulefile_options = [
+    pathlib.Path("/projects/aea_compute/modulefiles/aea-quinoa"),
+    quinoa_hpc_modulefile,
+]
+quinoa_modulefile = next((path for path in quinoa_modulefile_options if path.exists()), None)
 
 # Comments used in tutorial code snippets: marker-4
 
@@ -80,30 +94,53 @@
     "project_name": project_name,
     "project_dir": project_dir,
     "version": version,
+    "remote_directory": pathlib.Path(f"/users/{env['user']}/WAVES-TUTORIAL/tutorial_quinoa"),
 }
 for key, value in project_variables.items():
     env[key] = value
 
-# Make the project package importable for Python and Abaqus Python environments
-env.PrependENVPath("PYTHONPATH", project_dir)
-
 # Comments used in tutorial code snippets: marker-5
 
 # Add builders and pseudo-builders
-env.Append(BUILDERS={})
+env.Append(
+    BUILDERS={
+        "SSHQuinoaSolver": waves.scons_extensions.ssh_builder_actions(
+            waves.scons_extensions.sbatch_quinoa_builder_factory(
+                environment=f"module use {quinoa_hpc_modulefile.parent} && module load {quinoa_hpc_modulefile.name} &&"
+            ),
+            remote_server="ro-rfe.lanl.gov",
+        )
+    }
+)
+
+# Source the Quinoa environment and store it separately from the Conda/Cubit environment to avoid PATH interference
+if quinoa_modulefile:
+    envQuinoa = waves.scons_extensions.shell_environment(
+        "module unload aea-nightly aea-weekly aea-quarterly aea-release && "
+        f"module use {quinoa_modulefile.parent} && "
+        f"module load {quinoa_modulefile.name}"
+    )
+else:
+    envQuinoa = Environment()
+envQuinoa.AddMethod(waves.scons_extensions.add_program, "AddProgram")
+envQuinoa["inciter"] = envQuinoa.AddProgram(["inciter"])
+envQuinoa["charmrun"] = envQuinoa.AddProgram(["charmrun"])
+
+# Add WAVES builders. If sbatch is found, use it.
+# Should allow the same scons alias to run directly on sstbigbird, but submit as an sbatch job on HPC
+quinoa_builder_factory = waves.scons_extensions.quinoa_builder_factory
+if env["SBATCH_PROGRAM"] is not None:
+    quinoa_builder_factory = waves.scons_extensions.sbatch_quinoa_builder_factory
+envQuinoa.Append(
+    BUILDERS={"QuinoaSolver": quinoa_builder_factory(program=envQuinoa["charmrun"], subcommand=envQuinoa["inciter"])}
+)
 
 # Comments used in tutorial code snippets: marker-6
 
 # Add simulation targets
-workflow_configurations = [
-    "tutorial_01_geometry",
-    "tutorial_02_partition_mesh",
-    "tutorial_03_solverprep",
-    "tutorial_04_simulation",
-]
-for workflow in workflow_configurations:
-    build_dir = env["variant_dir_base"] / workflow
-    SConscript(workflow, variant_dir=build_dir, exports={"env": env}, duplicate=False)
+SConscript(
+    "SConscript", variant_dir=env["variant_dir_base"], exports={"env": env, "envQuinoa": envQuinoa}, duplicate=True
+)
 
 # Comments used in tutorial code snippets: marker-7
 

Note that the Cubit Python file doesn’t perform any imports from the current modsim project package, so the PYTHONPATH modification is no longer required. This tutorial is created in a new, stand-alone subdirectory, so the previous tutorial workflow configurations are no longer available. Only the quinoa-local and quinoa-remote workflow configurations will be found by SCons at execution time.

Build Targets

9. Build the new targets

$ pwd
/home/roppenheimer/waves-tutorials/tutorial_quinoa
scons: Reading SConscript files ...
Checking whether '/apps/Cubit-16.12/cubit' program exists.../apps/Cubit-16.12/cubit
Checking whether 'cubit' program exists...no
Checking whether 'sbatch' program exists...no
Sourcing the shell environment with command 'module use /projects/aea_compute/modulefiles && module load aea-quinoa' ...
Checking whether 'inciter' program exists.../projects/aea_compute/aea-quinoa/bin/inciter
Checking whether '/users/cclong/QUINOA/quinoa/buildOS/Main/inciter' program exists...no
Checking whether 'charmrun' program exists.../projects/aea_compute/aea-quinoa/bin/charmrun
Checking whether '/users/cclong/QUINOA/quinoa/buildOS/Main/charmrun' program exists...no
scons: done reading SConscript files.
scons: Building targets ...
cd /home/roppenheimer/waves-tutorials/tutorial_quinoa/build && python /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.py --output-file /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.cub --xlength 1.0 --ylength 0.5 --zlength 0.5 > /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.cub.stdout 2>&1
cd /home/roppenheimer/waves-tutorials/tutorial_quinoa/build && /projects/aea_compute/aea-quinoa/bin/charmrun +p1 /projects/aea_compute/aea-quinoa/bin/inciter --control /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/flow.lua --input /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.exo > /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/local.stdout 2>&1
scons: done building targets.

Output Files

Explore the contents of the build directory using the tree command against the build directory, as shown below.

$ pwd
/home/roppenheimer/waves-tutorials/tutorial_quinoa
$ tree build/
build
|-- SConscript
|-- box.cub
|-- box.cub.stdout
|-- box.exo
|-- box.py
|-- diag
|-- flow.lua
|-- flow.q
|-- inciter_input.log
|-- inciter_screen.log
|-- local.stdout
|-- mesh_edge_pdf.0.txt
|-- mesh_ntet_pdf.0.txt
|-- mesh_vol_pdf.0.txt
|-- out.e-s.0.1.0
`-- restart
    |-- MainChares.dat
    |-- RO.dat
    `-- sub0
        |-- Chares_0.dat
        |-- Groups_0.dat
        |-- NodeGroups_0.dat
        `-- arr_0.dat

2 directories, 21 files