Model does not restart reproducibly in debug mode when initially starting from GFS initial conditions #381
Comments
spencerkclark
added a commit
that referenced
this issue
Sep 7, 2023
This PR refactors the build infrastructure in this repo to eliminate the need for the Docker component. All development and testing is now done in the `nix` shell. This should be a quality of life improvement for anyone developing the fortran model, as it no longer requires maintaining checksums in two separate build environments. In so doing it introduces the following changes: - New `make` rules are provided for compiling the model in different modes: - `build` -- build executables in `repro` (our production mode) and `debug` mode. - `build_repro` -- build only the `repro` mode executable. - `build_debug` -- build only the `debug` mode executable. - Tests are run with each of the executables available in the local `bin` directory, and are tagged with the associated compile mode. - An option, `check_layout_invariance`, is provided to trigger regression tests be run with a 1x2 domain decomposition instead of a 1x1 domain decomposition to check invariance to the domain decomposition layout; this is used for the all the coarse-graining regression tests and replaces the previous `test_run_reproduces_across_layouts` test that would run in the docker image. - `debug`-mode and `repro`-mode simulations produce different answers, which is something we noticed in #364 when upgrading compiler versions as well, and so require different reference checksums. In working on this PR, we ran the fortran model in `debug` mode in more contexts than we had previously, some of which turned up errors, which we currently work around by using `pytest.skip` (something we had implicitly already been doing before): - #365 - #381 Working on this PR also brought my attention to the fact that `pytest`'s `tmpdir` fixture does not automatically get cleaned up after each test; `pytest` versions older than 7.3.0 keep around directories from the last three runs of `pytest`, which fill up disk space quickly since running these tests requires creating 10's of run directories, each with their own initial conditions and input files (#380). For the time being I manually clean up these run directories after successful tests. Resolves #340.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I encountered this issue in #379 when trying to run the
test_restart_reproducibilitytest with the fortran model compiled in debug mode. For the time being I will simply skip these tests, but we may at some point look into the cause. A clue is that these tests pass if I remove the-finit-logical=truefrom theFFLAGS_DEBUGvariable in the configure file:It is not clear if this is something we should be worried about, since this restart reproducibility test passes without this flag (including in repro mode, which is the mode in which we do all of our production runs). For what it is worth, this flag is also not included in the debug mode flags of SHiELD (see the flags in the template here).
Another notable aspect of this issue is that this only happens for simulations started from GFS initial conditions (e.g. the
default.ymlconfig file); for therestart.ymlconfig, this restart reproducibility test always passes (whether we run in debug mode or not, whether we use the-finit-logical=trueflag in debug mode or not).The text was updated successfully, but these errors were encountered: