Cloud-J v7.7.2 in GEOS-Chem 14.4.2 results in a large slowdown

[yantosca]

Your name

Bob Yantosca

Your affiliation

Harvard + GCST

What happened? What did you expect to happen?

We have noticed that the Cloud-J photolysis results in a large slowdown in GEOS-Chem 14.4.2, and in 14.5.0-alpha.2 and 14.5.0-alpha.4 (which were added after 14.4.2 was released).

We think this might be related to the implementation of error checks in Cloud-J, which return an error code up to the main program. @lizziel is investigating.

What are the steps to reproduce the bug?

1. Clone GEOS-Chem 14.4.2 (in main branch) as described at geos-chem.readthedocs.io
2. Create a run directory for the fullchem_benchmark simulation (see geos-chem.readthedocs.io)
3. Compile and run (see geos-chem.readthedocs.io)
4. Turn on gcclassic_timers output in the geoschem_config.yml file

Please attach any relevant configuration and log files.

This is the benchmark timing output between 14.4.1 and 14.4.2. We had omitted to notice the large timing increase at the time.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% GEOS-Chem Classic Benchmark Timing Information
%%%
%%% Ref = gcc-4x5-1Mon-14.5.0-alpha.1
%%% Dev = gcc-4x5-1Mon-14.5.0-alpha.2
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Timer                              Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
GEOS-Chem                        20613.988           30749.413      4.917e+01
HEMCO                             2220.961            2303.218      3.704e+00
All chemistry                     7626.508           17515.298      1.297e+02
=> Gas-phase chem                 4914.933            4942.063      5.520e-01
=> Photolysis                      667.263           10010.823      1.400e+03
=> Aerosol chem                   1548.158            2062.317      3.321e+01
=> Linearized chem                  28.812              29.837      3.558e+00
Transport                         1710.883            1697.413     -7.873e-01
Convection                        2392.341            2392.918      2.412e-02
Boundary layer mixing             2314.712            2409.030      4.075e+00
Dry deposition                      51.311              52.113      1.563e+00
Wet deposition                     746.941             755.283      1.117e+00
Diagnostics                       2364.631            2430.962      2.805e+00
Unit conversions                  1254.879            1262.640      6.185e-01

What GEOS-Chem version were you using?

14.4.2

What environment were you running GEOS-Chem on?

Local cluster

What compiler and version were you using?

gcc 10.2.0

Will you be addressing this bug yourself?

Yes, but I will need some help

In what configuration were you running GEOS-Chem?

GCClassic

What simulation were you running?

Full chemistry

As what resolution were you running GEOS-Chem?

4x5

What meterology fields did you use?

GEOS-FP

Additional information

14.4.2 benchmark plots:

• https://gc-dashboard.org/difference?primary_key=diff-gcc-4x5-1Mon-14.5.0-alpha.1-gcc-4x5-1Mon-14.5.0-alpha.2

14.4.2 benchmark log:

• https://s3.amazonaws.com/benchmarks-cloud/benchmarks/1Mon/gcc/gcc-4x5-1Mon-14.4.2/RunGCC.txt

[yantosca]

Also see PR geoschem/gcpy#332, which adds code to GCPy to flag large differences in benchmark timer table output.

[yantosca]

The performance issue was fixed by @lizziel in Cloud-J version 7.7.3, which will now ship with GEOS-Chem version 14.4.3. We can now close out this issue.