Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Now flag GEOS-Chem Classic/GCHP timers that differ by more than 10% in the benchmark timing tables #332

Merged
merged 2 commits into from
Aug 19, 2024
Merged

Now flag GEOS-Chem Classic/GCHP timers that differ by more than 10% in the benchmark timing tables #332

merged 2 commits into from
Aug 19, 2024

Conversation

yantosca
Copy link
Contributor

@yantosca yantosca commented Aug 12, 2024 •
edited
Loading

Name and Institution (Required)

Name: Bob Yantosca
Institution: Harvard + GCST

Describe the update

We have updated the benchmark_scrape_gcclassic_timers.py and benchmark_scrape_gchp_timers.py scripts to flag individual GEOS-Chem Classic and GCHP timers that vary by more than +/- 10%. This will help us to identify performance bottlenecks much more easily.

Expected changes

Benchmark timing tables now will contain an asterisk after timers that vary by more than +/- 10%.

GEOS-Chem Classic:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% GEOS-Chem Classic Benchmark Timing Information
%%%
%%% Ref = gcc-14.4.2
%%% Dev = gcc-14.4.1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Timer                              Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
GEOS-Chem                        28458.415           21173.062     -2.560e+01 *
HEMCO                             2307.501            2287.382     -8.719e-01
All chemistry                    15052.359            8087.397     -4.627e+01 *
=> Gas-phase chem                 5536.890            5295.887     -4.353e+00
=> Photolysis                     7353.315             691.712     -9.059e+01 *
=> Aerosol chem                   1597.813            1598.909      6.859e-02
=> Linearized chem                  30.455              29.045     -4.630e+00
Transport                         1689.448            1660.802     -1.696e+00
Convection                        2490.201            2413.583     -3.077e+00
Boundary layer mixing             2403.692            2271.712     -5.491e+00
Dry deposition                      51.891              51.624     -5.145e-01
Wet deposition                     774.040             755.015     -2.458e+00
Diagnostics                       2367.490            2446.812      3.350e+00
Unit conversions                  1422.235            1267.408     -1.089e+01 *

^^^ note: Ref and Dev were swapped in this output inadvertently, but you get the idea.

GCHP

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% GCHP Benchmark Timing Information
%%%
%%% Ref = gchp-14.4.2
%%% Dev = gchp-14.4.1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


GCHPchem Timer                     Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
GCHPchem                          7017.550            7032.760      2.167e-01
--SetService                         0.280               0.270     -3.571e+00
----generic                          0.000               0.000            nan
--Initialize                        30.660              30.090     -1.859e+00
----INITIALIZE                      30.660              30.090     -1.859e+00
------generic                       12.930              13.140      1.624e+00
--Record                             0.020               0.020      0.000e+00
----generic                          0.010               0.010      0.000e+00
--Run                             6947.260            6981.810      4.973e-01
----GenRunMine                    6947.220            6981.760      4.972e-01
------RUN                         6907.250            6936.510      4.236e-01
--------CP_BFRE                      3.070               3.360      9.446e+00
--------DO_CHEM                   6869.460            6897.840      4.131e-01
----------GC_CONV                  523.710             501.900     -4.165e+00
----------GC_DRYDEP                  8.800               8.770     -3.409e-01
----------GC_EMIS                  184.410             186.890      1.345e+00
----------GC_TURB                  619.830             595.020     -4.003e+00
----------GC_CHEM                 4736.540            4751.700      3.201e-01
----------GC_WETDEP                445.050             468.660      5.305e+00
----------GC_DIAGN                  36.980              37.190      5.679e-01
--------CP_AFTR                      0.010               0.010      0.000e+00
--Finalize                           3.220               3.120     -3.106e+00
----generic                          3.210               3.120     -2.804e+00


Summary                            Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
All                              25606.259           25584.746     -8.401e-02
--SetService                         1.522               1.430     -6.045e+00
----GCHP                             1.499               1.407     -6.137e+00
------GCHPctmEnv                     0.001               0.001      0.000e+00
------GCHPchem                       0.456               0.450     -1.316e+00
------DYNAMICS                       0.981               0.896     -8.665e+00
----HIST                             0.000               0.000            nan
----EXTDATA                          0.000               0.000            nan
--Initialize                        98.315             104.915      6.713e+00
----GCHP                            69.100              75.827      9.735e+00
------GCHPctmEnv                     0.013               0.013      0.000e+00
------GCHPchem                      61.798              65.188      5.486e+00
------DYNAMICS                       5.680               9.107      6.033e+01 *
----HIST                             7.027               7.170      2.035e+00
----EXTDATA                         21.023              21.338      1.498e+00
--Run                            25502.983           25475.018     -1.097e-01
----EXTDATA                       1067.898            1074.784      6.448e-01
----GCHP                         17065.536           17103.029      2.197e-01
------GCHPctmEnv                    44.852              45.314      1.030e+00
------GCHPchem                   11228.451           11255.168      2.379e-01
------DYNAMICS                    5790.771            5801.109      1.785e-01
----HIST                           142.520             145.059      1.782e+00
--Finalize                           3.292               3.189     -3.129e+00
----GCHP                             3.272               3.175     -2.965e+00
------GCHPctmEnv                     0.005               0.006      2.000e+01 *
------GCHPchem                       3.261               3.163     -3.005e+00
------DYNAMICS                       0.003               0.003      0.000e+00
----HIST                             0.007               0.006     -1.429e+01 *
----EXTDATA                          0.010               0.005     -5.000e+01 *

Reference(s)

If this is a science update, please provide a literature citation.

Related Github Issue

@yantosca
Flag significant differences in benchmark timing tables
4b2762b 
gcpy/benchmark/modules/benchmark_scrape_gcclassic_timers.py
gcpy/benchmark/modules/benchmark_scrape_gchp_timers.py
- Flag timers (with " *") that vary by more than +/- 10% in the
  benchmark timing table output

CHANGELOG.md
- Updated accordingly
@yantosca yantosca added topic: Benchmark Plots and Tables Issues pertaining to generating plots/tables from benchmark output category: Bug Fix Fixes a bug that was previously reported labels Aug 12, 2024
@yantosca yantosca added this to the 1.5.1 milestone Aug 12, 2024
@yantosca yantosca requested a review from lizziel August 12, 2024 18:25
@yantosca yantosca self-assigned this Aug 12, 2024
@yantosca yantosca changed the base branch from main to dev August 12, 2024 18:46
@yantosca yantosca mentioned this pull request Aug 12, 2024
Closed
@lizziel
Copy link
Contributor

lizziel commented Aug 13, 2024

This updates looks good but I'm curious why we use scientific notation for the % diff. It would be easier to scan, and might not even need the *s, if we list the percentages plus maybe just one decimal place.

@lizziel
Copy link
Contributor

lizziel commented Aug 13, 2024

Having a column for difference would also be good to have.

@yantosca Octobox
Copy link
Contributor Author

Thanks @lizziel. I can implement a floating point format rather than the scientific notation for % diff.

@yantosca
Now use floating-point for % diff in benchmark timing tables
b07327d 
gcpy/benchmark/modules/benchmark_gcclassic_scrape_timers.py
gcpy/benchmark/modules/benchmark_gchp_scrape_timers.py
- Changed the "% diff" column from exponential format (12.3e) to
  floating-point format (12.3f) to make it easier to see large
  differences

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor Author

@lizziel: I reverted to floating point format in commit b07327d.
See updated output below:

GCClassic

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% GEOS-Chem Classic Benchmark Timing Information
%%%
%%% Ref = gcc-14.4.1
%%% Dev = gcc-14.4.2
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Timer                              Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
GEOS-Chem                        21173.062           28458.415         34.409 *
HEMCO                             2287.382            2307.501          0.880
All chemistry                     8087.397           15052.359         86.121 *
=> Gas-phase chem                 5295.887            5536.890          4.551
=> Photolysis                      691.712            7353.315        963.060 *
=> Aerosol chem                   1598.909            1597.813         -0.069
=> Linearized chem                  29.045              30.455          4.855
Transport                         1660.802            1689.448          1.725
Convection                        2413.583            2490.201          3.174
Boundary layer mixing             2271.712            2403.692          5.810
Dry deposition                      51.624              51.891          0.517
Wet deposition                     755.015             774.040          2.520
Diagnostics                       2446.812            2367.490         -3.242
Unit conversions                  1267.408            1422.235         12.216 *

GCHP

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% GCHP Benchmark Timing Information
%%%
%%% Ref = gchp-14.4.1
%%% Dev = gchp-14.4.2
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


GCHPchem Timer                     Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
GCHPchem                          7032.760            7017.550         -0.216
--SetService                         0.270               0.280          3.704
----generic                          0.000               0.000            nan
--Initialize                        30.090              30.660          1.894
----INITIALIZE                      30.090              30.660          1.894
------generic                       13.140              12.930         -1.598
--Record                             0.020               0.020          0.000
----generic                          0.010               0.010          0.000
--Run                             6981.810            6947.260         -0.495
----GenRunMine                    6981.760            6947.220         -0.495
------RUN                         6936.510            6907.250         -0.422
--------CP_BFRE                      3.360               3.070         -8.631
--------DO_CHEM                   6897.840            6869.460         -0.411
----------GC_CONV                  501.900             523.710          4.345
----------GC_DRYDEP                  8.770               8.800          0.342
----------GC_EMIS                  186.890             184.410         -1.327
----------GC_TURB                  595.020             619.830          4.170
----------GC_CHEM                 4751.700            4736.540         -0.319
----------GC_WETDEP                468.660             445.050         -5.038
----------GC_DIAGN                  37.190              36.980         -0.565
--------CP_AFTR                      0.010               0.010          0.000
--Finalize                           3.120               3.220          3.205
----generic                          3.120               3.210          2.885


Summary                            Ref [s]             Dev [s]         % Diff
-------------------------------------------------------------------------------
All                              25584.746           25606.259          0.084
--SetService                         1.430               1.522          6.434
----GCHP                             1.407               1.499          6.539
------GCHPctmEnv                     0.001               0.001          0.000
------GCHPchem                       0.450               0.456          1.333
------DYNAMICS                       0.896               0.981          9.487
----HIST                             0.000               0.000            nan
----EXTDATA                          0.000               0.000            nan
--Initialize                       104.915              98.315         -6.291
----GCHP                            75.827              69.100         -8.872
------GCHPctmEnv                     0.013               0.013          0.000
------GCHPchem                      65.188              61.798         -5.200
------DYNAMICS                       9.107               5.680        -37.630 *
----HIST                             7.170               7.027         -1.994
----EXTDATA                         21.338              21.023         -1.476
--Run                            25475.018           25502.983          0.110
----EXTDATA                       1074.784            1067.898         -0.641
----GCHP                         17103.029           17065.536         -0.219
------GCHPctmEnv                    45.314              44.852         -1.020
------GCHPchem                   11255.168           11228.451         -0.237
------DYNAMICS                    5801.109            5790.771         -0.178
----HIST                           145.059             142.520         -1.750
--Finalize                           3.189               3.292          3.230
----GCHP                             3.175               3.272          3.055
------GCHPctmEnv                     0.006               0.005        -16.667 *
------GCHPchem                       3.163               3.261          3.098
------DYNAMICS                       0.003               0.003          0.000
----HIST                             0.006               0.007         16.667 *
----EXTDATA                          0.005               0.010        100.000 *

@yantosca yantosca merged commit f2fe1fe into dev Aug 19, 2024
14 checks passed
@yantosca yantosca deleted the feature/flag-sig-diffs-in-timer-tables branch August 19, 2024 15:03
@yantosca yantosca modified the milestones: 1.5.1, 1.6.0 Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lizziel lizziel Awaiting requested review from lizziel

Assignees

@yantosca yantosca

Labels
category: Bug Fix Fixes a bug that was previously reported topic: Benchmark Plots and Tables Issues pertaining to generating plots/tables from benchmark output
Projects
None yet
Milestone
1.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@yantosca @lizziel