Add domain level to wave gridded COM path #3137

AntonMFernando-NOAA · 2024-12-03T16:40:08Z

Description

This pull request addresses the organization of wave gridded products within the file structure. Currently, all wave gridded products are placed in products/wave/gridded, without any further categorization. To align with the structure used by other component products (such as grib2), this update introduces a domain/resolution directory level to group output more effectively.
For example, instead of having all products in a single directory, the structure will now include specific directories based on domain and resolution (e.g., products/wave/gridded/arctic.9km, products/wave/gridded/global.0p16). This change improves organization and scalability for wave gridded product outputs.
Resolves Add domain level to wave gridded COM path #2677

Type of change

COM_WAVE_GRID_TMPL updated (or a new template, similar to COM_ATMOS_GRIB_TMP and COM_ATMOS_GRIB_GRID_TMPL) to include a directory level for the domain and resolution
Gridded wave output placed into separate directories based on domain/resolution within products/wave/gridded
Output otherwise remain unchanged

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

CI tests with HERA and HERCULES

Checklist

jobs/JGLOBAL_WAVE_POST_SBS

jobs/JGLOBAL_ARCHIVE

…NOAA/global-workflow into wavegrid_compath

jobs/JGLOBAL_ARCHIVE

jobs/JGLOBAL_WAVE_POST_SBS

AntonMFernando-NOAA

@WalterKolczynski-NOAA @JessicaMeixner-NOAA Did more changes. Let me know if you agree.

ush/wave_domain_grid.sh

scripts/exgfs_wave_post_gridded_sbs.sh

AntonMFernando-NOAA · 2025-01-09T20:37:03Z

scripts/exgfs_wave_prdgen_gridded.sh

+         echo "Directory ${!com_dir} created."
+     else
+         echo "Directory ${!com_dir} already exists."
+     fi


@WalterKolczynski-NOAA I think this code should move to jobs\JGLOBAL_WAVE_PRDGEN_GRIDDED. But, I don't know which var to use to define the COM paths. I know it is $grids. But, $grids is only defined in exgfs_wave_prdgen_gridded.sh, not in any config files. Any idea what to do here?

Agree it should be in the j-job. Move grids to the parm file as GEMPAK_GRIDS, then use that to construct the COM paths in the j-job (other instances of grids in this script will obviously need to be updated as well).

Tagging @AminIlia-NOAA for the prdgen job.

AntonMFernando-NOAA · 2025-01-09T20:42:46Z

parm/archive/gfswave.yaml.j2

@@ -1,20 +1,23 @@
 gfswave:
    {% set head = "gfswave.t" + cycle_HH + "z." %}
+    {% set grdNAME = "global.2p00" %}   


@WalterKolczynski-NOAA Agree?

I think this should be waveGRD, but I would have to look closely to be 100% sure.

@WalterKolczynski-NOAA waveGRD is uglo_100km. When used waveGRD arch job fails.

I think this needs to be variable based on what waveinterpGRD and wavepostGRD are. I think you are hard-coding to specific cases instead of being generic here.

Okay, it should use whatever your new process_grdID function spits out then.

@JessicaMeixner-NOAA waveinterpGRD and wavepostGRD are not environmental variables for arch job. That's why I hard-coded it like that. Any suggestions?

Add wave to the list of config files sourced at the top of archive j-job.

@JessicaMeixner-NOAA One way to tackle this is as @WalterKolczynski-NOAA suggested, update the process_grdID function or add another function to wave_domain_grid_res.sh for arch job.

AntonMFernando-NOAA · 2025-01-09T20:51:25Z

parm/archive/gefs_extracted_wave.yaml.j2

@@ -2,6 +2,7 @@
 {% set cycle_YMD = current_cycle | to_YMD %}
 {% set cycle_YMDH = current_cycle | to_YMDH %}
 {% set head = RUN + "wave.t" + cycle_HH + "z." %}
+{% set grdNAME = "global.5p00" %}   


@WalterKolczynski-NOAA Agree with this?

Tagging @sbanihash for this as well.

This may need to be based on process_grdID as well.

I think this is also incorrectly hard-coded and also should be variables.

scripts/exgfs_wave_post_gridded_sbs.sh

WalterKolczynski-NOAA · 2025-01-10T09:17:24Z

parm/archive/gfswave.yaml.j2

    name: "GFSWAVE"
    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/gfswave.tar"
    required:
        # Wave GRIB2 regional forecast products
+        {% set file_path = COMIN_WAVE_GRID ~ "/" ~ grdNAME %}


COMIN_WAVE_GRID should include the ${GRDNAME}_${GRDRES} level when defined in the j-job.

ush/wave_grib2_sbs.sh

JessicaMeixner-NOAA

Thanks for these updates @AntonMFernando-NOAA !

Please let me know when you're ready for me to do a new test with the multiple output grids.

JessicaMeixner-NOAA · 2025-01-10T13:41:20Z

parm/archive/gefs_extracted_wave.yaml.j2

@@ -2,6 +2,7 @@
 {% set cycle_YMD = current_cycle | to_YMD %}
 {% set cycle_YMDH = current_cycle | to_YMDH %}
 {% set head = RUN + "wave.t" + cycle_HH + "z." %}
+{% set grdNAME = "global.5p00" %}   


Tagging @sbanihash for this as well.

jobs/JGLOBAL_WAVE_POST_SBS

JessicaMeixner-NOAA · 2025-01-10T13:44:56Z

scripts/exgfs_wave_prdgen_gridded.sh

+         echo "Directory ${!com_dir} created."
+     else
+         echo "Directory ${!com_dir} already exists."
+     fi


Tagging @AminIlia-NOAA for the prdgen job.

…NOAA/global-workflow into wavegrid_compath

jobs/JGLOBAL_WAVE_PRDGEN_GRIDDED

jobs/JGLOBAL_ARCHIVE

ush/wave_domain_grid.sh

AntonMFernando-NOAA

@WalterKolczynski-NOAA I think this is ready to be reviewed again.
@JessicaMeixner-NOAA You can run a new test with the multiple output grids, if you agree with the changes.

AntonMFernando-NOAA added 4 commits December 3, 2024 11:29

modified config.com

b6a732e

modified jobs\JGLOBAL_ARCHIVE

0a5ca0e

modified jobs\JGLOBAL_EXTRACTVARS

cb496d1

modified jobs\JGLOBAL_WAVE_POST_SBS

3492b8c

JessicaMeixner-NOAA reviewed Dec 3, 2024

View reviewed changes

jobs/JGLOBAL_WAVE_POST_SBS Outdated Show resolved Hide resolved

AntonMFernando-NOAA and others added 3 commits December 4, 2024 12:19

modified jobs\JGLOBAL_WAVE_PRDGEN_GRIDDED

92cba9c

updated the relevant yaml files

a0aa55b

Merge branch 'NOAA-EMC:develop' into wavegrid_compath

fe0f375

WalterKolczynski-NOAA reviewed Dec 4, 2024

View reviewed changes

jobs/JGLOBAL_ARCHIVE Outdated Show resolved Hide resolved

AntonMFernando-NOAA and others added 4 commits December 4, 2024 14:43

modified parm\archive\gefs_extracted_wave.yaml.j2

52eda4b

Merge branch 'NOAA-EMC:develop' into wavegrid_compath

2f220ea

modified jobs\JGLOBAL_ARCHIVE

c2ce78f

Merge branch 'wavegrid_compath' of https://github.com/AntonMFernando-…

403a395

…NOAA/global-workflow into wavegrid_compath

github-advanced-security bot found potential problems Dec 5, 2024

View reviewed changes

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

modified jobs\JGLOBAL_ARCHIVE

261c833

github-advanced-security bot found potential problems Dec 5, 2024

View reviewed changes

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

AntonMFernando-NOAA added 6 commits December 5, 2024 11:48

modified jobs\JGLOBAL_ARCHIVE

8b01949

modified typos JGLOBAL_ARCHIVE and JGLOBAL_WAVE_POST_SBS

257cd4a

modifed jobs\JGLOBAL_ARCHIVE

4674706

modified jobs\JGLOBAL_ARCHIVE

bac1bb9

modified jobs\JGLOBAL_EXTRACTVARS

477df47

modified jobs\JGLOBAL_WAVE_POST_SBS

f1e54ba

github-advanced-security bot found potential problems Dec 5, 2024

View reviewed changes

jobs/JGLOBAL_WAVE_POST_SBS Fixed Show fixed Hide fixed

AntonMFernando-NOAA added 7 commits December 5, 2024 12:31

modified jobs\JGLOBAL_WAVE_POST_SBS

a91797e

modified jobs\JGLOBAL_WAVE_POST_SBS

832141f

modified JGLOBAL

cccf13a

modified JGLOBAL jobs

2ce47a6

modified jobs\JGLOBAL_ARCHIVE

fd3e0e4

modified jobs\JGLOBAL_WAVE_PRDGEN_GRIDDED

75bb84a

modified JGLOBAl scripts

acee85c

modified scripts\exgfs_wave_prdgen_gridded.sh

c51d912

AntonMFernando-NOAA requested review from JessicaMeixner-NOAA and WalterKolczynski-NOAA January 9, 2025 20:50

AntonMFernando-NOAA commented Jan 9, 2025

View reviewed changes

WalterKolczynski-NOAA reviewed Jan 10, 2025

View reviewed changes

ush/wave_grib2_sbs.sh Outdated Show resolved Hide resolved

JessicaMeixner-NOAA reviewed Jan 10, 2025

View reviewed changes

AntonMFernando-NOAA and others added 5 commits January 10, 2025 11:59

Merge branch 'develop' into wavegrid_compath

8ea2253

edited com_dir

e5ab3e0

Merge branch 'wavegrid_compath' of https://github.com/AntonMFernando-…

c68d4c4

…NOAA/global-workflow into wavegrid_compath

update jobs\JGLOBAL_WAVE_POST_SBS

ae45df6

modified jobs\JGLOBAL_ARCHIVE

8528a8f

github-advanced-security bot found potential problems Jan 11, 2025

View reviewed changes

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 11, 2025

View reviewed changes

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

jobs/JGLOBAL_ARCHIVE Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 11, 2025

View reviewed changes

AntonMFernando-NOAA force-pushed the wavegrid_compath branch from e8b4123 to 8528a8f Compare January 11, 2025 06:23

AntonMFernando-NOAA added 9 commits January 11, 2025 01:27

modified jobs\JGLOBAL_WAVE_POST_SBS

26a9c16

modified parm\archive\gfswave.yaml.j2

098ffe7

changed com var in exscript and JJob

c31c5cb

modified parm\archive\gefs_extracted_wave.yaml.j2

c79ee4c

modified wave_prdgen_gridded vars

0e03672

modified parm\archive\gefs_extracted_wave.yaml.j2

b4f277d

fixed a typo in jobs\JGLOBAL_WAVE_PRDGEN_GRIDDED

a180913

modified scripts\exgfs_wave_prdgen_gridded.sh

59abb66

modified ush\wave_domain_grid.sh

4818b54

github-advanced-security bot found potential problems Jan 12, 2025

View reviewed changes

ush/wave_domain_grid.sh Fixed Show fixed Hide fixed

AntonMFernando-NOAA added 2 commits January 11, 2025 20:56

modified ush\wave_domain_grid.sh

30fe421

modified ush\wave_domain_grid.sh

7cb47fa

AntonMFernando-NOAA commented Jan 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add domain level to wave gridded COM path #3137

Add domain level to wave gridded COM path #3137

AntonMFernando-NOAA commented Dec 3, 2024 •

edited

Loading

AntonMFernando-NOAA left a comment

AntonMFernando-NOAA Jan 9, 2025

WalterKolczynski-NOAA Jan 10, 2025

JessicaMeixner-NOAA Jan 10, 2025

AntonMFernando-NOAA Jan 9, 2025

WalterKolczynski-NOAA Jan 10, 2025

AntonMFernando-NOAA Jan 10, 2025 •

edited

Loading

JessicaMeixner-NOAA Jan 10, 2025

WalterKolczynski-NOAA Jan 10, 2025

AntonMFernando-NOAA Jan 10, 2025

WalterKolczynski-NOAA Jan 10, 2025

AntonMFernando-NOAA Jan 10, 2025

AntonMFernando-NOAA Jan 9, 2025

JessicaMeixner-NOAA Jan 10, 2025

WalterKolczynski-NOAA Jan 10, 2025

JessicaMeixner-NOAA Jan 10, 2025

WalterKolczynski-NOAA Jan 10, 2025 •

edited

Loading

JessicaMeixner-NOAA left a comment

JessicaMeixner-NOAA Jan 10, 2025

JessicaMeixner-NOAA Jan 10, 2025

AntonMFernando-NOAA left a comment •

edited

Loading

Add domain level to wave gridded COM path #3137

Are you sure you want to change the base?

Add domain level to wave gridded COM path #3137

Conversation

AntonMFernando-NOAA commented Dec 3, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

AntonMFernando-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AntonMFernando-NOAA Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AntonMFernando-NOAA left a comment • edited Loading

Choose a reason for hiding this comment

AntonMFernando-NOAA commented Dec 3, 2024 •

edited

Loading

AntonMFernando-NOAA Jan 10, 2025 •

edited

Loading

WalterKolczynski-NOAA Jan 10, 2025 •

edited

Loading

AntonMFernando-NOAA left a comment •

edited

Loading