Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mesoscale NAM/RAP Fixes and Additions - no restart #603

Merged
merged 21 commits into from
Nov 19, 2024
Merged

Mesoscale NAM/RAP Fixes and Additions - no restart #603

merged 21 commits into from
Nov 19, 2024

Conversation

PerryShafran-NOAA
Copy link
Contributor

@PerryShafran-NOAA PerryShafran-NOAA commented Nov 15, 2024 •
edited by malloryprow
Loading

Note to developers: You must use this PR template!

Description of Changes

The following Fixes and Additions are addressed in this PR:

  • Removes WARNINGs for any missing files (addresses Bugzilla)
  • Reduced levels to only required levels
  • Fixes DATAROOT for precip plot job
  • No longer makes empty directories
  • Adds split_by_subset to the processing
  • Changes subprocess.Popen to subprocess.run
  • Defines $NET before $HOMEevs in J-jobs
  • Removes concatenating of stats log files
  • Increases walltime for some snowfall/precip plots job due to new WCOSS2 environment
  • Updates resources for grid2obs plots job based on NCO recommendations

Also reduces the walltime in the dev driver and ecf scripts.
Adds the bufr library to the ecf scripts.

Please include a summary of the changes and the related GitHub issue(s). Please also include relevant motivation and context.

Developer Questions and Checklist

  • Is this a high priorty PR? If so, why and is there a date it needs to be merged by?

No

  • Do you have any planned upcoming annual leave/PTO?

I will be off starting at 2 pm on Friday (11/15)
Off for Thanksgiving week (11/25-11/29)
WCOSS2 shut down next week (11/19-11/22)

  • Are there any changes needed for when the jobs are supposed to run?

No

  • [x ] The code changes follow NCO's EE2 Standards.
  • [x ] Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
  • [ x] References the feature branch for HOMEevs are removed from the code.
  • [ x] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
  • [ x] Jobs over 15 minutes in runtime have restart capability.
  • [x ] If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
  • [ x] Jobs contain the approriate file checking and don't run METplus for any missing data.
  • [x ] Code is using METplus wrappers structure and not calling MET executables directly.
  • [ x] Log is free of any ERRORs or WARNINGs.

Testing Instructions

Please include testing instructions for the PR assignee. Include all relevant input datasets needed to run the tests.

To test:

  1. Clone the repository.
  2. Check out the feature branch feature/mesoscale_fixes_v2.
  3. Set the fix directory
  4. Set export EVSINspcotlk=/lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs/v2.0/prep/cam before you run steps 5 and 6.
  5. Run dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_nam_grid2obs_stats.sh. Job normally runs at vhr=07.
  6. Run dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_rap_grid2obs_stats.sh. Job normally runs at vhr=07.
  7. Run dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_{nam,rap}_precip_stats.sh. Job normally runs hourly. Run every valid hour of the day, ensuring that jobs for vhr 00 to 22 are finished before the vhr=23 job is run.
  8. Run dev/drivers/scripts/stats/mesoscale/jevs_mesocale_{nam,rap}_snowfall_stats.sh. Job normally runs at 00, 06, 12, 18Z. Run for all four valid hours.
  9. Run dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_grid2obs_plots.sh. Set COMIN to point to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs_beta5/$evs_ver_2d. Job normally runs at vhr=01.
  10. Run dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_headline_plots.sh. Set COMIN to point to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs_beta5/$evs_ver_2d. Job normally runs at vhr=01.
  11. Run dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_precip_plots.sh. Set COMIN to point to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs_beta5/$evs_ver_2d. Job normally runs at vhr=13.
  12. Run dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_snowfall_plots.sh. Set COMIN to point to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs_beta5/$evs_ver_2d. Job normally runs at vhr=13.

@malloryprow malloryprow self-assigned this Nov 15, 2024
@malloryprow malloryprow added the enhancement New feature or request label Nov 15, 2024
@malloryprow malloryprow added this to the EVS v2.0.x milestone Nov 15, 2024
@malloryprow
Copy link
Contributor

stats - grid2obs

Both submitted with vhr=07. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/evs/v2.0/stats/mesoscale.

jevs_mesoscale_nam_grid2obs_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_nam_grid2obs_stats_00.o206205184
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.206205184.dbqs01

jevs_mesoscale_rap_grid2obs_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_rap_grid2obs_stats_00.o206205241
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_rap_grid2obs_stats_00.206205241.dbqs01

@PerryShafran-NOAA
Copy link
Contributor Author

The NAM job failed because RESTART_DIR is not set in the environment. This had worked previously. Darn, let me look.

@PerryShafran-NOAA
Copy link
Contributor Author

I think I fixed the error. Please clear the COMOUT directories and run again.

@malloryprow
Copy link
Contributor

Agreed :) Do you think the snowfall jobs need to be run in a similar fashion to the precip?

@PerryShafran-NOAA
Copy link
Contributor Author

It's probably a good idea. Let's do that, but do note that snowfall only runs 4x per day rather than hourly.

@malloryprow
Copy link
Contributor

Oh yeah, well maybe I don't need to. I can run the jobs to 00, 06, and 12. I'll be around at 18Z to submit that then.

@PerryShafran-NOAA
Copy link
Contributor Author

Your call!

@malloryprow
Copy link
Contributor

Okay ran the snowfall for 00, 06, and 12, and the precip for 00-16.

@malloryprow
Copy link
Contributor

stats - snowfall

COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/evs/v2.0/stats/mesoscale

jevs_mesoscale_nam_snowfall_stats.sh

Log Files: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_nam_snowfall_stats_00.o20675*
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_nam_snowfall_stats_00.20675*

jevs_mesoscale_rap_snowfall_stats.sh

Log Files: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/stats/mesoscale/jevs_mesoscale_rap_snowfall_stats_00.o20675*
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_rap_snowfall_stats_00.20675*

@PerryShafran-NOAA
Copy link
Contributor Author

@malloryprow I'll go ahead and check this out when there are final stat files to compare. I've got my own jobs in cron and hopefully they'll run.

@PerryShafran-NOAA
Copy link
Contributor Author

Snowfall checks out! Your stat files are identical to mine and also to emc.vpppg, for both NAM and RAP. No errors or warnings in any of the log files.

@malloryprow
Copy link
Contributor

precip should be good to check the output this morning

@PerryShafran-NOAA
Copy link
Contributor Author

NAM and RAP precip stats look good. Output stat files look good, and no WARNINGs or ERRORs.

@malloryprow
Copy link
Contributor

In scripts/stats/mesoscale/exevs_mesoscale_nam_precip_stats.sh, scripts/stats/mesoscale/exevs_mesoscale_nam_snowfall_stats.shscripts/stats/mesoscale/exevs_mesoscale_rap_precip_stats.sh,scripts/stats/mesoscale/exevs_mesoscale_rap_snowfall_stats.sh, can you remove

echo "******************************"
echo "Begin to print METplus Log files "
cat $DATA/logs/*
echo "End to print METplus Log files "

@PerryShafran-NOAA
Copy link
Contributor Author

Done, thanks for the reminder!

@malloryprow
Copy link
Contributor

malloryprow commented Nov 19, 2024 •
edited
Loading

plots

Jobs have been submitted. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/evs/v2.0/plots/mesoscale.

jevs_mesoscale_grid2obs_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_grid2obs_plots.o206867585
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_grid2obs_plots.206867585.dbqs01

jevs_mesoscale_headline_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_headline_plots.o206867594
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_headline_plots.206867594.dbqs01

jevs_mesoscale_precip_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_precip_plots.o206867598
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_precip_plots.206867598.dbqs01

jevs_mesoscale_snowfall_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_snowfall_plots.o206867626
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_snowfall_plots.206867626.dbqs01

@malloryprow
Copy link
Contributor

I think we can better adjust the resources for jevs_mesoscale_headline_plots.sh

  1. Has select = 1:ncpus=128 but looking at the /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_headline_plots.206867594.dbqs01/headline/plots/plotting_job_scripts/poe_jobs1 there are only 16 jobs running in parallel. I think we can set select = 1:ncpus=16 and export nproc=16.
  2. Has mem=150GB but it is really only using ~20GB.

@PerryShafran-NOAA
Copy link
Contributor Author

@malloryprow Thank you, I will do that.

In the meantime, I can say that the grid2obs and snowfall plot tarballs match mine, so they look as expected. Awaiting my precip plot job to finish. The headline tarball matches mine as well.

Let me make those changes in the headline plot job.

@PerryShafran-NOAA
Copy link
Contributor Author

Resources for the headline plot job are updated.

@malloryprow
Copy link
Contributor

plots

Running the headline plots with new resources to make sure all is well. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/evs/v2.0/plots/mesoscale.

jevs_mesoscale_headline_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr603/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_headline_plots.o206881105
DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_mesoscale_headline_plots.206881105.dbqs01

@PerryShafran-NOAA
Copy link
Contributor Author

The headline plot job is confirmed - tarballs match, and no errors or warnings.

Still awaiting the precip job.

@PerryShafran-NOAA
Copy link
Contributor Author

The precip job is confirmed - plot tarballs match.

Also, all 4 plot tarballs match what is in emc.vpppg as well.

I think that we are good to merge!

malloryprow
malloryprow approved these changes Nov 19, 2024
View reviewed changes
Copy link
Contributor

@malloryprow malloryprow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are good and testing successful.

AndrewBenjamin-NOAA
AndrewBenjamin-NOAA approved these changes Nov 19, 2024
View reviewed changes
Copy link

@AndrewBenjamin-NOAA AndrewBenjamin-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed these changes, and approve this PR.

@malloryprow malloryprow merged commit d43a207 into NOAA-EMC:develop Nov 19, 2024
@malloryprow
Copy link
Contributor

Thanks @PerryShafran-NOAA! Check everything off the Fixes and Additions related to this PR.

BinbinZhou-NOAA added a commit to BinbinZhou-NOAA/EVS that referenced this pull request Nov 20, 2024
@BinbinZhou-NOAA
Merge branch 'feature/sref_fixes' of https://github.com/BinbinZhou-NO…
99d4ebf 
…AA/EVS into feature/sref_fixes

* 'feature/sref_fixes' of https://github.com/BinbinZhou-NOAA/EVS:
  Fixing bug in the ceiling processing for the perfdiag (NOAA-EMC#608)
  Mesoscale NAM/RAP Fixes and Additions - no restart (NOAA-EMC#603)
BinbinZhou-NOAA added a commit to BinbinZhou-NOAA/EVS that referenced this pull request Nov 26, 2024
@BinbinZhou-NOAA
Merge branch 'feature/href_fixes' of https://github.com/BinbinZhou-NO…
dd146f1 
…AA/EVS into feature/href_fixes

* 'feature/href_fixes' of https://github.com/BinbinZhou-NOAA/EVS:
  Feature/subseasonal mpmd plots fix (NOAA-EMC#611)
  feature/EVS-NWPS_regions (NOAA-EMC#609)
  Feature/sref fixes (NOAA-EMC#607)
  Fixing bug in the ceiling processing for the perfdiag (NOAA-EMC#608)
  Mesoscale NAM/RAP Fixes and Additions - no restart (NOAA-EMC#603)
  Feature/subseasonal mpmd fix for stats (NOAA-EMC#602)
@malloryprow malloryprow modified the milestones: EVS v2.0.x, EVS v2.0.0 Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AndrewBenjamin-NOAA AndrewBenjamin-NOAA AndrewBenjamin-NOAA approved these changes

@malloryprow malloryprow malloryprow approved these changes

Assignees

@malloryprow malloryprow

Labels
enhancement New feature or request
Projects
None yet
Milestone
EVS v2.0.0
Development

Successfully merging this pull request may close these issues.
4 participants
@PerryShafran-NOAA @malloryprow @AliciaBentley-NOAA @AndrewBenjamin-NOAA