-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix wave restart for cold start and add ic version file #3112
base: develop
Are you sure you want to change the base?
Fix wave restart for cold start and add ic version file #3112
Conversation
The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated. Additionally, the restarts were also not being copied from the staged directory to `$DATA`, so now they are. The process is identical to that of non-RERUN warm starts, so the code is re- factored a bit to avoid duplication. Resolves NOAA-EMC#3109
This won't work yet because I need to move all the wave restarts in ICSDIR. |
I have a directory of just 1 IC test so I can get it moved and tested from there pretty easily. Quick question, I should be staging in the previous cycle gdas, correct? |
Yes. Should be the same as we have now, except gfs ➡ gdas |
@WalterKolczynski-NOAA - I have test running. It's not "clean" in the sense that I merged your changes into my branch - but the stage-IC job succeeded and forecast is in the queue. I'll report in the morning and thank you so much for a quick fix to this problem! |
@WalterKolczynski-NOAA - My test from last night has a wave IC!!!! Thank you again for this quick update. Even just having this branch means we can move forward with getting some runs comparing different physics options for the wave model now. |
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
New IC versions have been created for the relocated wave restarts. This required adding an IC versions file. Note: the high-res cases (C768/C1152) likely will still not work. They use the wave grid name as the restart suffix, but for single-grid waves, ending in |
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
I have access to add additional links to files owned by climate group or can coordinate this being found. Can you let me know which directories? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thanks for the IC updates/fixes @WalterKolczynski-NOAA ! Approve pending successful completion of CI testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tested the wave parts of these changes and approve those changes. This PR however touches a lot more than just wave ICs with the IC version change, so maybe posting the output of the CI for more people to look at before giving the official approval might be a good idea?
@WalterKolczynski-NOAA - Can you remind me again of which machine which ICs I need to update for the wave model? I wanted to say you said an HR3? I can also work with @jiandewang to get this updated on WCOSS2 and run an HR4 test there as part of the review as well. |
@WalterKolczynski-NOAA i just looked on wcoss2, this looks update? So can I run an HR4 like test there now? In the meantime, I'm not sure what you need changed otherwise so I'll wait for your response on testing and what else needs changed. |
@JessicaMeixner-NOAA
Needs to be repeated on all machines. (These changes are related to a previous PR fixing wave restarts #3009.) |
Experiment C48_S2SW FAILED on Hera in Build# 2 in |
Experiment C48_ATM FAILED on Hera in Build# 2 with error logs:
Follow link here to view the contents of the above file(s): (link) |
Experiment C48_ATM FAILED on Hera in Build# 2 in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @WalterKolczynski-NOAA !
Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 in |
Experiment C96_S2SWA_gefs_replay_ics FAILED on Hera in Build# 2 in |
Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 in |
Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 in |
Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 2 in |
Experiment C96_atm3DVar FAILED on Hera in Build# 2 in |
CI Failed on Hera in Build# 2
Experiment C48_ATM_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:39 UTC 2024 |
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 with error logs:
Follow link here to view the contents of the above file(s): (link) |
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 in |
Experiment C48_S2SW FAILED on Hera in Build# 3 with error logs:
Follow link here to view the contents of the above file(s): (link) |
Experiment C48_S2SW FAILED on Hera in Build# 3 in |
CI Failed on Hera in Build# 3
|
Description
The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated.
Additionally, the restarts were also not being copied from the staged directory to
$DATA
, so now they are. The process is identical to that of non-RERUN warm starts, so the code is refactored a bit to avoid duplication.As part of updating the ICs with the new restart location, an IC version file is added to support different version numbers for different IC directories. Unlike other version files, this one uses an associative array rather than individual variables.
Resolves #3109
Type of change
Change characteristics
How has this been tested?
Checklist