Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix wave restart for cold start and add ic version file #3112

Open
wants to merge 10 commits into
base: develop
Choose a base branch
from

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Nov 19, 2024

Description

The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated.

Additionally, the restarts were also not being copied from the staged directory to $DATA, so now they are. The process is identical to that of non-RERUN warm starts, so the code is refactored a bit to avoid duplication.

As part of updating the ICs with the new restart location, an IC version file is added to support different version numbers for different IC directories. Unlike other version files, this one uses an associative array rather than individual variables.

Resolves #3109

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • C48_S2SW test on Hercules
  • C48_S2SWA_gefs test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

The stage job was incorrectly putting wave restarts into the gfs
directory. The forecast job looks for them in the gdas directory,
so this is updated.

Additionally, the restarts were also not being copied from the
staged directory to `$DATA`, so now they are. The process is
identical to that of non-RERUN warm starts, so the code is re-
factored a bit to avoid duplication.

Resolves NOAA-EMC#3109
@WalterKolczynski-NOAA
Copy link
Contributor Author

This won't work yet because I need to move all the wave restarts in ICSDIR.

@JessicaMeixner-NOAA
Copy link
Contributor

This won't work yet because I need to move all the wave restarts in ICSDIR.

I have a directory of just 1 IC test so I can get it moved and tested from there pretty easily. Quick question, I should be staging in the previous cycle gdas, correct?

@WalterKolczynski-NOAA
Copy link
Contributor Author

This won't work yet because I need to move all the wave restarts in ICSDIR.

I have a directory of just 1 IC test so I can get it moved and tested from there pretty easily. Quick question, I should be staging in the previous cycle gdas, correct?

Yes. Should be the same as we have now, except gfs ➡ gdas

@JessicaMeixner-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA - I have test running. It's not "clean" in the sense that I merged your changes into my branch - but the stage-IC job succeeded and forecast is in the queue. I'll report in the morning and thank you so much for a quick fix to this problem!

@JessicaMeixner-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA - My test from last night has a wave IC!!!! Thank you again for this quick update. Even just having this branch means we can move forward with getting some runs comparing different physics options for the wave model now.

Adds a new vesion file for IC directories. Unlike other version
files, this one uses an associative array instead of different
variables.

With the version file in place, the versions are updated on most
of the directories to switch to the relocated wave restarts.

Refs: NOAA-EMC#3109
@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as ready for review November 20, 2024 18:31
@WalterKolczynski-NOAA
Copy link
Contributor Author

New IC versions have been created for the relocated wave restarts. This required adding an IC versions file.

Note: the high-res cases (C768/C1152) likely will still not work. They use the wave grid name as the restart suffix, but for single-grid waves, ending in .ww3 is expected. Unlike the other directories, where I made this change previously by adding symlinks, I can't make create links for the high-res wave restarts because they point to the prototype ICs owned by the climate group. (I guess I could redo it to just point at the file instead of the directory, but I haven't.)

Adds a new vesion file for IC directories. Unlike other version
files, this one uses an associative array instead of different
variables.

With the version file in place, the versions are updated on most
of the directories to switch to the relocated wave restarts.

Refs: NOAA-EMC#3109
@JessicaMeixner-NOAA
Copy link
Contributor

New IC versions have been created for the relocated wave restarts. This required adding an IC versions file.

Note: the high-res cases (C768/C1152) likely will still not work. They use the wave grid name as the restart suffix, but for single-grid waves, ending in .ww3 is expected. Unlike the other directories, where I made this change previously by adding symlinks, I can't make create links for the high-res wave restarts because they point to the prototype ICs owned by the climate group. (I guess I could redo it to just point at the file instead of the directory, but I haven't.)

I have access to add additional links to files owned by climate group or can coordinate this being found. Can you let me know which directories?

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Nov 21, 2024
Copy link
Member

@KateFriedman-NOAA KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for the IC updates/fixes @WalterKolczynski-NOAA ! Approve pending successful completion of CI testing.

Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested the wave parts of these changes and approve those changes. This PR however touches a lot more than just wave ICs with the IC version change, so maybe posting the output of the CI for more people to look at before giving the official approval might be a good idea?

@JessicaMeixner-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA - Can you remind me again of which machine which ICs I need to update for the wave model?

I wanted to say you said an HR3?

I can also work with @jiandewang to get this updated on WCOSS2 and run an HR4 test there as part of the review as well.

@JessicaMeixner-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA - Can you remind me again of which machine which ICs I need to update for the wave model?

I wanted to say you said an HR3?

I can also work with @jiandewang to get this updated on WCOSS2 and run an HR4 test there as part of the review as well.

@WalterKolczynski-NOAA i just looked on wcoss2, this looks update? So can I run an HR4 like test there now? In the meantime, I'm not sure what you need changed otherwise so I'll wait for your response on testing and what else needs changed.

@WalterKolczynski-NOAA
Copy link
Contributor Author

WalterKolczynski-NOAA commented Nov 21, 2024

@WalterKolczynski-NOAA - Can you remind me again of which machine which ICs I need to update for the wave model?

I wanted to say you said an HR3?

I can also work with @jiandewang to get this updated on WCOSS2 and run an HR4 test there as part of the review as well.

@JessicaMeixner-NOAA
HR3, on all machines. Just need to add a symlink in each of the wave restart directories that uses the .ww3 suffix instead of .uglo_m1g16. This code run should do the trick:

20:35:04 HR3marine/> \
for restart in $(find /scratch1/NCEPDEV/climate/role.ufscpara/IC/HR3marine -name '*restart.uglo_m1g16'); do
  dir=$(dirname ${restart})
  target=$(basename ${restart})
  link=${target/%.uglo_m1g16/.ww3}
  cd ${dir}
  ln -s ${target} ${link}
done

Needs to be repeated on all machines.

(These changes are related to a previous PR fixing wave restarts #3009.)

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_S2SW FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C48_S2SW_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_ATM FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f018.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_ATM FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C48_ATM_8029c4cd

Copy link
Member

@KateFriedman-NOAA KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C96C48_hybatmDA_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C96_atm3DVar FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C96_atm3DVar_8029c4cd

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

CI Failed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI_dh/3112


**CI Failed** on Hera in Build# 1<br>Built and ran in directory `/scratch1/NCEPDEV/global/CI_dh/3112`

Experiment C48_ATM_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:39 UTC 2024
Experiment C48_ATM_8029c4cd Terminated: **
Experiment C96_atm3DVar_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:40 UTC 2024
Experiment C96_atm3DVar_8029c4cd Terminated: **
Experiment C96_S2SWA_gefs_replay_ics_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:41 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_8029c4cd Terminated: **
Experiment C96C48_hybatmaerosnowDA_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:41 UTC 2024
Experiment C96C48_hybatmaerosnowDA_8029c4cd Terminated: **
Experiment C48_S2SW_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:41 UTC 2024
Experiment C48_S2SW_8029c4cd Terminated: **
Experiment C96C48_ufs_hybatmDA_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:43 UTC 2024
Experiment C96C48_ufs_hybatmDA_8029c4cd Terminated: **
Experiment C96C48_hybatmDA_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:43 UTC 2024
Experiment C96C48_hybatmDA_8029c4cd Terminated: **
Experiment C48_S2SWA_gefs_8029c4cd Terminated with tasks failed and dead at Tue Nov 26 14:49:46 UTC 2024
Experiment C48_S2SWA_gefs_8029c4cd Terminated: **
Experiment C48_S2SWA_gefs_8029c4cd Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Tue Nov 26 16:06:06 UTC 2024
Experiment C48_S2SWA_gefs_8029c4cd Terminated: FAIL
Error logs:
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem000_seg0.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem002_seg0.log
Experiment C48_S2SW_8029c4cd Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Nov 26 16:12:00 UTC 2024
Experiment C48_S2SW_8029c4cd Terminated: FAIL
Error logs:
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_ocean_prod_f060.log
Experiment C48_ATM_8029c4cd Terminated with 0
FAIL
FAIL tasks failed and 6 dead at Tue Nov 26 16:30:26 UTC 2024
Experiment C48_ATM_8029c4cd Terminated: FAIL
Error logs:
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_ATM_8029c4cd/logs/2021032312/gfs_atmos_prod_f018.log

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Nov 26, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem000_seg0.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem001_seg0.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem002_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C48_S2SWA_gefs_8029c4cd

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_S2SW FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f012.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f018.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f021.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f024.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f027.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f030.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f033.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f036.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f039.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f042.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f045.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f048.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f051.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f054.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f057.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f060.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f063.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f066.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f069.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f072.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 26, 2024

Experiment C48_S2SW FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/EXPDIR/C48_S2SW_8029c4cd

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

CI Failed on Hera in Build# 3
Built and ran in directory /scratch1/NCEPDEV/global/CI_dh/3112


Experiment C48_S2SWA_gefs_8029c4cd Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Tue Nov 26 20:03:05 UTC 2024
Experiment C48_S2SWA_gefs_8029c4cd Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem000_seg0.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem001_seg0.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SWA_gefs_8029c4cd/logs/2021032312/gefs_fcst_mem002_seg0.log
Experiment C48_S2SW_8029c4cd Terminated with 0
FAIL
FAIL tasks failed and 25 dead at Tue Nov 26 20:15:12 UTC 2024
Experiment C48_S2SW_8029c4cd Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f012.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f018.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f021.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f024.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f027.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f030.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f033.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f036.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f039.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f042.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f045.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f048.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f051.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f054.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f057.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f060.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f063.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f066.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f069.log
/scratch1/NCEPDEV/global/CI_dh/3112/RUNTESTS/COMROOT/C48_S2SW_8029c4cd/logs/2021032312/gfs_atmos_prod_f072.log
Experiment C96_S2SWA_gefs_replay_ics_8029c4cd Completed 1 Cycles: *SUCCESS* at Tue Nov 26 21:03:40 UTC 2024
Experiment C48mx500_3DVarAOWCDA_8029c4cd Completed 2 Cycles: *SUCCESS* at Tue Nov 26 21:21:59 UTC 2024
Experiment C48_ATM_8029c4cd Completed 2 Cycles: *SUCCESS* at Tue Nov 26 21:28:04 UTC 2024
Experiment C96_atm3DVar_8029c4cd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 22:47:47 UTC 2024
Experiment C96C48_hybatmDA_8029c4cd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 22:47:57 UTC 2024
Experiment C96C48_hybatmaerosnowDA_8029c4cd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 22:54:07 UTC 2024
Experiment C96C48_ufs_hybatmDA_8029c4cd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 23:30:52 UTC 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

WW3 ICs are not read for HR like experiements.
6 participants