NaN in input field error

[benjamin-cash]

Another issue I have run into with my runs is the following:

FATAL from PE   122: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability

Image              PC                Routine            Line        Source
ufs_model.x        000000000430D1F7  mpp_mod_mp_mpp_er          80  mpp_util_mpi.inc
ufs_model.x        00000000045BB496  mpp_efp_mod_mp_mp         195  mpp_efp.F90
ufs_model.x        0000000004513029  mpp_domains_mod_m         143  mpp_global_sum.fh
ufs_model.x        0000000001CC1810  fv_grid_utils_mod        3077  fv_grid_utils.F90
ufs_model.x        0000000001CFAA79  fv_mapz_mod_mp_la         794  fv_mapz.F90

I have traced this error to the sfc initial condition files I am using, which do indeed included NaNs as the missing value. I have a set of working sfc ic files from the regression tests that do not include any NaNs which don't produce this issue. I'm assuming it is the missing field NaNs that are causing the crash - is there a flag that can be set to cause the code to ignore those NaNs, do we need to use a different missing value, or is it potentially a different problem entirely? I've attached the full log file here.
@pjpegion for your awareness
fcst_mem000_seg0.log

[benjamin-cash]

Some digging around uncovered the answer - there is a check purely for the presence of NaNs that the sfc_data files were failing, because missing values were being set to NaN. Simply replacing the NaNs in the file with -1e30 allowed the model to run without blowing up, so clearly those values were not actually being used in any of the calculations.