Understanding the evaluation metrics results

[sege01]

Hi @kashif,

I had a small query regarding the evaluation metrics. I got the following results on my dataset using the Transformer conditioned MAF model.

CRPS: 1.3200637762163687
ND: 1.6099393168354736
NRMSE: nan
MSE: nan

Could you please help me with understanding these results? What do these values indicate related to the predictions made by the model? What could be the reason behind the nan as output for MSE and NRMSE?

Your response will be very helpful.
Thank you so much in advance and I am sorry if the question sounds naive.

[kashif]

thanks @sege01 for the question... so are these metrics for the sum of the multivariate forecasts? Just trying to understand it myself why it could be nan..

[sege01]

@kashif Thank you for your reply. The above-mentioned metrics were not the sum of the multivariate forecasts but below are the metrics for the sum of the multivariate forecasts.

CRPS-Sum: 0.699638813420346
ND-Sum: 1.204730089973001
NRMSE-Sum: nan
MSE-Sum: nan

Could it indicate the very bad forecast?

[kashif]

hmm hard to say... I dont remember anymore but I can have a look... some metrics are undefined when the prediction and target are the same or both are zero etc. so I would need to go back and have a look...

[sege01]

Thank you.
Also, these 'NaN' outputs are with MAF only.

[kashif]

I see... hmm then it might be from the prediction head yes... so if you data is categorical or binned, e.g. only has x.5, x.25, etc. then you might need to dequantize it by adding some uniform noise to it...

[sege01]

Okay. Because of that maybe there are high values for CRPS and ND also?
Also then why is this not the case with NVP?

[sege01]

Hi @kashif ,
I had one more query regarding the CRPS. Can CRPS_sum be calculated for DeepAR also?
Thank you.

[kashif]

@sege01 no DeepAR is not a multivariate method so if you want to do something similar then you would use DeepVAR with a diagonal covariance matrix and then CRPS_sum would correspond to the sum of the variates' CRPS which are independent... hopefully that makes sense?

[sege01]

Yes. Makes sense. Thank you so much.