Discrepancy in best model v/s (predicted) visualization results

[ABJ66]

Hi all,

I am exploring statsforecast using the reference page: https://nixtlaverse.nixtla.io/statsforecast/docs/getting-started/getting_started_complete.html.

There are roughly 200 unique_ids which I'd like to forecast + do a back-test/cross validation as well. Using rmse as the evaluation metric to identify best model while cross validation.

During this course, I observed the following which seems a bit off for 10 sample unique_ids:
The best model suggested by statsforecast (using cross validation) does not seem to hold true when observing/visualizing the predictions from various models. Though I understand this may not be the like to like comparison, I've ensured that using correct settings for horizon, step_size and n_windows, I am able to expose increasing amount of 'training' data sequentially to all the possible models.

For example in the attached screenshot, the best model suggested is ARIMA whereas visually it seems that one of CES or Holts-Winter seem more appropriate. This happens for all the 10 samples in one way or another. "Current approach" is just a custom approach which is not available in statsforecast but has been incorporated as an additional algorithm.

Would request inputs/comments if this is the right way of looking at the data/results or is there anything more that might need to be included.

Statsforecast version: v.1.7.3. Thanks in advance.

[ABJ66]

Hi all,

Would appreciate if someone could provide any leads on the above. In summary, best model suggested from cross validation does not visually suitable for predictions as per the attached image and earlier commentary.

Thank you in advance.

[ABJ66]

@jmoralez , @federicogarza: Could you please look into this discrepancy. I am observing that the best models across unique_ids are always producing almost constant forecast which does not seem correct. Adding an image for your reference, where the best model as per cross validation is SES. Pls let me know if this is indeed an issue or possibly the way I am running the code.

Following are the forecast and cv settings used:

forecast_df1 = sf.forecast(
    df=df1,
    h=6,
    fitted = True
  )

cv_df1 = sf.cross_validation(
    df = df1,
    h = 12,
    step_size = 12, 
    n_windows = 3
  )

Attached is the data for this unique_id:

ds	y
1	25
2	23
3	22
4	24.66667
5	28.33333
6	30.66667
7	27
8	25.66667
9	21.33333
10	22.33333
11	21.33333
12	23.66667
13	24.33333
14	25.33333
15	22
16	20.66667
17	14.66667
18	16.33333
19	16.33333
20	21.66667
21	24
22	25.33333
23	21
24	19
25	20
26	24.33333
27	28
28	27
29	27
30	28.66667
31	29.66667
32	29.33333
33	26
34	24.33333
35	20.33333
36	17.66667
37	16.66667
38	18.66667
39	21
40	23
41	25.66667
42	27.33333
43	26
44	23.66667
45	23.66667
46	23.33333
47	21.66667
48	21.33333
49	22.33333
50	26
51	26.33333
52	26.33333
53	24.66667
54	21.33333
55	20.33333

And the forecasts from various models for next 6 months (56 to 61) of which SES has been selected as the best model as per RMSE criterion:

ds	HoltWinters	AutoARIMA	CES	SES	CurrentApproach
56	23.66667	20.134317	21.107635	22.777067	22.111111
57	23.66667	19.976959	19.495298	22.777067	21.259258
58	23.33333	19.820831	19.412916	22.777067	21.234568
59	21.66667	19.665924	16.433445	22.777067	21.534979
60	21.33333	19.512228	15.718492	22.777067	21.342936
61	22.33333	19.359732	16.23962	22.777067	21.370827

Thank you in advance.

[elephaint]

Hi - visual interpretation may be misleading - what is the forecast error that all these models produce on the cross-validation set? (see the tutorial for computing the errors)

[ABJ66]

Attached is the cross validation results for reference, followed by best model selection basis RMSE (evaluation_df):

cross validation results:

ds	cutoff	y	HoltWinters	AutoARIMA	CES	SES	CurrentApproach
20	19	21.66667	25.666666	16.333334	16.01426	18.26323	15.777778
21	19	24	21.333334	16.333334	16.08391	18.26323	16.148148
22	19	25.33333	22.333334	16.333334	15.84353	18.26323	16.08642
23	19	21	21.333334	16.333334	15.85037	18.26323	16.004114
24	19	19	23.666666	16.333334	15.66221	18.26323	16.079561
25	19	20	24.333334	16.333334	15.62997	18.26323	16.0567
26	19	24.33333	25.333334	16.333334	15.47516	18.26323	16.046791
27	19	28	22	16.333334	15.41877	18.26323	16.061016
28	19	27	20.666666	16.333334	15.28543	18.26323	16.054836
29	19	27	14.666667	16.333334	15.21431	18.26323	16.054214
30	19	28.66667	16.333334	16.333334	15.09495	18.26323	16.056688
31	19	29.66667	16.333334	16.333334	15.015	18.26323	16.055246
32	31	29.33333	21.666666	29.515697	25.13408	27.34093	28.444445
33	31	26	24	29.365496	19.44466	27.34093	28.925926
34	31	24.33333	25.333334	29.216059	21.83732	27.34093	29.012346
35	31	20.33333	21	29.067381	17.8282	27.34093	28.794239
36	31	17.66667	19	28.91946	20.49322	27.34093	28.910837
37	31	16.66667	20	28.772293	23.73164	27.34093	28.905807
38	31	18.66667	24.333334	28.625875	24.12326	27.34093	28.870295
39	31	21	28	28.480202	20.86195	27.34093	28.895645
40	31	23	27	28.33527	20.17408	27.34093	28.890581
41	31	25.66667	27	28.191074	14.94278	27.34093	28.885508
42	31	27.33333	28.666666	28.047615	17.61629	27.34093	28.890577
43	31	26	29.666666	27.904884	16.06343	27.34093	28.888889
44	43	23.66667	29.333334	25.492216	27.51638	24.57725	26.333332
45	43	23.66667	26	25.302425	25.74464	24.57725	26.555555
46	43	23.33333	24.333334	25.114048	25.87628	24.57725	26.296295
47	43	21.66667	20.333334	24.927073	22.62573	24.57725	26.395061
48	43	21.33333	17.666666	24.741489	21.80011	24.57725	26.415636
49	43	22.33333	16.666666	24.557287	22.75317	24.57725	26.368998
50	43	26	18.666666	24.374458	24.89486	24.57725	26.393232
51	43	26.33333	21	24.192989	25.1866	24.57725	26.392622
52	43	26.33333	23	24.012871	24.68177	24.57725	26.384951
53	43	24.66667	25.666666	23.834095	23.50379	24.57725	26.390268
54	43	21.33333	27.333334	23.656649	24.49732	24.57725	26.38928
55	43	20.33333	26	23.480524	23.66506	24.57725	26.388166

RMSE across models:

HoltWinters	5.265849
AutoARIMA	6.084584
CES	6.080464
SES	5.130453
CurrentApproach	6.636197
best_model	SES