Problem getting fitted values using cross validation with a spark dataframe

[Jonathan-87]

What happened + What you expected to happen

When using cross validation with a spark dataframe we get an error when we want to recover fitted values saying "Exception: Please run cross_validation method using fitted=True".

Versions / Dependencies

statsforecast 1.7.4

Reproduction script

import datetime as dt
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from statsforecast.utils import AirPassengersDF
from statsforecast import StatsForecast
from statsforecast.models import AutoETS

models = [AutoETS(season_length=12)]

sf = StatsForecast(
models=models,
freq='M',
verbose=True,
n_jobs=-1
)

horizon = 12
n_windows = 5

spk_df = spark.createDataFrame(AirPassengersDF)
spk_df = spk_df.repartition("unique_id")
spk_df.cache().persist()

cv_results = sf.cross_validation(
df=spk_df,
h=horizon,
step_size=horizon,
n_windows=n_windows,
refit=True,
level=[80, 90],
id_col="unique_id",
time_col="ds",
target_col="y",
fitted=True
)

insample_fcst = sf.cross_validation_fitted_values()

Issue Severity

None

[jmoralez]

Hey @Jonathan-87, the cross validation method doesn't support fitted values with distributed, only the forecast method does.

[Jonathan-87]

Oh ok good to know I have not seen a reference about this in the documentation, maybe I missed it. I supposed it will be the same for fit and fit_predict method (i write another issue about this)