Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How does backtesting work in foundation time series models ? #377

Open
arunbharadwaj2009 opened this issue May 29, 2024 · 1 comment
Open
Assignees
Labels

Comments

@arunbharadwaj2009
Copy link

When I build a non-foundation time series, I have clearly demarcated training and testing sets. But with foundation models, how can I apply the model to a backdated time series ? Technically it may be easy to do this but how can I be sure that the model is not cheating ? To be sure that there is no cheating taking place, the makes of the foundation model need to open source all the datasets that were used to train the model. Once I am sure that the y variable I am predicting on is not a part of the foundation models training dataset (or related y variables), then I can say with complete certainty that the model can be applied for periods back in time. Can I see which all datasets have been used to train TimeGPT ? Moirai (Salesforce) have given a breakdown of which categories of time series data sets (finance, energy etc) they have used to train their model, in one of the papers.

@elephaint elephaint self-assigned this Jul 30, 2024
@elephaint
Copy link
Contributor

Hi - apologies for the late reply.

This is a good question! At this moment we're not able to share the details around the datasets we use. If you forecast the unknown future, you're sure that data wasn't used in training TimeGPT. Also, if you use a proprietary dataset, it is obviously also not part of our training curriculum, so if you'd like to be completely certain that there is no 'cheating' going on, that may be the way to go for you.

Hope this helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants