Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model static data file dependencies #60

Open
hellkite500 opened this issue Jul 7, 2023 · 1 comment
Open

Model static data file dependencies #60

hellkite500 opened this issue Jul 7, 2023 · 1 comment

Comments

@hellkite500
Copy link

To support a deployable sandbox with a selection of available models, we need to identify, document, and curate the required static resources for each model.

This issue is intended to track known existing data dependencies required, where they currently exist, and how we intend to support deploying these into a given sandbox instance.

In order to support a standard, reproducible sandbox with well defined applications, these static data should be pinned to particual model code versions that support those, and I suspect we will need to copy these files into an s3 sandbox archive to guarantee reproducibility? Or we will want to checksum each static resource and keep those available to verify each deployment (assuming the source of the data will be available as long as this project is.)

@patrick-tripp
Copy link
Member

We have been saving versioned static data for the NOSOFS model applications since early in this project. We have also saved this data for each of the model applications we have tested. Each institution's model application will have different dependencies and might do things a lot differently, e.g. LiveOcean.

See:
https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/workflows/scripts/getNOSOFSfix.sh
and
https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/workflows/scripts/get_fixfiles_s3.sh

Saving them to an S3 bucket is the best bet to ensure long-term availability.

@KatherinePowell-NOAA KatherinePowell-NOAA added this to the Establish access to input data for models milestone Aug 31, 2023
@KatherinePowell-NOAA KatherinePowell-NOAA removed this from the Establish access to input data for models milestone Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants