Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove broken caching and use workaround for broken pip install #28

Merged
merged 39 commits into from
Aug 23, 2023

Conversation

christopherwharrop-noaa
Copy link
Collaborator

@christopherwharrop-noaa christopherwharrop-noaa commented Jul 20, 2023

This PR does four things:

  1. Removes use of docker layer caching from CI workflow. The docker layer caching doesn't work properly because there isn't enough memory on the runner to hold both the cache and the built images. Even when the cache is read, docker-compose runs out of disk space when starting the containers. Docker compose V2 was tried, but it is also broken and fails to find the image it just created (see this). Attempts to upgrade docker compose did not solve the problem.
  2. Works around Spack/pip conflicts. Spack and pip do not play well together. This is a known issue. During the flux-core and flux-sched installations, also gets installed as a dependency. However, it mysteriously disappears when the install of flux completes. This reinstalls pip, which simply updates the Spack view without rebuilding it, so that Parsl may be installed with pip. However, another problem arises because when parsl is installed via pip, pip does something strange that causes a cffi package incompatibility which breaks flux. So after installing Parsl via pip, flux must be reinstalled to reinstate the correct cffi package back into the Spack view. Once all this is done, a functioning Parsl and Flux are installed.
  3. Move the container registry to ghcr.io and use Github packages.
  4. Add a CI badge the README.

@christopherwharrop-noaa christopherwharrop-noaa changed the title Add a status badge to the README Remove broken caching and use workaround for broken pip install Aug 21, 2023
@christopherwharrop-noaa
Copy link
Collaborator Author

@NaureenBharwaniNOAA - This is finally ready. There were a LOT of problems that I had to debug and it took a long time to sort through all the various issues.

Copy link
Collaborator

@NaureenBharwaniNOAA NaureenBharwaniNOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I missed your PR. Just a couple comments for confirmation. The sections of the code that need to reinstall pip and flux when appropriate are clear. LGTM.

docker/frontend/Dockerfile Show resolved Hide resolved
docker/install/install_flux.sh Show resolved Hide resolved
docker/install/install_parsl.sh Show resolved Hide resolved
Copy link
Collaborator

@NaureenBharwaniNOAA NaureenBharwaniNOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@christopherwharrop-noaa christopherwharrop-noaa merged commit 8aac3f1 into main Aug 23, 2023
1 check passed
@christopherwharrop-noaa christopherwharrop-noaa deleted the feature/status-badge branch August 29, 2023 12:29
ilaflott pushed a commit to ilaflott/ExascaleWorkflowSandbox that referenced this pull request May 28, 2024
…-GSL#28)

* Add a status badge to the README

* Don't use caching in CI when pushing container images to registry

* Remove use of docker image cache in push steps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants