Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spacy git dependency breaks pypi deploy #809

Open
kevinschaper opened this issue Sep 17, 2024 · 3 comments · May be fixed by #920
Open

Spacy git dependency breaks pypi deploy #809

kevinschaper opened this issue Sep 17, 2024 · 3 comments · May be fixed by #920
Assignees
@ptgolden @kevinschaper @madanucd
Milestone
2024-11 Release

Comments

@kevinschaper
Copy link
Member

Our deploys of monarch-py as a library have been failing for a while, it looks like we need to get our spacy dependency pointed at a release version.

Using a plaintext file to store credentials

Publishing monarch-py (v1.8.0-rc2) to PyPI
 - Uploading monarch_py-1.8.0rc2-py3-none-any.whl 0%

 - Uploading monarch_py-1.8.0rc2-py3-none-any.whl 100%
HTTP Error 400: Can't have direct dependency: spacy@ git+https://github.com/explosion/spaCy.git@master. See https://packaging.python.org/specifications/core-metadata for more information. | b"<html>\n <head>\n  <title>400 Can't have direct dependency: spacy@ git+https://github.com/explosion/spaCy.git@master. See https://packaging.python.org/specifications/core-metadata for more information.\n \n <body>\n  <h1>400 Can't have direct dependency: spacy@ git+https://github.com/explosion/spaCy.git@master. See https://packaging.python.org/specifications/core-metadata for more information.\n  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>\nCan&#x27;t have direct dependency: spacy@ git+https://github.com/explosion/spaCy.git@master. See https://packaging.python.org/specifications/core-metadata for more information.\n\n\n \n"
Error: Process completed with exit code 1.
@kevinschaper kevinschaper added this to the 2024-10 Release milestone Sep 17, 2024
@kevinschaper kevinschaper self-assigned this Sep 17, 2024
@kevinschaper
Copy link
Member Author

kevinschaper commented Sep 17, 2024 •
edited
Loading

I fixed the initial dependency problem, but it looks like it's not very happy about the model dependency either:

TTP Error 400: Can't have direct dependency: en_core_sci_sm@ https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz. 

See https://packaging.python.org/specifications/core-metadata for more information. | b"<html>\n <head>\n  <title>400 Can't have direct dependency: en_core_sci_sm@ https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz. 

See https://packaging.python.org/specifications/core-metadata for more information.\n \n <body>\n  <h1>400 Can't have direct dependency: en_core_sci_sm@ https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz. 

See https://packaging.python.org/specifications/core-metadata for more information.\n  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>\nCan&#x27;t have direct dependency: en_core_sci_sm@ https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz. See https://packaging.python.org/specifications/core-metadata for more information.\n\n\n \n"

@madanucd
Copy link
Contributor

Hi @kevinschaper,

It looks like the issue stems from PyPI’s restriction on installing dependencies from external URLs—specifically, the SciSpacy model hosted on an S3 bucket. We had included the en_core_sci_sm model, which points to a file outside of PyPI, leading to the failure.

The Fix: To resolve this, we’ll need to remove any external URL references from the package metadata (like setup.py or pyproject.toml). Instead, the model dependency should be handled outside of the package installation process—either by asking users to install the model manually or by automating its installation within the CI pipeline.

How to Handle the Model Installation in CI: To automate this, we can add a step in the GitHub Actions workflow to install the SciSpacy model during the CI build process. I noticed there’s a workflow called publish-backend, which seems like a good place to add this step. Here’s an example of how we can install the SciSpacy model during the CI build:

- name: Install SciSpacy Model
  run: |
    pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz

This will download and install the model to the default location
(e.g., /usr/local/lib/python3.x/site-packages/en_core_sci_sm/).

If this aligns with the fix you mentioned earlier, feel free to disregard this.

I haven’t used act for local workflow testing or set up a TestPyPI account for testing package publishing. Could you kindly accommodate these changes and let me know if everything works as expected?

Thank you!

@kevinschaper
Copy link
Member Author

We should use pystow, assuming that we can give an alternate path to spacy.

@madanucd madanucd mentioned this issue Nov 26, 2024
Closed
2 tasks
@ptgolden ptgolden linked a pull request Nov 27, 2024 that will close this issue
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ptgolden ptgolden

@kevinschaper kevinschaper

@madanucd madanucd

Labels
None yet
Projects
None yet
Milestone
2024-11 Release
4 participants
@ptgolden @kevinschaper @sagehrke @madanucd