Pip reports version conflict in wrong dependencies

[jwodder]

Description

I have a locally-developed package named dandisets-healthstatus. Its dependencies include pydantic ~= 2.0 and (in an all extra) dandidav @ git+https://github.com/... & dandi (a PyPI package, without any version constraints). dandidav depends on dandi >= 0.58.2. The latest version of dandi on PyPI is 0.59.0.

Unfortunately, what I failed to initially realize was that dandi currently requires Pydantic 1.x (with the upper-version bound enforced by a dependency of dandi). Thus, when I tried to install dandisets-healthstatus with pip install .[all], pip correctly failed due to a version conflict, but the packages that it reported as conflicting actually weren't:

ERROR: Cannot install dandisets-healthstatus and dandisets-healthstatus[all]==0.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    dandisets-healthstatus[all] 0.0.0 depends on dandi; extra == "all"
    dandidav 0.0.0 depends on dandi>=0.58.2

If dandiset-healthstatus's dandi dependency is omitted from the list of requirements, then installation fails with a more accurate error message:

ERROR: Cannot install dandi, dandidav and dandisets-healthstatus==0.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    dandisets-healthstatus 0.0.0 depends on pydantic~=2.0
    dandi 0.58.2 depends on pydantic>=1.9.0
    dandischema 0.8.4 depends on pydantic<2.0 and >=1.8.1
    dandisets-healthstatus 0.0.0 depends on pydantic~=2.0
    dandi 0.58.2 depends on pydantic>=1.9.0
    dandischema 0.8.3 depends on pydantic<2.0 and >=1.8.1

Expected behavior

Pip should have reported that the Pydantic version requirements conflicted rather than the dandi version requirements.

pip version

23.3.2

Python version

3.11.7

OS

macOS Sonoma 14.2.1

How to Reproduce

1. git clone https://github.com/dandi/dandisets-healthstatus.git (I promise it won't bite)
2. cd dandisets-healthstatus
3. git checkout ea4ebaa6
4. cd code
5. If you are testing this in the future, you may need to restrict the dandi requirement in setup.cfg to <= 0.60.0.
6. python3 -m venv venv
7. venv/bin/pip install .[all]

Output

A log of all output from pip install -vvv can be found at https://gist.github.com/jwodder/cd49394c1dfb2bb1e17a5aa32f5c4a49.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Unfortunately I am unable to reporduce on Linux using Python 3.11 and I do not have access to an Apple/Mac platform.

I suspect this is the same cause as this issue: #12317

Would you be able to test this branch of Pip and let me know if or what error you get: https://github.com/notatallshaw/pip/tree/23.3.1-without-backjump

[jwodder]

@notatallshaw Were you trying to reproduce the problem by following my instructions? I accidentally omitted the [all] extra from the pip install command (now fixed).

Regardless, when I use that branch (as of commit b21aebb), the installation eventually fails due to an error while trying to build a wheel for PyYAML. The -vvv logs are at https://gist.github.com/jwodder/7e0739f99a70d08663f8051f50d40209.

[notatallshaw]

Ah yes, I can reproduce now, and it appears to be a different issue than the one I suspected. Unfortunately it is not clear to me immediately what the cause of the issue is, it will likely take some investigation.

[notatallshaw]

FYI, this does not appear to be caused by any "recent" optimization in Pip, I can reproduce on Pip 23.0.1 (this rules out some potential candidate causes).