pydantic-core fails to install on Python 3.13

[andy-maier]

From https://github.com/zhmcclient/python-zhmcclient/actions/runs/11495456556/job/31995020656?pr=1685:

Building wheels for collected packages: coverage, pydantic-core, sphinx-git, lazy-object-proxy
  Building wheel for coverage (pyproject.toml): started
  Building wheel for coverage (pyproject.toml): finished with status 'done'
  Created wheel for coverage: filename=coverage-6.5.0-py3-none-any.whl size=177753 sha256=6db5ca4f9ea4d24fa4efbf65256b64bd818b87a14cf95e62f0bf02d68ab3850c
  Stored in directory: /home/runner/.cache/pip/wheels/e9/97/3a/0c1a575b666efba54305644ef38826e450811e84425a226757
  Building wheel for pydantic-core (pyproject.toml): started
  Building wheel for pydantic-core (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for pydantic-core (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [114 lines of output]
      Running `maturin pep517 build-wheel -i /opt/hostedtoolcache/Python/3.13.0/x64/bin/python --compatibility off`
      📦 Including license file "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/LICENSE"
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.13 at /opt/hostedtoolcache/Python/3.13.0/x64/bin/python
      📡 Using build options features, bindings from pyproject.toml
         Compiling autocfg v1.1.0
         Compiling proc-macro2 v1.0.69
         Compiling quote v1.0.29
         Compiling unicode-ident v1.0.10
         Compiling target-lexicon v0.12.9
         Compiling python3-dll-a v0.2.9
         Compiling once_cell v1.18.0
         Compiling libc v0.2.147
         Compiling heck v0.4.1
         Compiling static_assertions v1.1.0
         Compiling pyo3-build-config v0.20.0
         Compiling lexical-util v0.8.5
         Compiling num-traits v0.2.16
         Compiling version_check v0.9.4
         Compiling cfg-if v1.0.0
         Compiling syn v2.0.38
         Compiling pyo3-ffi v0.20.0
         Compiling num-integer v0.1.45
         Compiling lock_api v0.4.10
         Compiling parking_lot_core v0.9.8
         Compiling rustversion v1.0.13
         Compiling num-bigint v0.4.4
         Compiling memoffset v0.9.0
         Compiling tinyvec_macros v0.1.1
         Compiling smallvec v1.11.1
         Compiling scopeguard v1.1.0
         Compiling tinyvec v1.6.0
         Compiling pyo3-macros-backend v0.20.0
         Compiling pyo3 v0.20.0
         Compiling lexical-write-integer v0.8.5
         Compiling lexical-parse-integer v0.8.6
         Compiling ahash v0.8.6
         Compiling serde v1.0.190
         Compiling memchr v2.6.3
         Compiling lexical-parse-float v0.8.5
         Compiling aho-corasick v1.0.2
         Compiling lexical-write-float v0.8.5
         Compiling strum_macros v0.25.3
         Compiling serde_derive v1.0.190
         Compiling pyo3-macros v0.20.0
         Compiling unicode-normalization v0.1.22
         Compiling parking_lot v0.12.1
         Compiling getrandom v0.2.10
         Compiling indoc v2.0.4
         Compiling zerocopy v0.7.20
         Compiling hashbrown v0.14.0
         Compiling equivalent v1.0.1
         Compiling serde_json v1.0.108
         Compiling unicode-bidi v0.3.13
         Compiling regex-syntax v0.8.2
         Compiling percent-encoding v2.3.0
         Compiling unindent v0.2.3
         Compiling form_urlencoded v1.2.0
         Compiling idna v0.4.0
         Compiling indexmap v2.0.0
         Compiling regex-automata v0.4.3
         Compiling strum v0.25.0
         Compiling lexical-core v0.8.5
         Compiling pydantic-core v2.14.6 (/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54)
         Compiling ryu v1.0.14
         Compiling itoa v1.0.8
      error: failed to run custom build command for `pydantic-core v2.14.6 (/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54)`
      
      Caused by:
        process didn't exit successfully: `/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/target/release/build/pydantic-core-a87b3f2314a8b4f1/build-script-build` (exit status: 101)
        --- stdout
        cargo:rustc-cfg=Py_3_6
        cargo:rustc-cfg=Py_3_7
        cargo:rustc-cfg=Py_3_8
        cargo:rustc-cfg=Py_3_9
        cargo:rustc-cfg=Py_3_10
        cargo:rustc-cfg=Py_3_11
        cargo:rustc-cfg=Py_3_12
        cargo:rustc-cfg=Py_3_13
        cargo:rerun-if-changed=python/pydantic_core/core_schema.py
        cargo:rerun-if-changed=generate_self_schema.py
      
        --- stderr
        Traceback (most recent call last):
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 192, in eval_forward_ref
            return type_._evaluate(core_schema.__dict__, None, set())
                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'
      
        During handling of the above exception, another exception occurred:
      
        Traceback (most recent call last):
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 240, in <module>
            main()
            ~~~~^^
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 210, in main
            value = get_schema(s, definitions)
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 54, in get_schema
            return type_dict_schema(obj, definitions)
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 152, in type_dict_schema
            field_type = eval_forward_ref(field_type)
          File "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/generate_self_schema.py", line 195, in eval_forward_ref
            return type_._evaluate(core_schema.__dict__, None)
                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'
        thread 'main' panicked at build.rs:29:9:
        generate_self_schema.py failed with exit status: 1
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/opt/hostedtoolcache/Python/3.13.0/x64/bin/python" PYTHON_SYS_EXECUTABLE="/opt/hostedtoolcache/Python/3.13.0/x64/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-cftwxyan/pydantic-core_49873da4f04c4862876f337e55e98c54/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/opt/hostedtoolcache/Python/3.13.0/x64/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pydantic-core
  Building wheel for sphinx-git (setup.py): started
  Building wheel for sphinx-git (setup.py): finished with status 'done'
  Created wheel for sphinx-git: filename=sphinx_git-11.0.0-py3-none-any.whl size=16171 sha256=5feba1600d07ade84d64ed6c970da283066aa9a288168641d3aea22b4f6c5851
  Stored in directory: /home/runner/.cache/pip/wheels/3b/f5/2f/af7b4f4773454c0d5dd4d2976ad6743ee7f9608ec9e1864fba
  Building wheel for lazy-object-proxy (pyproject.toml): started
  Building wheel for lazy-object-proxy (pyproject.toml): finished with status 'done'
  Created wheel for lazy-object-proxy: filename=lazy_object_proxy-1.10.0-cp313-cp313-linux_x86_64.whl size=71096 sha256=d678e7e0addb4047b6a3ea7c5e6db47f42b352d315f2ea3b5eb1e84b3e91efb4
  Stored in directory: /home/runner/.cache/pip/wheels/5a/8a/a3/6b270f863df3862ab7e401bc1c97bebee5b87452b51189ebea
Successfully built coverage sphinx-git lazy-object-proxy
Failed to build pydantic-core
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pydantic-core)

This happened with pydantic-core version 2.14.6.

Investigation revealed that this install issue was fixed in pydantic-core version 2.18.3, which for the first time was pulled in by pydantic version 2.7.2.

The reason the old pydantic-core version 2.14.6 was used is that the safety-schemas package version 0.0.7 specifies pydantic ~= 2.5.0 which uses the so-called compatibility operator ~= which is equivalent to >=2.5.0, <2.6.0. This prevents installing the version of pydantic that has the bug fix.

safety-schemas 0.0.5 allowed any version for the pydantic package; That's why this issue started to occur with the release of safety-schemas 0.0.7.

I have created pyupio/safety#620 to report the issue to the safety / safety-schemas packages.

Circumvention is to exclude safety-schemas 0.0.7.