macOS dyld unable to find Python dylib for cc_binary with embedded interpreter

[martinblech]

Minimal reproduction example in https://github.com/martinblech/pybind11_test

When I run the pybind11_test target, I get Library not loaded: /install/lib/libpython3.11.dylib . It seems like the binary was linked using an incorrect dylib path. Full error message below:

> bazel run :pybind11_test                     ─╯
INFO: Analyzed target //:pybind11_test (2 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //:pybind11_test up-to-date:
  bazel-bin/pybind11_test
INFO: Elapsed time: 1.030s, Critical Path: 0.97s
INFO: 4 processes: 2 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/pybind11_test
dyld[99901]: Library not loaded: /install/lib/libpython3.11.dylib
  Referenced from: <4E94E925-A24E-3750-9588-FC250A643885> /private/var/tmp/_bazel_user/b961ee880161efb851b787580cc7e512/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/pybind11_test
  Reason: tried: '/install/lib/libpython3.11.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/install/lib/libpython3.11.dylib' (no such file), '/install/lib/libpython3.11.dylib' (no such file), '/usr/local/lib/libpython3.11.dylib' (no such file), '/usr/lib/libpython3.11.dylib' (no such file, not in dyld cache)
[1]    99901 abort      bazel run :pybind11_test

[junyer]

Could you just clarify whether the "embedded" interpreter means the interpreter from the hermetic toolchain (as installed by rules_python) or the system interpreter? If the latter, how are you configuring the toolchain?

(Ping, @rickeylev.)

[junyer]

BTW, this shouldn't be necessary given the use of pybind11_bazel version 2.11.1.bzl.3.

[martinblech]

Could you just clarify whether the "embedded" interpreter means the interpreter from the hermetic toolchain (as installed by rules_python) or the system interpreter? If the latter, how are you configuring the toolchain?

I'm not sure. I'm using whatever comes by default with the homebrew bazel install. As you can see in the example reproduction project, there's no toolchain customization whatsoever at the project level.

BTW, this shouldn't be necessary given the use of pybind11_bazel version 2.11.1.bzl.3.

If I comment that line, I get:

bazel run :pybind11_test                     ─╯
ERROR: no such package '@@[unknown repo 'pybind11' requested from @@]//': The repository '@@[unknown repo 'pybind11' requested from @@]' could not be resolved: No repository visible as '@pybind11' from main repository
ERROR: /Users/user/Documents/projects/pybind11_test/BUILD.bazel:1:10: no such package '@@[unknown repo 'pybind11' requested from @@]//': The repository '@@[unknown repo 'pybind11' requested from @@]' could not be resolved: No repository visible as '@pybind11' from main repository and referenced by '//:pybind11_test'
ERROR: Analysis of target '//:pybind11_test' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.292s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

[junyer]

I'm not sure. I'm using whatever comes by default with the homebrew bazel install. As you can see in the example reproduction project, there's no toolchain customization whatsoever at the project level.

Interestingly, Homebrew Bazel forces Homebrew Python. I'm unsure whether that could interfere with the hermetic toolchain that rules_python installs. Would it be feasible for you to retry this build using "normal" Bazel just to rule out that possibility?

If I comment that line, I get:

Mea culpa, I hadn't twigged that the pybind11_test target depended on @pybind11 directly instead of using a pybind_*() rule. You could probably use pybind_library_test() instead of cc_binary(), I think, but I'm doubtful that's going to affect the situation with dyld.

[martinblech]

Just uninstalled the bazel formula and installed bazelisk instead. Does this qualify as a "normal" Bazel install? Still getting the same error:

bazel run :pybind11_test
INFO: Analyzed target //:pybind11_test (72 packages loaded, 592 targets configured).
INFO: Found 1 target...
Target //:pybind11_test up-to-date:
  bazel-bin/pybind11_test
INFO: Elapsed time: 1.264s, Critical Path: 0.92s
INFO: 8 processes: 6 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 8 total actions
INFO: Running command line: bazel-bin/pybind11_test
dyld[41075]: Library not loaded: /install/lib/libpython3.11.dylib
  Referenced from: <4E94E925-A24E-3750-9588-FC250A643885> /private/var/tmp/_bazel_user/b961ee880161efb851b787580cc7e512/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/pybind11_test
  Reason: tried: '/install/lib/libpython3.11.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/install/lib/libpython3.11.dylib' (no such file), '/install/lib/libpython3.11.dylib' (no such file), '/usr/local/lib/libpython3.11.dylib' (no such file), '/usr/lib/libpython3.11.dylib' (no such file, not in dyld cache)

[junyer]

Just uninstalled the bazel formula and installed bazelisk instead. Does this qualify as a "normal" Bazel install? Still getting the same error:

Insofar as even Homebrew Bazelisk would presumably be downloading a "normal" Bazel binary, I believe that answers that. :(

Do you know whether an equivalent "hello, world" without using pybind11 fails in the same way? If so, the problem is likely to be on the rules_python side. I really don't understand why its hermetic toolchain doesn't Just Work™ here...

[junyer]

FYI, as per #77 (comment), there may indeed be a subtle bug in rules_python. The presence of /install/lib/… paths there seems like more than just a curious coincidence. In this case, however, I'm not confident there's any reasonable workaround. Could you please file a bug against rules_python?

[nicholasjng]

FYI, these paths come from the libpythonXY.dylib packaged in the python-standalone builds that rules_python uses. For more context, this is also discussed (and remains unsolved) in bazelbuild/rules_python#1820.

To be more precise: That path is the LC_LOAD_DYLIB entry for libpython3.X in a Bazel target built using the "@rules_python//python/cc:current_py_cc_libs" dep, see bazelbuild/rules_python#1820 (comment).

I'm not sure whether this is a Bazel bug, and Bazel should fix up this path after the build/during linking, or if the Python toolchain macro should run a path fixup script (it seems install_name_tool is the tool for the job here) on the dylib right in the toolchain target. Contributions welcome :^)