Skip to content
This repository has been archived by the owner on Jul 1, 2023. It is now read-only.

Need -fPIC for static builds #449

Open
powderluv opened this issue Jun 10, 2022 · 0 comments · May be fixed by #450
Open

Need -fPIC for static builds #449

powderluv opened this issue Jun 10, 2022 · 0 comments · May be fixed by #450

Comments

@powderluv
Copy link

Building PyTorch as a static build fails in TensorPipe:

How to repro:

Checkout latest PyTorch codebase and build with

$ USE_MKLDNN=OFF USE_CUDA=OFF BUILD_SHARED_LIBS=OFF BUILD_TESTS=OFF EXTRA_CAFFE2_CMAKE_FLAGS="-DTORCH_STATIC=1" python3 setup.py build --cmake-only
$ cmake --build build

It would error with:

/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'                             
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC                                                                                                                              
/usr/bin/ld: final link failed: bad value                                                                                                                            
collect2: error: ld returned 1 exit status                                                                                                                           
[5619/5838] Linking CXX shared library lib/libjitbackend_test.so                                                                                                     
FAILED: lib/libjitbackend_test.so                                                                                                                                    
: && /usr/bin/c++ -fPIC -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -
DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missi
ng-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno
-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiag
nostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type
 -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG  -Wl,--no-as-needed -rdynamic -shared -Wl,-soname,libjitbackend
_test.so -o lib/libjitbackend_test.so test_jit/CMakeFiles/jitbackend_test.dir/test_backend_lib.cpp.o  -Wl,-rpath,:::::::  lib/libtorch.a  -Wl,--whole-archive,"/home/
anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl,--no-whole-archive  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a  /usr/lib/
gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-wh
ole-archive  lib/libc10.a  lib/libqnnpack.a  lib/libpytorch_qnnpack.a  lib/libnnpack.a  lib/libXNNPACK.a  lib/libpthreadpool.a  -lm  lib/libfbgemm.a  lib/libcpuinfo.
a  lib/libclog.a  lib/libasmjit.a  lib/libtensorpipe.a  lib/libtensorpipe_uv.a  lib/libgloo.a  -lpthread  lib/libfoxi_loader.a  lib/libkineto.a  -lgcc_s  -lgcc  -lrt
  -ldl  -lm  sleef/lib/libsleef.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-archive  lib/libonnx_proto.a  lib/libprotobuf.
a && :                                                                                                                                                               
/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC                                                                                                                              
/usr/bin/ld: final link failed: bad value                                                                                                                            
collect2: error: ld returned 1 exit status                                                                                                                           
[5621/5838] Linking CXX shared library lib/libbackend_with_compiler.so                                                                                               
FAILED: lib/libbackend_with_compiler.so                                                                                                                              
: && /usr/bin/c++ -fPIC -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -
DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missi
ng-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno
-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiag
nostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type
 -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG  -Wl,--no-as-needed -rdynamic -shared -Wl,-soname,libbackend_wi
th_compiler.so -o lib/libbackend_with_compiler.so test_jit/CMakeFiles/backend_with_compiler.dir/test_backend_compiler_lib.cpp.o test_jit/CMakeFiles/backend_with_comp
iler.dir/test_backend_compiler_preprocess.cpp.o test_jit/CMakeFiles/backend_with_compiler.dir/__/__/__/torch/csrc/jit/mobile/profiler_edge.cpp.o  -Wl,-rpath,::::::: 
 lib/libtorch.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl,--no-whole-archive  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/
lib/x86_64-linux-gnu/libpthread.a  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-whole-archive  lib/libc10.a  lib/libqnnpack.a  lib/libpytorch_qnnpack.a  lib/libnnpack.a  lib/libXNNPACK.a  lib/libpthreadp
ool.a  -lm  lib/libfbgemm.a  lib/libcpuinfo.a  lib/libclog.a  lib/libasmjit.a  lib/libtensorpipe.a  lib/libtensorpipe_uv.a  lib/libgloo.a  -lpthread  lib/libfoxi_loa
der.a  lib/libkineto.a  -lgcc_s  -lgcc  -lrt  -ldl  -lm  sleef/lib/libsleef.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-ar
chive  lib/libonnx_proto.a  lib/libprotobuf.a && :                                
/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'                             
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC                                                                                                                              /usr/bin/ld: final link failed: bad value                                         
collect2: error: ld returned 1 exit status                                        
[5630/5838] Building CXX object test_tensorexpr/CMakeFiles/tutorial_tensorexpr.dir/tutorial.cpp.o
ninja: build stopped: subcommand failed.                                                         ```


I applied ```

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 77df76d..111cddb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -9,6 +9,7 @@ cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
 project(tensorpipe LANGUAGES C CXX)
 
 set(CMAKE_CXX_STANDARD 14)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)

but then there are further linking errors

FAILED: bin/test_cpp_rpc                                                                                                                                  [103/47579]: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_P
YTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-fie
ld-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/__/common/main.cpp
.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/e2e_test_base.cpp.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/test_wire_serialization.cpp.o test_cpp_rpc/CMakeFiles/test_c
pp_rpc.dir/test_e2e_tensorpipe.cpp.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/test_tensorpipe_serialization.cpp.o -o bin/test_cpp_rpc  -Wl,-rpath,:::::::  lib/libtor
ch.a  lib/libgtest.a  lib/libtensorpipe.a  lib/libtensorpipe.a  lib/libtensorpipe_uv.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl
,--no-whole-archive  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-whole-archive  lib/libc10.a  lib/libqnnpack.a  lib/l
ibpytorch_qnnpack.a  lib/libnnpack.a  lib/libXNNPACK.a  lib/libpthreadpool.a  -lm  lib/libfbgemm.a  lib/libcpuinfo.a  lib/libclog.a  lib/libasmjit.a  lib/libgloo.a  
-lpthread  lib/libfoxi_loader.a  lib/libkineto.a  -lgcc_s  -lgcc  -lrt  -ldl  -lm  sleef/lib/libsleef.a  -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-archive  lib/libonnx_proto.a  lib/libprotobuf.a && :                                                                                         /usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::shutdownImpl()':    tensorpipe_agent.cpp:(.text+0x800): undefined reference to `tensorpipe::Context::join()'                                                                             
/usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::sendCompletedRespons
eMessage(std::shared_ptr<tensorpipe::Pipe>&, c10::ivalue::Future&, unsigned long, std::vector<c10::Stream, std::allocator<c10::Stream> >)::{lambda(tensorpipe::Error 
const&)#2}::operator()(tensorpipe::Error const&) const':                                                                                                             
tensorpipe_agent.cpp:(.text+0xc2c): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()'                                                            
/usr/bin/ld: tensorpipe_agent.cpp:(.text+0xc60): undefined reference to `tensorpipe::Error::what[abi:cxx11]() const'                                                 /usr/bin/ld: tensorpipe_agent.cpp:(.text+0xd49): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()'                                               /usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::sendCompletedResponseMessage(std::shared_ptr<tensorpipe::Pipe>&, c10::ivalue::Future&, unsigned long, std::vector<c10::Stream, std::allocator<c10::Stream> >)::{lambda(tensorpipe::Error const&)#1}::operator()(tensorpipe::Error const&) const':                                                                                                             tensorpipe_agent.cpp:(.text+0xe5c): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()'
powderluv added a commit to powderluv/tensorpipe that referenced this issue Jun 10, 2022
@powderluv
Set Position independent code in CMake
6589c54 
Static builds of PyTorch get further along.
Partially fixes pytorch#449
@powderluv powderluv linked a pull request Jun 10, 2022 that will close this issue
Open
powderluv added a commit to powderluv/builder that referenced this issue Jun 10, 2022
@powderluv
Fix libtorch static builds that regressed
8fc6312 
libtorch-static used to ship libtorch.a until 1.0.1 and has
regressed since forcing people to build from source.

This fixes the build commands and add BUILD_SHARED_LIBS=OFF
It also removes build it as a debug build, and turns off
USE_TENSORPIPE until pytorch/tensorpipe#449
is fixed.

Tested on my system to verify a libtorch.a is produced but can't
test the CI
@powderluv powderluv mentioned this issue Jun 10, 2022
Merged
powderluv added a commit to powderluv/builder that referenced this issue Jun 10, 2022
@powderluv
Fix libtorch static builds that regressed
8fef7c9 
libtorch-static used to ship libtorch.a until 1.0.1 and has
regressed since forcing people to build from source.

This fixes the build commands and add BUILD_SHARED_LIBS=OFF
It turns off USE_TENSORPIPE until
pytorch/tensorpipe#449 is fixed.

Tested on my system to verify a libtorch.a is produced but can't
test the CI
@powderluv powderluv mentioned this issue Jun 22, 2022
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Set Position independent code in CMake powderluv/tensorpipe
1 participant
@powderluv