Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail to use this library #7

Open
MrWWheat opened this issue Jun 17, 2021 · 18 comments
Open

Fail to use this library #7

MrWWheat opened this issue Jun 17, 2021 · 18 comments

Comments

@MrWWheat
Copy link

MrWWheat commented Jun 17, 2021

Hi,

I am trying to run the example, but when I run go get github.com/DataIntelligenceCrew/go-faiss it shows:

# github.com/DataIntelligenceCrew/go-faiss
../../autotune.go:12:6: could not determine kind of name for C.FaissParameterSpace
../../autotune.go:39:2: could not determine kind of name for C.faiss_ParameterSpace_free
../../autotune.go:18:10: could not determine kind of name for C.faiss_ParameterSpace_new
../../autotune.go:29:7: could not determine kind of name for C.faiss_ParameterSpace_set_index_parameter
../../autotune.go:27:8: could not determine kind of name for C.free

I use local Mac env, and I have copy libfaiss_c.so to /usr/local/lib/

By any chance you know the reason? Thanks!

Just tried again, the error becomes:

# github.com/DataIntelligenceCrew/go-faiss
./autotune.go:5:10: fatal error: 'faiss/c_api/AutoTune_c.h' file not found
#include <faiss/c_api/AutoTune_c.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
@p-ouellette
Copy link
Collaborator

Did you install the header files? If you did but they aren't being found, you can set the environment variable CGO_CFLAGS to -I <path_to_include_dir>.

@MrWWheat
Copy link
Author

Do I need to install the header files? I thought just copy libfaiss_c.so to /usr/local/lib/ is enough.

How should I install this faiss/c_api/AutoTune_c.h header file?

@p-ouellette
Copy link
Collaborator

p-ouellette commented Jun 17, 2021

Yes, CGO needs the header files. They should be installed if you use make install after you compile faiss. You can use the cmake option -DCMAKE_INSTALL_PREFIX= to set the install location.

@MrWWheat
Copy link
Author

When I run the make -C build install command after make -C build -j faiss, it shows:

(base) qi@Tians-MacBook-Pro faiss % make -C build install
[ 61%] Built target faiss
[ 62%] Built target swigfaiss_swig_compilation
[ 64%] Linking CXX shared library libfaiss_python_callbacks.dylib
Undefined symbols for architecture x86_64:
  "_PyGILState_Ensure", referenced from:
      (anonymous namespace)::PyThreadLock::PyThreadLock() in python_callbacks.cpp.o
  "_PyGILState_Release", referenced from:
      (anonymous namespace)::PyThreadLock::~PyThreadLock() in python_callbacks.cpp.o
  "_PyObject_CallFunction", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_AsString", referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_FromStringAndSize", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_Size", referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "faiss::FaissException::FaissException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*, char const*, int)", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "faiss::IOReader::fileno()", referenced from:
      vtable for PyCallbackIOReader in python_callbacks.cpp.o
  "faiss::IOWriter::fileno()", referenced from:
      vtable for PyCallbackIOWriter in python_callbacks.cpp.o
  "typeinfo for faiss::FaissException", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "typeinfo for faiss::IOReader", referenced from:
      typeinfo for PyCallbackIOReader in python_callbacks.cpp.o
  "typeinfo for faiss::IOWriter", referenced from:
      typeinfo for PyCallbackIOWriter in python_callbacks.cpp.o
  "vtable for faiss::FaissException", referenced from:
      faiss::FaissException::~FaissException() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for faiss::IOReader", referenced from:
      faiss::IOReader::IOReader() in python_callbacks.cpp.o
      faiss::IOReader::~IOReader() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for faiss::IOWriter", referenced from:
      faiss::IOWriter::IOWriter() in python_callbacks.cpp.o
      faiss::IOWriter::~IOWriter() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [faiss/python/libfaiss_python_callbacks.dylib] Error 1
make[1]: *** [faiss/python/CMakeFiles/faiss_python_callbacks.dir/all] Error 2
make: *** [all] Error 2

Maybe this is why it didn't install the header files....By any chance you know the reason?...

@p-ouellette
Copy link
Collaborator

p-ouellette commented Jun 18, 2021 via email

@MrWWheat
Copy link
Author

MrWWheat commented Jun 18, 2021

make -C install shows make: *** install: No such file or directory. Stop.

@p-ouellette
Copy link
Collaborator

p-ouellette commented Jun 18, 2021 via email

@MrWWheat
Copy link
Author

MrWWheat commented Jun 18, 2021

Oh, it works! But I can not find the libfaiss.so after the build...It is not in the /build/c_api or /build/lib

Currently make -C build install shows:

(base) qi@Tians-MacBook-Pro faiss % make -C build install
[ 64%] Built target faiss
[ 81%] Built target faiss_c
[ 83%] Built target gtest
[ 85%] Built target gtest_main
[100%] Built target faiss_test
Install the project...
-- Install configuration: ""
-- Up-to-date: /usr/local/lib/libfaiss.dylib
-- Up-to-date: /usr/local/include/faiss/AutoTune.h
-- Up-to-date: /usr/local/include/faiss/Clustering.h
-- Up-to-date: /usr/local/include/faiss/IVFlib.h
-- Up-to-date: /usr/local/include/faiss/Index.h
-- Up-to-date: /usr/local/include/faiss/Index2Layer.h
-- Up-to-date: /usr/local/include/faiss/IndexBinary.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryFromFloat.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryHNSW.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryHash.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryIVF.h
-- Up-to-date: /usr/local/include/faiss/IndexFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexHNSW.h
-- Up-to-date: /usr/local/include/faiss/IndexIVF.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQ.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQFastScan.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQR.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFSpectralHash.h
-- Up-to-date: /usr/local/include/faiss/IndexLSH.h
-- Up-to-date: /usr/local/include/faiss/IndexLattice.h
-- Up-to-date: /usr/local/include/faiss/IndexNNDescent.h
-- Up-to-date: /usr/local/include/faiss/IndexNSG.h
-- Up-to-date: /usr/local/include/faiss/IndexPQ.h
-- Up-to-date: /usr/local/include/faiss/IndexPQFastScan.h
-- Up-to-date: /usr/local/include/faiss/IndexPreTransform.h
-- Up-to-date: /usr/local/include/faiss/IndexRefine.h
-- Up-to-date: /usr/local/include/faiss/IndexReplicas.h
-- Up-to-date: /usr/local/include/faiss/IndexResidual.h
-- Up-to-date: /usr/local/include/faiss/IndexScalarQuantizer.h
-- Up-to-date: /usr/local/include/faiss/IndexShards.h
-- Up-to-date: /usr/local/include/faiss/MatrixStats.h
-- Up-to-date: /usr/local/include/faiss/MetaIndexes.h
-- Up-to-date: /usr/local/include/faiss/MetricType.h
-- Up-to-date: /usr/local/include/faiss/VectorTransform.h
-- Up-to-date: /usr/local/include/faiss/clone_index.h
-- Up-to-date: /usr/local/include/faiss/index_factory.h
-- Up-to-date: /usr/local/include/faiss/index_io.h
-- Up-to-date: /usr/local/include/faiss/impl/AdditiveQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/AuxIndexStructures.h
-- Up-to-date: /usr/local/include/faiss/impl/FaissAssert.h
-- Up-to-date: /usr/local/include/faiss/impl/FaissException.h
-- Up-to-date: /usr/local/include/faiss/impl/HNSW.h
-- Up-to-date: /usr/local/include/faiss/impl/LocalSearchQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/NNDescent.h
-- Up-to-date: /usr/local/include/faiss/impl/NSG.h
-- Up-to-date: /usr/local/include/faiss/impl/PolysemousTraining.h
-- Up-to-date: /usr/local/include/faiss/impl/ProductQuantizer-inl.h
-- Up-to-date: /usr/local/include/faiss/impl/ProductQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ResidualQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ResultHandler.h
-- Up-to-date: /usr/local/include/faiss/impl/ScalarQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ThreadedIndex-inl.h
-- Up-to-date: /usr/local/include/faiss/impl/ThreadedIndex.h
-- Up-to-date: /usr/local/include/faiss/impl/io.h
-- Up-to-date: /usr/local/include/faiss/impl/io_macros.h
-- Up-to-date: /usr/local/include/faiss/impl/lattice_Zn.h
-- Up-to-date: /usr/local/include/faiss/impl/platform_macros.h
-- Up-to-date: /usr/local/include/faiss/impl/pq4_fast_scan.h
-- Up-to-date: /usr/local/include/faiss/impl/simd_result_handlers.h
-- Up-to-date: /usr/local/include/faiss/invlists/BlockInvertedLists.h
-- Up-to-date: /usr/local/include/faiss/invlists/DirectMap.h
-- Up-to-date: /usr/local/include/faiss/invlists/InvertedLists.h
-- Up-to-date: /usr/local/include/faiss/invlists/InvertedListsIOHook.h
-- Up-to-date: /usr/local/include/faiss/utils/AlignedTable.h
-- Up-to-date: /usr/local/include/faiss/utils/Heap.h
-- Up-to-date: /usr/local/include/faiss/utils/WorkerThread.h
-- Up-to-date: /usr/local/include/faiss/utils/distances.h
-- Up-to-date: /usr/local/include/faiss/utils/extra_distances-inl.h
-- Up-to-date: /usr/local/include/faiss/utils/extra_distances.h
-- Up-to-date: /usr/local/include/faiss/utils/hamming-inl.h
-- Up-to-date: /usr/local/include/faiss/utils/hamming.h
-- Up-to-date: /usr/local/include/faiss/utils/ordered_key_value.h
-- Up-to-date: /usr/local/include/faiss/utils/partitioning.h
-- Up-to-date: /usr/local/include/faiss/utils/quantize_lut.h
-- Up-to-date: /usr/local/include/faiss/utils/random.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_avx2.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_emulated.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_neon.h
-- Up-to-date: /usr/local/include/faiss/utils/utils.h
-- Up-to-date: /usr/local/include/faiss/invlists/OnDiskInvertedLists.h
-- Up-to-date: /usr/local/share/faiss/faiss-config.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-config-version.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-targets.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-targets-noconfig.cmake
-- Up-to-date: /usr/local/include/faiss/c_api/AutoTune_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/Clustering_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexFlat_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexIVFFlat_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexIVF_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexLSH_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexPreTransform_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexScalarQuantizer_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexShards_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/Index_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/MetaIndexes_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/VectorTransform_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/clone_index_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/error_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/error_impl.h
-- Up-to-date: /usr/local/include/faiss/c_api/faiss_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/impl/AuxIndexStructures_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/index_factory_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/index_io_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/macros_impl.h
-- Up-to-date: /usr/local/include/faiss/c_api/utils/distances_c.h

@p-ouellette
Copy link
Collaborator

Is it in build/faiss?

@MrWWheat
Copy link
Author

MrWWheat commented Jun 18, 2021

No, I can only find libfaiss.dylib, I searched all files in faiss, and there is no libfaiss.so...

@p-ouellette
Copy link
Collaborator

Do you have libfaiss_c.so in build/c_api? If not, you could try building again with -DFAISS_ENABLE_PYTHON=OFF passed to cmake. You got the same error as #5.

@MrWWheat
Copy link
Author

Yes, I tried again, my commands are:

cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
make -C build
make -C build install

But still no libfaiss.so in build...

@p-ouellette
Copy link
Collaborator

Oh I think MacOS uses .dylib instead of .so. It should work with the dylib.

@MrWWheat
Copy link
Author

Yes, but when I run go get github.com/DataIntelligenceCrew/go-faiss, it shows:

# github.com/DataIntelligenceCrew/go-faiss
ld: library not found for -lfaiss_c
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And libfaiss.dylib is already in the /usr/local/lib

@p-ouellette
Copy link
Collaborator

The error indicates that it can't find libfaiss_c, not libfaiss. Also /usr/local/lib might not be searched. You could try installing the library in /usr/lib instead or adding /usr/local/lib to DYLD_LIBRARY_PATH. I don't use MacOS so I'm not sure.

@amvicc
Copy link

amvicc commented Jan 24, 2023

I had same issue you need just copy libfaiss_c.dylib to /usr/local/lib.
sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib/libfaiss_c.dylib

After that everything worked fine.

In total on MacOs I made this steps:
git clone -b v1.7.2 https://github.com/facebookresearch/faiss.git
cd faiss
export CMAKE_PREFIX_PATH=/opt/homebrew/opt/openblas:/opt/homebrew/opt/libomp:/opt/homebrew
cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DFAISS_ENABLE_PYTHON=OFF .
make -C build
sudo make -C build install
sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib/libfaiss_c.dylib

@fkurushin
Copy link

fkurushin commented Feb 1, 2023

ive tried to install go-faiss using instructions above , despite I've got the same error.

Adding export CGO_CFLAGS="-I /usr/local/include" to my bash profile solved problem.

PS After i have another one error, solved by export LIBRARY_PATH=${LIBRARY_PATH}:/usr/local/lib

PSS in the begging I have to switch 1.7.2 -> 1.7.3

@ToluClassics
Copy link

I had same issue you need just copy libfaiss_c.dylib to /usr/local/lib. sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib/libfaiss_c.dylib

After that everything worked fine.

In total on MacOs I made this steps: git clone -b v1.7.2 https://github.com/facebookresearch/faiss.git cd faiss export CMAKE_PREFIX_PATH=/opt/homebrew/opt/openblas:/opt/homebrew/opt/libomp:/opt/homebrew cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DFAISS_ENABLE_PYTHON=OFF . make -C build sudo make -C build install sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib/libfaiss_c.dylib

You are a lifesaver lol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants