Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trouble installing Corrfunc 2.3.3 #212

Closed
julianakwan opened this issue Feb 3, 2020 · 33 comments
Closed

Trouble installing Corrfunc 2.3.3 #212

julianakwan opened this issue Feb 3, 2020 · 33 comments
Labels
Installation
Milestone
v2.3.4

Comments

@julianakwan
Copy link

General information

  • Corrfunc version: 2.3.3
  • platform: Linux x86_64
  • installation method (pip/source/other?): source

Issue description

The theory python library doesn't build.

Expected behavior

In my .local/lib/python2.7/site-packages/Corrfunc, I would expect a directory called theory, containing wp.py, DD.py etc.

Actual behavior

When I type 'make' in the Corrfunc directory:

make -C theory
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'abiflags'

What have you tried so far?

Minimal failing example

import Corrfunc


# rest of sample code goes here...
@julianakwan
Copy link
Author

Also "mocks" didn't get installed either.

@lgarrison
Copy link
Collaborator

Interesting, it failed in our Travis build too, but didn't error out: https://travis-ci.org/manodeep/Corrfunc/jobs/645261386#L1330-L1333

The problem is in this line where we build PYTHON_LIBS: https://github.com/manodeep/Corrfunc/blob/master/common.mk#L450

sys.abiflags doesn't exist in Python 2.7.

There's an interesting discussion here about how abiflags may not be relevant anymore, and is an empty string in Python 3.8: https://bugs.python.org/issue36707

@manodeep But actually, are we even using the PYTHON_LIBS variable? I can't find any references to it in our code base, is it being interpreted automatically by setup.py?

@manodeep
Copy link
Owner

manodeep commented Feb 3, 2020

@julianakwan Thanks for the report - we should be able to fix this issue shortly.

@lgarrison Yeah I don't see any references for PYTHON_LIBS either. Might just comment that out for the time being.

@manodeep manodeep added this to the v2.3.4 milestone Feb 3, 2020
@manodeep
Copy link
Owner

manodeep commented Feb 3, 2020

@lgarrison Maybe this is as good a time as any to discuss the timeline for dropping python2 support. Since supporting python2 comes at no additional cost (for now), should we simply retain the compatibility?

@manodeep
Copy link
Owner

manodeep commented Feb 3, 2020

@julianakwan I can see that the error occurs but the installation proceeds as per normal. I can also import Corrfunc, and run the tests. Can you please try the following under your python2 environment:

>>> import Corrfunc
>>> Corrfunc.__version__
>>> from Corrfunc.tests import tests
>>> tests()

@lgarrison
Copy link
Collaborator

lgarrison commented Feb 3, 2020 via email

@julianakwan
Copy link
Author

julianakwan commented Feb 3, 2020 via email

@manodeep
Copy link
Owner

manodeep commented Feb 3, 2020

I tried on the local supercomputer with python2 and I still can't reproduce the issue. @lgarrison Can you please check?

@julianakwan In the meantime, will you please navigate to the /homes/jkwan/.local/lib/python2.7/site-packages/ directory and delete any directories containing *Corrfunc*, and any theory, mocks directories. After that, will you please go back to the source directory and install with the following commands:

$ make -j4 distclean 
$ python -m pip install --user --verbose . &> corrfunc_install.log

and attach the generated corrfunc_install.log file here.

@julianakwan
Copy link
Author

julianakwan commented Feb 5, 2020 via email

@lgarrison
Copy link
Collaborator

I don't think the log made it; you might have to attach it to your comment in GitHub rather than send it as an email attachment. But since you don't have pip, could you actually run the following instead:

$ make -j4 distclean
$ python setup.py install --user --verbose &> corrfunc_install.log

@julianakwan
Copy link
Author

Here it is again, this time using:
python setup.py install --user --verbose &> corrfunc_install.log

@julianakwan
Copy link
Author

corrfunc_install.log

@julianakwan
Copy link
Author

Also in /homes/jkwan/.local/lib/python2.7/site-packages/ I have 4 corrfunc related items:

Corrfunc
Corrfunc-2.3.3-py2.7.egg-info
theory
mocks

But in the theory and mocks directories there is only a file called 'tests' no *.py files

@manodeep
Copy link
Owner

manodeep commented Feb 5, 2020

@julianakwan Thanks for the log. The theory and mocks directories should contain a data sub-directory. From the install log, it looks like most stuff was copied across correctly, except for the theory and mocks directories under site-packages/Corrfunc/. @julianakwan Will you please report back the results from find /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/ -name *.py? Also, will you please report back the distutils (or setuptools) version that you are using?

@lgarrison Any ideas on why a seemingly successful install might not copy the Corrfunc/theory/*.py and the Corrfunc/mocks/*.py files?

(Related note: the theory and mocks directories under site-packages is an unfortunate outcome of the current Corrfunc setup and is noted in #207. By design, those two directories only contain test data and not any of the python scripts.)

A somewhat related note for @julianakwan - the compiler you are using (gcc-4.8.5) is nearly 5 years old. You will likely see better performance from a newer compiler, if that is an option for you.

@julianakwan
Copy link
Author

This is what I get from using find:

/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/call_correlation_functions.py
/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/call_correlation_functions_mocks.py
/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/init.py
/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/tests.py
/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/utils.py
/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/io.py

My version of distutils is 2.7.15

I've noticed that there is a later version of gcc on this cluster (8.2.0) but the modules that I'm using has been built with 4.8.5.

@lgarrison
Copy link
Collaborator

Hmm, I cannot reproduce yet either. I used Python 2 inside a conda environment. pip and setup.py both worked.

@manodeep
Copy link
Owner

manodeep commented Feb 7, 2020

@julianakwan Will you please try to install the previous version Corrfunc (v2.3.2) and see if that installs successfully? That way, we will at least have an idea as to what might be the root cause.

If v2.3.2 does not install correctly either, will you please try progressively earlier versions until you get one that installs correctly.

Thanks for your patience - hopefully we can get to the bottom of this.

@julianakwan
Copy link
Author

I can go back to 2.3.1 successfully.

In the meantime, I've just been copying over the theory and mocks directories to my python path. That seems to work also.

@manodeep
Copy link
Owner

@julianakwan This will take further digging. What is your python version?

@julianakwan
Copy link
Author

It is Python 2.7.15

@manodeep
Copy link
Owner

Hmm that python version should come with pip. What happens if you try python -m pip install -e . ?

@julianakwan
Copy link
Author

I get:
[jkwan@usernode ~]$ python -m pip install -e .
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /cosmo_tortoise/homes/jkwan
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@manodeep
Copy link
Owner

@julianakwan My apologies - please run that command at the Corrfunc root directory

@julianakwan
Copy link
Author

julianakwan commented Feb 12, 2020 •
edited by lgarrison
Loading

Oh sorry!

Here it is again:

[jkwan@usernode ~/Corrfunc]$ python -m pip install -e .
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///cosmo_tortoise/homes/jkwan/Corrfunc
    ERROR: Command errored out with exit status 1:
     command: /cosmo_tortoise/software/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.15-wkqj7mbdjxbiive4b2x2fvkk7xmtxn43/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"'; __file__='"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /cosmo_tortoise/homes/jkwan/Corrfunc/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@manodeep
Copy link
Owner

manodeep commented Feb 12, 2020 •
edited
Loading

Ohh good - you do have pip but the install appears to be missing setuptools. Can you please install setuptools with:

wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | python - --user~

(complete instructions are here).

Google failed - these instructions were for an immensely old version of setuptools - please disregard

@lgarrison
Copy link
Collaborator

lgarrison commented Feb 12, 2020 via email

@manodeep
Copy link
Owner

Those were the instructions on the setuptools PyPI page, but upon further checking I see that those were for setuptools in 2014!

The latest version does not support py2 - can't seem to locate a version that does. Perhaps distutils?

I am also surprised that setuptools is causing an ImportError -- the import at the top of setup.py is protected and falls back to distutils. @lgarrison Do you think this error is occuring because of the requirement of setuptools in setup_requires?

@manodeep
Copy link
Owner

All of this trouble with python2 really makes me think that we should drop python2 support sometime in the next year (after a major release).

@lgarrison
Copy link
Collaborator

lgarrison commented Feb 12, 2020 via email

@manodeep
Copy link
Owner

manodeep commented Apr 2, 2020

@julianakwan Will you please retry installing Corrfunc after installing setuptools with python -m pip install setuptools (please add --user if needed).

(I am assuming the issue you were facing has been resolved but would be good to know what the resolution was)

@julianakwan
Copy link
Author

Sorry for being slow -

Yes, I was able to install Corrfunc using setuptools. Thanks for your help!

@manodeep
Copy link
Owner

manodeep commented Apr 6, 2020

@julianakwan Thanks for the update and no worries at all. Glad the installation issue was all sorted out

@lgarrison
Copy link
Collaborator

Just noting that I see the setuptools error is coming from the one-liner that boostraps Corrfunc's setup.py, not setup.py itself. That is, this line from @julianakwan's report:

     command: /cosmo_tortoise/software/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.15-wkqj7mbdjxbiive4b2x2fvkk7xmtxn43/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"'; __file__='"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info

I guess this is a pip-ism? I've seen this pattern before but don't totally understand where it comes from. It's not something we wrote, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Installation
Projects
None yet
Milestone
v2.3.4
Development

No branches or pull requests
3 participants
@manodeep @lgarrison @julianakwan