Docker: python packages via distro package manager?

[mmacata]

After introduction of PEP-668 in the alpine Dockerimage, pip could no longer be used. See #3463 for more details.
Now as this was solved by using a different alpine based base image, there are two different python versions installed - one used by the system and one by the "user" (or application). So far so good but now there is the problem that python packages installed via distro package manager can no longer be used. See also docker-library/python#909

What would be the alternatives for us? Go back to align system python and user python and not be able to use pip? Using virtual environments inside docker (please not 🙃 )? Compile everything with "user" python? (Seems overhead) Or can anyone think of a better solution?

[wenzeslaus]

What are the Python packages which need to be installed from pip? In the #3463 in the Dockerfile, I can see pip installing, pip, setuptools, six, and grass-session. I don't know why any of them should be there. I suggest to do whatever it takes to remove them.

[mmacata]

Ok, and that would also mean that when the final image is used afterwards, pip cannot be used anymore. (Or only in a virtualenv)

[wenzeslaus]

Yes, I think that's a decision of the Linux distribution and partially also of the specific underlying Docker image maintainers and Python maintainers, i.e., not something GRASS GIS should should try to counter. Does that approach make sense to you?

[bazzaar]

I'm here, because Edouard Choinière suggested I might share the workflow that I documented on the grass-users mailing list, in building Grass84dev on OpenSuse linux from source within a Python v3.11 virtual environment, and so not using the base system Python.

I'm not a Docker user, but I understood the Docker ethos is to remove anything from the base system that's redundant, and therefore keep the image size to a minimum.

In OpenSuse there can be multiple Pythons installed concurrently using the distro's package manager (YAST). In OpenSuse 15.5 there's Python 3.6 (the base system) and Python 3.11.

A user can easily create a Python virtual environment in a user project directory, using 'python3 -m venv', pointing to either of these Pythons. I set an alias in the .bashrc file : python3 -> python3.11, for example.

The new venv automatically gets populated with pip, where does it get that from (I'm not sure). Once the venv is activated, pip is available for use within the venv. It can be used to upgrade itself.

In order to build a Python package (such as wxPython) in the activated venv, pip can be used to install both setuptools and six into the virtual environment.

I've not found any reason to use pip outside of a venv, to add extra Python packages to the system (beyond those offered in YAST)

[neteler]

when the final image is used afterwards, pip cannot be used anymore

This would sound like a major showstopper to me. Usually I want to use an image to then do something on top.

[wenzeslaus]

I don't know the details, but that seems like something to discuss with the Linux distribution, no? For pure Linux distro (with its own Python packages) the decision (by Alpine, Ubuntu, others?) now seems to be to limit pip to virtual environments (venv, pipenv, pipx). I'm guessing their answer is to use system package or virtual environments. Or a Docker image based on conda is the answer here? (That's what we are using in Binder.)

[mmacata]

Thank you all for your opinion and experiences, @bazzaar , @wenzeslaus and @neteler . After some more discussions, using virtual environments really seems to be the best solution. I still think it might be one step too much virtualisation / encapsulation, but I cannot think of any other solution.

For the GRASS GIS alpine Dockerimage, I would go back to basic alpine image and remove any pip related commands. A user can then create it's own virtual environment and use it if needed #3530