Allow passing callable venv_backend for creation of session virtualenv

[wpk-nist-gov]

This addresses #751

This PR allows for callable venv_backend to nox.session. This functionality can be used to, for example,

• Create conda environments from environment.yaml files during creation
• Easily override the virtual environment location

This is a general solution that should be applicable to a number of open issues.

Main changes:

• Update nox.sessions.SessionRunner._create_venv to look for a callable.
• Included a section in Customized virtual environment creation in docs/tutorial.rst and example usage
  in docs/cookbook.rst.
• Added (simple) tests of this functionality to tests_session.py. Nothing too deep, but there is coverage.
• Changed the conda_tests session of noxfile.py to use this functionality. This now creates conda environments using
  conda env create -f .... with environment files under environment/py{py_version}-conda-test.yaml. These files are created from requirements-conda-test.txt. There is also a session to use conda-lock.
  This is primarily for demonstration purposes.

Open issues:

• The conda examples refer to private variables (SessionRunner._reused and SessionRunner._clean_location). Does it make sense to make these public? I think that this is "advanced" functionality, so I'm happy to go with it.
• The callable has the signature

def venv_backend(
    location: str,
    interpreter: str | None,
    reuse_existing: bool,
    venv_params: Any,
    runner: SessionRunner,
) -> CondaEnv/VirtualEnv/..."
...

The parameters where copied from the previous SessionRunner._create_env calls to VirtualEnv, etc, with the addition of runner. I added this if someone wants to access the calling runner instance. This might be seen as repetitive, as all these parameters are derived from runner.

Please let me know how I can improve the PR. Thank you!

[wpk-nist-gov]

Any chance someone could take a look at this?

[jayqi]

It was mentioned in #751 that this could be a solution for supporting micromamba (#680). It would be great if there were an example of that in the cookbook.

[wpk-nist-gov]

Thank you for the suggestion @jayqi! I added sessions micromamba_tests to noxfile.py and updated the cookbook example to use both conda env create and micromamba create.

[cjolowicz]

I think this merits broader discussion because you can't use the callable without accessing undocumented Nox internals.

[henryiii]

(I was thinking that, was getting worried about how much of nox's internals would be expected to be touched by users with this. #762 changed the internals a bit, and we might not have been able to do that if this was public)

[wpk-nist-gov]

Thank you for taking a look at this PR! I agree that using private/undocumented internals is less than ideal. But I still think functionality like this would be awesome. Before #762, I was myself playing with using uv via the callable backend. I'd like to take a look at #762 to get a feel of what internals have changed.

Would it make sense to propose making a minimal set of the private internals touched here public? I'd be happy to take a crack at that.

Thanks again!

[henryiii]

Let's wait till after the next release, then it might be interesting to discuss. A few more environment related changes are going in.

[wpk-nist-gov]

I think I have a better solution, that will require less interaction with nox internals. There are two things I'm trying to override with the callback.

1. venv location
2. venv creation command.

For (1), the obvious use case is using nox to manage development environments. Something like the following:

@nox.session(venv_location=".venv", ....)
def dev(session: nox.Session) -> None:
    ....

For (2), the use case is customizing the creation of conda environments. Instead of trying to hook into the virtualenv creation, you could instead just ask for the venv class to do everything (check for env reuse, delete previous venv, etc) but skip actually creating the virtualenv. In this case, it would be on the user to create the virtualenv in the session function. Something like:

@nox.session(venv_skip_create=True)
def test(session: nox.Session) -> None:

    if not session.virtualenv._reused:  # I'd like to make the "reused property public"
        session.run_always(
            "conda",
            "env",
            "create",
            "-f",
            f"py{session.python}-environment.yaml",
            "--prefix",
            session.virtualenv.location,  # might add session.location, session.location_name?
        )

    ...

where as noted, I'd like to make the reused property public, and perhaps expose some properties in session.virtualenv to session level.

I'm happy to work on this. Should I make a new PR with the above functionality? If so, should I split into 2 PR's or just one?

[henryiii]

IMO, venv location makes sense. Being able to set it to something other than .nox/{name} would be useful. Maybe that would be a good thing to start with?

[wpk-nist-gov]

Excellent! I'll put a new PR together with that functionality.