Numpy2.0 migration

[smiet]

The container build is failing because the SPEC and VMEC2000 compilation step are failing.

The migration guide to numpy2.0 is given here.

I will work on getting SPEC up and running with numpy 2.0, @landreman can you have a look at VMEC? from this link it looks like this is what should be added:

[build-system]
build-backend = ...
requires = [
    # Note for packagers: this constraint is specific to wheels
    # for PyPI; it is also supported to build against 1.xx still.
    # If you do so, please ensure to include a `numpy<2.0`
    # runtime requirement for those binary packages.
    "numpy>=2.0.0rc1",
    ...
]

[project]
dependencies = [
    "numpy>=1.23.5",
]

[landreman]

Hm, I tried that change in the vmec2000 repo, but there are various errors in the vmec2000 CI now. Looping in @mbkumar @andrewgiuliani @daringli @rogeriojorge in case anyone can help.

[smiet]

Now thinking we might want to temporarily force numpy<2.0 in all runners, there are other dependencies like f90wrap that use the deprecated oldest-supported-numpy that need to be patched, as well as that a new pyoculus needs to be uploaded to pypi as we resolve all the dependencies

[smiet]

Update: f90wrap needs to be adapted as well to make spec (and simsopt) run with numpy 2.0. Working with the upstream to merge a pull-request.

In the meantime we can try to force the runners to use a 1.x version of numpy, as I tried in PR #429

Dependencies that still need to be patched:

• F90wrap @smiet
• • jameskermode branch updated to numpy2.0
• • zhucaoxiang branch main_off updated to 2.0 (awaiting PR approval)
• SPEC @smiet
• pyoculus @smiet
• • PR submitted waiting for @zhisong (update to numpy2.0 zhisong/pyoculus#7)
• VMEC2000 @mkumbar @landreman @andrewgiuliani @daringli @rogeriojorge
• nptyping @mbkumar
• randomgen @florianwechsung (has not added numpy2.0 support, 'fails on examples/2_Intermediate/stage_two_optimization_stochastic.py' Numpy 2.0 compatibility bashtage/randomgen#360 )

[smiet]

nptyping does not seem to have any compatibility with numpy2.0, and when installed under 2.0 raises an exception upon import.
Currently it's dependency is set to numpy<2.0, such that installing simsopt will downgrade numpy to 1.26.

Can we replace the dependency? @mbkumar, I see you added it, what does nptyping allow us to do that numpy.typing cannot provide?

[mbkumar]

nptyping was the first to come out with type hinting for numpy data structures. It was added more than couple of years ago and I haven't kept up to date with the progress made by numpy in supporting type hinting. I think this is a good time to look at the type hinting landscape and see if we still need to use nptyping library.

[mbkumar]

Looks like development on nptying ceased last year. We should switch to numpy.typing. I'll work on it ASAP.

[smiet]

Tests are failing because of a few deprecated numpy uses. See the branch ml/vmec_update_for_numpy_2 and it's tests for progress.

@florianwechsung the randomgen package is not updated for numpy 2.0, could np.random be used, or is it essential for the stochastic optimizations?

[mbkumar]

@smiet
What is the status of f90wrap? Do we still need to use CZhu's repo? His repo is not passing python 3.11 test.

[smiet]

@mbkumar You need to use the main_off branch, this is up-to-date with remote, works with numpy2.0 and has an open pull request.

For now this branch is needed for the SPEC installation yes, the upstream branch fails because variables contain python keywords.

[smiet]

Sorry @mbkumar, the new branch also needs (for the time being) the numpy2_fix branch of SPEC. The PR is awaiting review.

See the changes I made the ml/vmec_update_for_numpy_2 workflows

    - name: Download SPEC
      run: git clone -b numpy2_fix --depth=1 https://github.com/PrincetonUniversity/SPEC

Hopefully this PR will be resolved tomorrow.

[mbkumar]

@smiet Thanks for letting me know. I'll wait till the PRs are merged. Please ping me once the PRs are merged.

[smiet]

@mbkumar You are a contributor to SPEC, If you give the PR a quick glance and agree with the changes, you can approve and merge.

[smiet]

@mbkumar The new version of SPEC is merged into master

[florianwechsung]

Tests are failing because of a few deprecated numpy uses. See the branch ml/vmec_update_for_numpy_2 and it's tests for progress.

@florianwechsung the randomgen package is not updated for numpy 2.0, could np.random be used, or is it essential for the stochastic optimizations?

I vaguely recall using randomgen because the standard numpy.random does not satisfy certain independence conditions when running in parallel. I'm travelling this week but will have a look on the weekend. Please ping me again if I don't come back to you by Monday.

[mbkumar]

@florianwechsung,

Looks like most of the randomgen codebase has been merged into numpy. numpy is now supplying PCG64 and its successor PCG64DXSM. The successor seems to solve the parallel issue.

If you don't have time, I can take the lead in replacingrandomgen.PCG64 with numpy.random.PCG64DXSM.