Publish precompiled wheels in Github release and add wheel index (#365)

* publish precompiled wheels * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * test on linux * Update building.yml * Update publish.yml * try to fix * trying TORCH_CUDA_ARCH_LIST=7.5 * try to fix * try fixing * move export TORCH_CUDA_ARCH_LIST=8.2 * try fix * using export TORCH_CUDA_ARCH_LIST="8.0;8.6" * Update publish.yml * using TORCH_CUDA_ARCH_LIST="8.0" * TORCH_CUDA_ARCH_LIST="7.5" * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * TORCH_CUDA_ARCH_LIST="6.2;7.2;7.5;8.7;8.9;9.0" * Update publish.yml * Update publish.yml * Update building.yml * TORCH_CUDA_ARCH_LIST="6.2" * export TORCH_CUDA_ARCH_LIST="7.2;7.5" * Update publish.yml * Update publish.yml * Update building.yml * Update publish.yml * Update publish.yml * Update building.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * trying to set TORCH_CUDA_ARCH_LIST in the setup.py file * Update setup.py * test * simplify * use ORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0" with cu118 * trying wheel build with cuda 11.7 * trying TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0" with cuda 11.7 * trying export TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0" with cuda 11.6 * trying export TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0" with cuda 11.5 * trying export TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0" with cuda 11.3 * test TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6" with cuda 11.3 * testing TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6" with cuda 11.5 * testing TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6" with cuda 11.6 * testing TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6" with cuda 11.3 * testing TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6" with cuda 11.7 * trying python 3.9 and 3.11 * Update publish.yml * Update publish.yml * trying python 3.8 * trying python 3.7 * fix python3.7 * test python 3.11 * Update publish.yml * Update publish.yml * testing python 3.7 * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * test cu113 * test cu115 * test cu116 * test cu117 * test cu118 * Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml * create 2 pipelines * change pipeline names * Update publish_source_and_no_cuda_wheel.yml * Update publish_precompiled_wheels.yml * Update publish_source_and_no_cuda_wheel.yml * trying to upload in github release * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * test BUILD_NO_CUDA=1 * test BUILD_NO_CUDA=1 * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * trying to uploade the wheels to release * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * Update publish_precompiled_wheels.yml * back to single file * Update publish.yml * Update publish.yml * Create wheels_index.html * Update wheels_index.html * deal with + * Update publish.yml * Update publish.yml * Update publish.yml * adding links to packages * document installation using a precompiled wheel * Update packages.html * Create wheels_index.html * Update wheels_index.html * trying using %2B in urls * moving index * trying to make it work with index url * add ptXcuX subpages * Update INSTALL_WIN.md * trying to reuse the build workflow * try publish on manual release * Update publish.yml * Update publish.yml * trying reusable workflow * trying to make it work * try reusable workflow * Update publish.yml * Update publish.yml * testing with few wheels * Update building.yml * Update building.yml * Update building.yml * Update publish.yml * Update building.yml * add script and github steps to generate the simple index pages * remove the html pages from the PR * remove wheels for python 3.8 and 3.9 from the releases * ruff format * Update publish.yml * fixing typos * Update generate_simple_index_pages.py * wheels -> whl * test * test * fix * some updates * fix * ninja-build * revert * fix * artifact v3 * all version with python3.10 * wheels -> whl --------- Co-authored-by: martin de la gorce <[email protected]> Co-authored-by: Ruilong Li <[email protected]>
nerfstudio-project · Sep 27, 2024 · cf4c1a0 · cf4c1a0
1 parent 721382b
commit cf4c1a0
Show file tree

Hide file tree

Showing 7 changed files with 369 additions and 46 deletions.
diff --git a/.github/workflows/building.yml b/.github/workflows/building.yml
@@ -1,26 +1,46 @@
-name: Building Wheels
+name: Build Wheels
 
-on: [workflow_dispatch]
 
+on: [workflow_call, workflow_dispatch]
 jobs:
-
-  wheel:
+  build_sdist:
+    name: Build source distribution and no binary wheel
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Strip unsupported tags in README
+      run: |
+        sed -i '/<!-- pypi-strip -->/,/<!-- \/pypi-strip -->/d' README.md
+    - name: Build sdist
+      run: BUILD_NO_CUDA=1 pipx run build --sdist
+    - name: Build wheel with no binaries
+      run:  BUILD_NO_CUDA=1 python setup.py bdist_wheel --dist-dir=dist
+    - uses: actions/upload-artifact@v3
+      with:
+        name: pypi_packages
+        path: dist/*.tar.gz
+
+
+  build_wheels:
     runs-on: ${{ matrix.os }}
     environment: production
 
     strategy:
       fail-fast: false
       matrix:
         os: [ubuntu-20.04, windows-2019]
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.10']
         torch-version: ['2.0.0', '2.1.0', '2.2.0', '2.3.0', '2.4.0']
         cuda-version: ['cu118', 'cu121', 'cu124']
+        # python-version: ['3.8']
+        # torch-version: ['2.0.0']
+        # cuda-version: ['cu118']
+
         exclude:
           - python-version: 3.12
             torch-version: 2.0.0
           - python-version: 3.12
             torch-version: 2.1.0
-
           - torch-version: 2.0.0
             cuda-version: 'cu113'
           - torch-version: 2.0.0
@@ -58,21 +78,19 @@ jobs:
           - torch-version: 2.4.0
             cuda-version: 'cu116'
           - torch-version: 2.4.0
-            cuda-version: 'cu117'
-
+            cuda-version: 'cu117'          
           - os: windows-2019
             torch-version: 2.0.0
             cuda-version: 'cu121'
 
-
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
         with:
           submodules: recursive
-
+  
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -146,3 +164,10 @@ jobs:
           python -c "import gsplat; print('gsplat:', gsplat.__version__)"
           cd ..
         shell: bash  # `ls -lah` does not exist in windows powershell
+
+        # v4 does not allow writing into the same folder, so stick with v3
+        # https://github.com/actions/upload-artifact/issues/478
+      - uses: actions/upload-artifact@v3
+        with:
+          name: compiled_wheels_python${{ matrix.python-version }}
+          path: dist/*.whl
diff --git a/.github/workflows/core_tests.yml b/.github/workflows/core_tests.yml
@@ -19,7 +19,7 @@ jobs:
           submodules: 'recursive'
 
       - name: Set up Python 3.8.12
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: "3.8.12"
       - name: Install dependencies

diff --git a/.github/workflows/generate_simple_index_pages.py b/.github/workflows/generate_simple_index_pages.py
@@ -0,0 +1,152 @@
+import requests
+import os
+import argparse
+from jinja2 import Template
+import re
+
+# Automatically get the repository name in the format "owner/repo" from the GitHub workflow environment
+GITHUB_REPO = os.getenv("GITHUB_REPOSITORY")
+
+
+def list_python_wheels():
+    # GitHub API URL for releases
+    releases_url = f"https://api.github.com/repos/{GITHUB_REPO}/releases"
+
+    response = requests.get(releases_url)
+
+    if response.status_code != 200:
+        raise Exception(
+            f"Failed to fetch releases: {response.status_code} {response.text}"
+        )
+
+    releases = response.json()
+
+    wheel_files = []
+
+    # Iterate through releases and assets
+    for release in releases:
+        assets = release.get("assets", [])
+        for asset in assets:
+            filename = asset["name"]
+            if filename.endswith(".whl"):
+                pattern = r"^(?P<name>[\w\d_.]+)-(?P<version>[\d.]+)(?P<local>\+[\w\d.]+)?-(?P<python_tag>[\w]+)-(?P<abi_tag>[\w]+)-(?P<platform_tag>[\w]+)\.whl"
+
+                match = re.match(pattern, filename)
+
+                if match:
+                    local_version = match.group("local")
+                    if local_version:
+                        local_version = local_version.lstrip(
+                            "+"
+                        )  # Return the local version without the '+' sign
+                    else:
+                        local_version = None
+                else:
+                    raise ValueError(f"Invalid wheel filename: {filename}")
+                wheel_files.append(
+                    {
+                        "release_name": release["name"],
+                        "wheel_name": asset["name"],
+                        "download_url": asset["browser_download_url"],
+                        "package_name": match.group("name"),
+                        "local_version": local_version,
+                    }
+                )
+
+    return wheel_files
+
+
+def generate_simple_index_htmls(wheels, outdir):
+    # Jinja2 template as a string
+    template_versions_str = """
+    <!DOCTYPE html>
+    <html>
+    <head><title>Python wheels links for {{ repo_name }}</title></head>
+    <body>
+    <h1>Python wheels for {{ repo_name }}</h1>
+
+    {% for wheel in wheels %}
+    <a href="{{ wheel.download_url }}">{{ wheel.wheel_name }}</a><br/>
+    {% endfor %}
+
+    </body>
+    </html>
+    """
+
+    template_packages_str = """
+    <html>
+    <body>
+    {% for package_name in package_names %}
+        <a href="{{package_name}}/">{{package_name}}</a><br/>
+    {% endfor %}
+    </body>
+    </html>
+    """
+
+    # Create a Jinja2 Template object from the string
+    template_versions = Template(template_versions_str)
+    template_packages = Template(template_packages_str)
+
+    # group the wheels by package name
+    packages = {}
+    for wheel in wheels:
+        package_name = wheel["package_name"]
+        if package_name not in packages:
+            packages[package_name] = []
+        packages[package_name].append(wheel)
+
+    # Render the HTML the list the package names
+    html_content = template_packages.render(
+        package_names=[str(k) for k in packages.keys()]
+    )
+    os.makedirs(outdir, exist_ok=True)
+    with open(os.path.join(outdir, "index.html"), "w") as file:
+        file.write(html_content)
+
+    # for each package, render the HTML to list the wheels
+    for package_name, wheels in packages.items():
+        html_page = template_versions.render(repo_name=GITHUB_REPO, wheels=wheels)
+        os.makedirs(os.path.join(outdir, package_name), exist_ok=True)
+        with open(os.path.join(outdir, package_name, "index.html"), "w") as file:
+            file.write(html_page)
+
+
+def generate_all_pages():
+    wheels = list_python_wheels()
+    if wheels:
+        print("Python Wheels found in releases:")
+        for wheel in wheels:
+            print(
+                f"Release: {wheel['release_name']}, Wheel: {wheel['wheel_name']}, URL: {wheel['download_url']}"
+            )
+    else:
+        print("No Python wheels found in the releases.")
+
+    # Generate Simple Index HTML pages the wheel with all local versions
+    generate_simple_index_htmls(wheels, outdir=args.outdir)
+
+    # group wheels per local version
+    wheels_per_local_version = {}
+    for wheel in wheels:
+        local_version = wheel["local_version"]
+        if local_version is not None and local_version not in wheels_per_local_version:
+            wheels_per_local_version[local_version] = []
+        wheels_per_local_version[local_version].append(wheel)
+
+    # create a subdirectory for each local version
+    for local_version, wheels in wheels_per_local_version.items():
+        os.makedirs(os.path.join(args.outdir, local_version), exist_ok=True)
+        generate_simple_index_htmls(
+            wheels, outdir=os.path.join(args.outdir, local_version)
+        )
+
+
+if __name__ == "__main__":
+    argparser = argparse.ArgumentParser(
+        description="Generate Python Wheels Index Pages"
+    )
+    argparser.add_argument(
+        "--outdir", help="Output directory for the index pages", default="."
+    )
+    args = argparser.parse_args()
+    generate_all_pages()
diff --git a/.github/workflows/generate_simple_index_pages.yml b/.github/workflows/generate_simple_index_pages.yml
@@ -0,0 +1,42 @@
+# This workflows will upload a Python Package using twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Update wheels index pages
+
+# trigger manually on github interface
+on:
+  workflow_dispatch:
+  push:
+    branches: [generate_wheels_simple_index_pages]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: production
+
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install requests jinja2
+      # call the script to generate the simple index pages
+      - name: Generate Simple Index Pages
+        run: python .github/workflows/generate_simple_index_pages.py --outdir ./whl
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
+
+
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./whl  # Directory where the simple index pages are located
+          destination_dir: whl  # The 'wh' folder in the GitHub Pages root