Update python up to 3.10 and github action versions (#1310)

* Update python up to 3.10 and github actions * Fix python-version * Fix issue with Mapping in python 3.10 * Fix parse_version function for 0.1.dev1_g8efdbf5 case * Trying to fix borken github CI * Trying 3.9 for main.yml CI workflow * Revert "Trying 3.9 for main.yml CI workflow" This reverts commit 386c67f. * Revert "Trying to fix borken github CI" This reverts commit 0c05fd9. * Update pytest * Revert "Revert "Trying to fix borken github CI"" This reverts commit d2326b9. * debug_call.sh * Debug issue with CI * Debug issue with CI * pstree * ps aux * -n 0 for pytest * Update checkout and setup-python actions * Update cache action * Trying to run unit-test wihout postgreesql.kill() * only run builder tests * Revert "only run builder tests" This reverts commit 41a98c2. * strace in debug_call.sh * sudo * Refactor debug_call.sh * Run pytests * Install gordo * scripts/tests.sh * Run on xdist * formatting tests * Fix formatting unit-tests * Fix docker tests * Do not use xdist for CI tests * dockertest mark right condition * Trying to fix unit-tests * Fix allelse test * Debug tests.sh * PYTHONPATH * not dockertest * Try to fix not dockertest condition * Trying to fix issue with not dockertest * Try to fix tests * python 3.10 downgrade github actions * Remove debug code * rm debug_call.sh * Run 3.9 and 3.10 * Do not test for 3.9 * Fix syntax error in main.yml GitHub action * Remove -x argument from bash in main.yml GitHub action * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md. Update condition for main GitHub actions * Attempt to use xdist for slow tests * xdist and PYTHONPATH optional * Fix for usages --------- Co-authored-by: Roman Ivaniuk <[email protected]>
equinor · Apr 11, 2023 · 8a5b671 · 8a5b671
1 parent 932116a
commit 8a5b671
Show file tree

Hide file tree

Showing 12 changed files with 185 additions and 199 deletions.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -1,23 +1,30 @@
 name: CI
 
 on:
+  pull_request:
+    paths-ignore:
+      - '**.md'
   push:
+    branches:
+      - 'master'
+    paths-ignore:
+      - '**.md'
 
 jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
         os: [ubuntu-latest]
-        component: [builder, cli, client, machine, reporters, serializer, server, util, workflow, formatting, allelse]
-        python-version: [3.9]
+        component: [builder, cli, client, machine, reporters, serializer, server, util, workflow, formatting, allelse, docker]
+        python-version: ["3.10"]
     steps:
       - uses: actions/checkout@v1
 
       - uses: actions/setup-python@v1
         with:
           python-version: ${{ matrix.python-version }}
-          architecture: 'x64'
+          architecture: "x64"
 
       - uses: actions/cache@v1
         if: startsWith(runner.os, 'Linux')
@@ -29,9 +36,8 @@ jobs:
 
       - name: Install
         run: |
-          pip install --upgrade "pip>=22.2.2,<23.0.0"
           pip install -r requirements/full_requirements.txt
           pip install -r requirements/test_requirements.txt
 
       - name: Test ${{ matrix.component }}
-        run: python setup.py test${{ matrix.component }}
+        run: bash scripts/tests.sh -n -p ${{ matrix.component }}
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -12,18 +12,17 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        python-version: [3.9]
+        python-version: ["3.10"]
     steps:
       - uses: actions/checkout@v1
 
       - uses: actions/setup-python@v1
         with:
           python-version: ${{ matrix.python-version }}
-          architecture: 'x64'
+          architecture: "x64"
 
       - name: Install deps
         run: |
-          pip install --upgrade "pip>=22.2.2,<23.0.0"
           pip install -r requirements/full_requirements.txt
 
       - name: Build

diff --git a/Dockerfile b/Dockerfile
@@ -1,5 +1,5 @@
 # Gordo base image
-FROM python:3.9-bullseye as builder
+FROM python:3.10-bullseye as builder
 
 # Copy source code
 COPY . /code
@@ -17,7 +17,7 @@ RUN cat /code/requirements/full_requirements.txt | grep tensorflow== > /code/pre
     && cat /code/requirements/full_requirements.txt | grep scipy== >> /code/prereq.txt \
     && cat /code/requirements/full_requirements.txt | grep catboost== >> /code/prereq.txt
 
-FROM python:3.9-slim-bullseye
+FROM python:3.10-slim-bullseye
 
 # Nonroot user for running CMD
 RUN groupadd -g 999 gordo && \

diff --git a/README.md b/README.md
@@ -19,19 +19,6 @@
 </div>
 
 ---
-
-## Table of content
-* [About](#About)
-* [Examples](#Examples)
-* [Install](#Install)
-* [Uninstall](#Uninstall)
-* [Developer manual](#Developer-manual)
-    * [How to prepare working environment](#How-to-prepare-working-environment)
-        * [How to update packages](#How-to-update-packages)
-    * [How to run tests locally](#How-to-run-tests-locally)
-        * [Tests system requirements](#Tests-system-requirements)
-        * [Run tests](#Run-tests)
-
 ## About
 
 Gordo fulfills the role of inhaling config files and supplying components to the pipeline of:
@@ -40,15 +27,19 @@ Gordo fulfills the role of inhaling config files and supplying components to the
 2. Training model
 3. Serving model
 
----
-
-## Examples
+## Components
 
-See our [example](./examples) notebooks for how to develop with `gordo` locally.
+* [gordo-controller](https://github.com/equinor/gordo-controller/) - Kubernetes controller for the Gordo CRD.
+* [gordo-core](https://github.com/equinor/gordo-core/) - Gordo core library.
+* [gordo-client](https://github.com/equinor/gordo-client/) - Gordo server's client. It is able to make predictions from deployed models.
 
 ---
+## Install
+
+[gordo-helm](https://github.com/equinor/gordo-helm) - you can use [gordo](https://github.com/equinor/gordo-helm/tree/main/charts/gordo) helm chart from this repository to deploy gordo infrastructure to your Kubernetes cluster. 
+
+### Python package 
 
-## Install 
 `pip install --upgrade gordo`  
 
 With additional extras:
@@ -57,26 +48,34 @@ With additional extras:
 Bleeding edge:  
 `pip install git+https://github.com/equinor/gordo.git`
 
-## Uninstall
-`pip uninstall gordo`
 
 ## Developer manual
+
 This section will explain how to start development of Gordo.
 
-### How to prepare working environment
-- Install pip-tools
+### Setup
+
+Create and activate a virtual environment first. As a default option, it can be [venv](https://docs.python.org/3/library/venv.html) module.
+
+Install pip-tools
 ```
 pip install --upgrade pip
 pip install --upgrade pip-tools
 ```
 
-- Install requirements
+Install requirements
 ```
 pip install -r requirements/full_requirements.txt
 pip install -r requirements/test_requirements.txt
 ```
 
+Install package:
+```
+python3 setup.py install
+```
+
 #### How to update packages
+
 Note: you have to install `pip-tools` version higher then `6` for requirements to have same multi-line output format.
 
 To update some package in `full_requirements.txt`:
@@ -86,22 +85,26 @@ To update some package in `full_requirements.txt`:
 pip-compile --upgrade --output-file=full_requirements.txt mlflow_requirements.in postgres_requirements.in requirements.in  
 ```
 
-### How to run tests locally
+### Examples
+
+See our [example](./examples) notebooks for how to develop with `gordo` locally.
 
-#### Tests system requirements
-To run tests it's required for your system to has (note: commands might differ from your OS):
-- Running docker process;
-- Available 5432 port for postgres container 
-(`postgresql` container is used, so better to stop your local instance for tests running). 
+### How to run tests locally
 
-#### Run tests
 List of commands to run tests can be found [here](/setup.cfg).
 Running of tests takes some time, so it's faster to run tests in parallel:
 ```
-python3 setup.py test
+pytest -n auto -m 'not dockertest' --ignore benchmarks
+```
+Run docker-related tests:
 ```
+pytest -m 'dockertest'
+```
+
+> **_NOTE:_**  To run tests it's required for your system to has (note: commands might differ from your OS):
+> - Running docker daemon.
+> - Available 5432 port for `postgres` container.
 
 > **_NOTE:_** this example is for Pycharm IDE to use `breakpoints` in the code of the tests.  
 > On the configuration setup for test running add to `Additional arguments:` in `pytest` 
 > section following string: `--ignore benchmarks --cov-report= --no-cov ` 
-> or TEMPORARY remove `--cov-report=xml` and `--cov=gordo` from `pytest.ini` file.
diff --git a/docker_push.sh b/docker_push.sh
diff --git a/gordo/machine/validators.py b/gordo/machine/validators.py
@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-import collections
 import copy
 import re
 import datetime
@@ -8,6 +7,8 @@
 import dateutil.parser
 import logging
 
+from collections.abc import Mapping
+
 from gordo.serializer import from_definition
 from gordo_core.sensor_tag import SensorTag
 
@@ -162,7 +163,7 @@ def fix_runtime(runtime_dict):
     # We must also limit/request errors
 
     for key, val in runtime_dict.items():
-        if isinstance(val, collections.Mapping):
+        if isinstance(val, Mapping):
             resource = val.get("resources")
             if resource:
                 runtime_dict[key]["resources"] = fix_resource_limits(resource)

diff --git a/gordo/util/version.py b/gordo/util/version.py
@@ -32,7 +32,7 @@ def get_version(self):
         return self.special.value
 
 
-release_re = re.compile(r"^(\d{1,5})(\.(\d+)(\.(\d+)(.*?)?)?)?$")
+release_re = re.compile(r"^(\d{1,5})(\.(\d+)((\.(\d+))?(.*?)?)?)?$")
 
 
 @dataclass(frozen=True)
@@ -119,7 +119,7 @@ def parse_version(
         return GordoPR(number)
     m = release_re.match(gordo_version)
     if m:
-        (major, _, minor, _, patch, suffix) = m.groups()
+        (major, _, minor, _, _, patch, suffix) = m.groups()
         return GordoRelease(
             int(major),
             int(minor) if minor else None,

diff --git a/requirements/test_requirements.in b/requirements/test_requirements.in
@@ -1,12 +1,12 @@
 -c full_requirements.txt
 docker>=4.0,<7.0
-pytest~=6.2
-pytest-xdist~=2.3
+pytest~=7.2
+pytest-xdist~=3.2
 pytest-mock~=3.6
 pytest-mypy~=0.9
-pytest-timeout~=1.4
-pytest-cov~=2.12
-pytest-benchmark~=3.4
+pytest-timeout~=2.1
+pytest-cov~=4.0
+pytest-benchmark~=4.0
 mock~=4.0
 responses~=0.13
 # Due to packaging>22.0 in black 23.0, azureml-core~=1.49 requires packaging<22.0