Skip to content

Commit

Permalink
Install scripts (#191)
Browse files Browse the repository at this point in the history
* Application bootstrap script for Linux and macOS

* Windows bootstrap script

* Creating random network name during bootstrap

Avoiding conflicts if several applications versions should exist on one machine

* Celery depends on Redis

* Fixed random string generation in PowerShell script

* Updated README

* Check DC version in GitHub Actions

* Special Docker Compose override file for CI

* Replaced xxd utility with od

xxd seems to be not installed by default in at least some of the major Linux distributions

* Stopping containers before removal

* od's -w flag is incompatible with macOS version of Bash
  • Loading branch information
ivnglkv authored Apr 9, 2024
1 parent a233f4e commit fc9e05b
Show file tree
Hide file tree
Showing 11 changed files with 320 additions and 36 deletions.
1 change: 1 addition & 0 deletions .env.dist
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
JDI_VERSION_LABEL=latest
SELENOID_PARALLEL_SESSIONS_COUNT=4
2 changes: 1 addition & 1 deletion .github/workflows/pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
lint:
runs-on: ubuntu-latest
env:
COMPOSE_FILE: docker-compose.yaml:docker-compose.override.dev.yaml
COMPOSE_FILE: docker-compose.yaml:docker-compose.override.ci.yaml
SELENOID_PARALLEL_SESSIONS_COUNT: 2
steps:
- name: Checkout repository
Expand Down
26 changes: 12 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,31 +168,29 @@ To validate models quality we use test web-pages, placed in directory **notebook
# Docker
## Take docker image from github:
### Release version
#### macOS
#### macOS / Linux
```shell
export SELENOID_PARALLEL_SESSIONS_COUNT=$(sysctl -n hw.ncpu) && docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:latest --force && curl --output docker-compose.yaml --url https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/docker-compose.yaml && curl --output browsers.json https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/browsers.json && docker-compose up
```
#### Linux
```shell
export SELENOID_PARALLEL_SESSIONS_COUNT=$(nproc) && docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:latest --force && curl --output docker-compose.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/docker-compose.yaml && curl --output browsers.json https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/browsers.json && docker compose up
curl --output jdi-bootstrap.sh https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/scripts/bootstrap.sh && \
bash jdi-bootstrap.sh
```

#### Windows
```shell
set SELENOID_PARALLEL_SESSIONS_COUNT=%NUMBER_OF_PROCESSORS%&& docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:latest --force && curl.exe --output docker-compose.yaml --url https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/docker-compose.yaml && curl.exe --output browsers.json --url https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/browsers.json && docker compose up
curl --output jdi-bootstrap.ps1 https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/master/scripts/bootstrap.ps1 && ^
powershell -executionpolicy bypass .\jdi-bootstrap.ps1
```

### Development version
#### macOS
#### macOS / Linux
```shell
export SELENOID_PARALLEL_SESSIONS_COUNT=$(sysctl -n hw.ncpu) && docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:develop --force && curl --output docker-compose.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.yaml && curl --output docker-compose.override.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.override.prebuilt-dev.yaml && curl --output browsers.json https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/browsers.json && docker-compose up
```
#### Linux
```shell
export SELENOID_PARALLEL_SESSIONS_COUNT=$(nproc) && docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:develop --force && curl --output docker-compose.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.yaml && curl --output docker-compose.override.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.override.prebuilt-dev.yaml && curl --output browsers.json https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/browsers.json && docker compose up
curl --output jdi-bootstrap.sh https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/scripts/bootstrap.sh && \
bash jdi-bootstrap.sh -b develop -l develop
```

#### Windows
```shell
set SELENOID_PARALLEL_SESSIONS_COUNT=%NUMBER_OF_PROCESSORS%&& docker rm --force jdi-qasp-ml-api && docker image rm ghcr.io/jdi-testing/jdi-qasp-ml:develop --force && curl.exe --output docker-compose.yaml --url https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.yaml && curl.exe --output docker-compose.override.yaml https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/docker-compose.override.prebuilt-dev.yaml && curl.exe --output browsers.json --url https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/browsers.json && docker compose up
curl --output jdi-bootstrap.ps1 https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/develop/scripts/bootstrap.ps1 && ^
powershell -executionpolicy bypass .\jdi-bootstrap.ps1 -Branch develop -Label develop
```

## Installing version from any other repository branch:
Expand Down
14 changes: 14 additions & 0 deletions docker-compose.override.arbitrary-tag.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
services:
api:
image: "ghcr.io/jdi-testing/jdi-qasp-ml:${JDI_VERSION_LABEL:?Define JDI QASP ML Docker image tag to use}"
environment:
ENV: LOCAL
depends_on: !override
- selenoid
- celery
- redis

celery:
image: "ghcr.io/jdi-testing/jdi-qasp-ml:${JDI_VERSION_LABEL:?Define JDI QASP ML Docker image tag to use}"

mongodb: !reset []
30 changes: 30 additions & 0 deletions docker-compose.override.ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Disclaimer
# We don't need this file basically and could have just used
# docker-compose.override.dev.yaml. But the problem is that _at this moment_
# Docker Compose in our GitHub Actions runner doesn't know how to deal with
# !reset construct.

services:
selenoid:
ports:
- "${SELENOID_PORT:-4445}:4444"

api:
image: !reset null
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/jdi-qasp-ml
environment:
ENV: LOCAL
depends_on: !override
- selenoid
- celery
- redis

celery:
image: !reset null
build:
context: .
dockerfile: Dockerfile
8 changes: 8 additions & 0 deletions docker-compose.override.dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,17 @@ services:
dockerfile: Dockerfile
volumes:
- .:/jdi-qasp-ml
environment:
ENV: LOCAL
depends_on: !override
- selenoid
- celery
- redis

celery:
image: !reset null
build:
context: .
dockerfile: Dockerfile

mongodb: !reset []
6 changes: 0 additions & 6 deletions docker-compose.override.prebuilt-dev.yaml

This file was deleted.

10 changes: 0 additions & 10 deletions docker-compose.override.rc.yaml

This file was deleted.

19 changes: 14 additions & 5 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ x-logging: &logging
x-restart-policy: &restart
restart: unless-stopped

x-labels: &labels
labels:
org.jdi-qasp-ml.project: "jdi-qasp-ml"
org.jdi-qasp-ml.version_label: "${JDI_VERSION_LABEL:-latest}"

services:
selenoid:
image: "aerokube/selenoid:latest"
Expand All @@ -26,7 +31,7 @@ services:
- "${JDI_DEFAULT_NETWORK_NAME:-jdi-qasp-ml-default}"
- "-limit"
- "${SELENOID_PARALLEL_SESSIONS_COUNT:?set it to the number of parallel running threads supported by your processor (-2 optionally)}"
<<: *restart
<<: [*restart, *labels]

api:
image: "ghcr.io/jdi-testing/jdi-qasp-ml:latest"
Expand All @@ -40,27 +45,31 @@ services:
- "${APP_EXPOSE_PORT:-5050}:5050"
environment:
- SELENOID_PARALLEL_SESSIONS_COUNT=$SELENOID_PARALLEL_SESSIONS_COUNT
<<: [*logging, *restart]
<<: [*logging, *restart, *labels]

celery:
image: "ghcr.io/jdi-testing/jdi-qasp-ml:latest"
command: bash -c "bash start_celery.sh"
<<: [*logging, *restart]
depends_on:
- redis
<<: [*logging, *restart, *labels]

redis:
image: "redis:alpine"
command: redis-server --maxmemory 2048mb --maxmemory-policy allkeys-lru
<<: [*logging, *restart]
<<: [*logging, *restart, *labels]

mongodb:
image: mongo:5.0
volumes:
- mongo-data:/data/db
<<: [*logging, *restart]
<<: [*logging, *restart, *labels]

networks:
default:
name: ${JDI_DEFAULT_NETWORK_NAME:-jdi-qasp-ml-default}
<<: *labels

volumes:
mongo-data:
<<: *labels
60 changes: 60 additions & 0 deletions scripts/bootstrap.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
param(
[Parameter(HelpMessage="Branch used as a source for downloading files")]
[string]$Branch = 'master',
[Parameter(HelpMessage="Amount of allowed parallel Selenium sessions")]
[int]$CoresToUse = (Get-WmiObject -Class Win32_Processor).NumberOfLogicalProcessors,
[Parameter(HelpMessage="Docker image label to pull")]
[string]$Label = 'latest',
[Parameter(HelpMessage="Docker Compose pull policy")]
[string]$PullPolicy = 'always',
[Parameter(HelpMessage="Run Docker Compose in 'detached' mode")]
[switch]$Detached,
[Parameter(HelpMessage="Do not remove old Docker resources with the same label")]
[switch]$NoCleanup
)

function Download-File {
param(
[string]$Url,
[string]$OutputPath
)
$webClient = New-Object System.Net.WebClient
$webClient.DownloadFile($Url, $OutputPath)
}

# Remove old Docker resources if cleanup flag is not set
if (-not $NoCleanup) {
$labelSelector = "org.jdi-qasp-ml.version_label=$Label"
docker stop $(docker ps -qf "label=$labelSelector") 2>$null
docker container rm -f $(docker ps -qaf "label=$labelSelector") 2>$null
docker volume rm -f $(docker volume ls -qf "label=$labelSelector") 2>$null
docker network rm -f $(docker network ls -qf "label=$labelSelector") 2>$null
Remove-Item -Path "docker-compose.yaml", "docker-compose.override.yaml", "browsers.json", ".env" -ErrorAction SilentlyContinue
}

$RepoUrl = "https://raw.githubusercontent.com/jdi-testing/jdi-qasp-ml/$Branch"

Download-File -Url ("$RepoUrl/docker-compose.yaml") -OutputPath "docker-compose.yaml"
Download-File -Url ("$RepoUrl/browsers.json") -OutputPath "browsers.json"

$randomNumber = Get-Random -Minimum 0 -Maximum 99999999
$hash = [System.Security.Cryptography.SHA256]::Create().ComputeHash([System.Text.Encoding]::UTF8.GetBytes($randomNumber.ToString()))
$randomString = ([System.BitConverter]::ToString($hash) -replace '-', '').Substring(0, 8)
$NetworkName = ("jdi-qasp-ml-$randomString")

@"
JDI_VERSION_LABEL=$Label
SELENOID_PARALLEL_SESSIONS_COUNT=$CoresToUse
JDI_DEFAULT_NETWORK_NAME=$NetworkName
"@ | Set-Content -Path ".env"

if ($Label -ne 'latest') {
Download-File -Url ("$RepoUrl/docker-compose.override.arbitrary-tag.yaml") -OutputPath "docker-compose.override.yaml"
}

# Start Docker Compose
if ($Detached) {
docker compose up -d --pull $PullPolicy
} else {
docker compose up --pull $PullPolicy
}
Loading

0 comments on commit fc9e05b

Please sign in to comment.