Long Read QC

Please see the documents in the docs folder for documentation on subjects other than development, testing and deployment.

• Background
• The QC process
• The QC database schema

Prepare for release

Prior to merging to the master branch:

• Update CHANGELOG.md
• Update the hardcoded version in
  • frontend/package.json
  • lang_qc/init.py
  • pyproject.toml
• If needed, update the Copyright year in frontend/src/App.vue

Install and run locally

You can install the package with pip install . from the repository's root. Alternatively you can use Poetry to install and manage a virtual environment. Poetry will demand python => 3.10 available via package manager or pyenv.

Then set two environment variables:

• DB_URL: currently refers to the mlwh's url, in the format: mysql+pymysql://user:pass@host:port/dbname
• CORS_ORIGINS: a comma-separated list of origins to allow CORS, e.g: http://localhost:3000,https://example.com:443

Finally, run the server: uvicorn lang_qc.main:app. Or uvicorn lang_qc.main:app --reload to reload the server on code changes.

Using docker-compose

Requirements

The deployment was tested with versions at least as recent as:

• Docker version 20.10.5 and 20.10.8
• docker-compose version 1.28.6 and 1.29.2

Think about using a more recent version of Ubuntu, for example focal rather than bionic. The ports you would like to expose should also be open on your server.

To run the server, create an env file with the following template:

OIDCProviderMetadataURL=<identityproviderhostURL>/.well-known/openid-configuration # e.g. https://accounts.google.com/.well-known/openid-configuration
OIDCClientID=<yourclientID> # set by the OIDC Provider
OIDCClientSecret=<yourclientsecret> # set by the OIDC Provider
ODICCryptoPassphrase=somesecurepassphrase # you can choose this
OIDCRedirectURI=https://example.com/login-redirect # must match what is set on the OIDC Provider side
CORS_ORGINS=https://example.com:<someport> # most likely not needed anymore, probably can be left blank
QCDB_URL=mysql+pymysql://<user>:<password>@<host>:<port>/<dbname>?charset=utf8mb4 # mlwh db
DB_URL=mysql+pymysql://<user>:<password>@<host>:<port>/<dbname>?charset=utf8mb4 # qc db
CERT_FOLDER=/absolute/path/to/certs/folder # must contain `cert.pem` and `key.pem`
SERVER_HOST=https://example.com:443 # can omit port if standard, used for frontend build to refer to right API server.
HTTPS_PORT=443 # port on which the deployment will be exposed.

You might want to chmod 600 /path/to/env/file as it contains passwords.

The env file can contain definitions that are user by the frontend. These definitions should be prefixed with VITE_, for example VITE_SOME_PORT="4567". To be visible to the build of the frontend container, they have to be copied to the frontend/.env file: (cat /path/to/env/file | grep VITE_) > frontend/.env

To build containers, from the root of this repository, run : Build: docker-compose --env-file /path/to/env/file build Run: docker-compose --env-file /path/to/env/file up -d

Development setup

Follow the same steps as above. Then use docker-compose.dev.yml to override docker-compose.yml:

Build:

docker-compose \
  --env-file /path/to/env/file
  -f docker-compose.yml
  -f docker-compose.dev.yml
  build

Run:

docker-compose \
  --env-file /path/to/env/file
  -f docker-compose.yml
  -f docker-compose.dev.yml
  up -d

The /lang_qc/ folder will be bind-mounted into the lang_qc docker container, and the server will hot-reload with your changes. Additionally, two databases will be spun up with the default credentials used when running pytest in the project (as of writing these will not have the schema setup by default: this is not a problem for running unit tests, but if you want to use these databases to test the web app then you must create the schemas and populate the databases with some data.).

Similarly, the /frontend/{src,public} folders will be bind-mounted into the longue_vue container, and the server will hot-reload your changes. You will still need to reload your browser to see the changes. Remember to change the url to the API in the JS code. Any change that is done in any file not in /frontend/{public,src} will require a docker-compose build.

Using Docker

This repository provides a Docker image. The image is configured to use SSL for the server, running uvicorn.

The following instructions assume you have a recent Ubuntu system with Docker installed, and your desired ports open.

Step 1

Copy over SSL certificates, put them in a folder and name them ca.pem, key.pem andcert.pem or create symlinks to them using these names.

Step 2

Create a config folder containing one file, uvicorn_env, with the following contents:

DB_URL=<insert your DB URL here>
CORS_ORIGINS=<comma-separated list of origins for which to allow CORS>

An example is:

DB_URL=mysql+pymysql://myuser:[email protected]:3306/mydbname
CORS_ORIGINS=http://localhost:3000,https://example.com:443

Step 3

Create an envfile containng the following:

export CERT_FOLDER=/path/to/cert/folder
export CONFIG_FOLDER=/path/to/config/folder
export SSL_PORT=<your desired port for the server> # e.g. SSL_PORT=443

Source this file: . envfile

Step 4

Run the server (replace devel with latest if you wish to run the master branch instead.):

docker run \
  --name npg_langqc \
  -p ${SSL_PORT}:443 \
  -v ${CERT_FOLDER}:/certs \
  -v ${CONFIG_FOLDER}:/config \
  ghcr.io/wtsi-npg/npg_langqc:devel