Testing your changes
Previous step: Local development setup
There are several groups of tests you can run on the Redash code. Any code that is submitted via a PR will need to pass all of them before it is merged into our master branch.
- Lint tests - very quick testing of the layout and structure of your code. Helps to ensure consistency through the code base
- Unit tests - Fairly quick testing of functions in the code. Helps catch problems without needing to run the time consuming full End-to-End (E2E) tests
- E2E tests - This runs the entire application locally in docker containers, doing all the needed (test) setup, then runs a fairly comprehensive test suite on the whole thing. Can take about 1/2 an hour to run, depending on the hardware its running on
The Lint and Unit tests both have backend (Python) and front end (javascript / typescript) tests available. The E2E tests only has one option, as it tests both the front and back end at the same time.
It's fairly quick and easy to run both the lint and unit tests together.
First, make sure you've set up a virtual environment (as per this page), then run make test:
$ make test
./bin/flake8_tests.sh
6.0.0 (mccabe: 0.7.0, pycodestyle: 2.10.0, pyflakes: 3.0.1) CPython 3.8.10 on Linux
pip 20.0.2 from /home/jc/redashvenv2/lib/python3.8/site-packages/pip (python 3.8)
0
[lots of extra stuff here]
The Python lint tests (flake8) generally show a lot of very verbose output. We should probably fix most of them some day... 😁
In that same make test run, after the flake8 Python linting it will run the Python backend unit tests:
========================= test session starts =========================
platform linux -- Python 3.8.17, pytest-5.2.1, py-1.11.0, pluggy-0.13.1
rootdir: /app, inifile: pytest.ini
plugins: cov-2.8.1, anyio-3.7.0
collected 736 items
tests/test_authentication.py ...................................
[etc]
============ 736 passed, 307 warnings in 111.48s (0:01:51) ============
At the end of those backend unit tests, it should show many tests passed and probably a lot of warnings. There should be no errors.
Next, it will then automatically run the frontend (javascript / typescript) unit tests:
yarn test
yarn run v1.22.19
$ run-s type-check jest
$ tsc --noEmit --project client/tsconfig.json
$ TZ=Africa/Khartoum jest
PASS client/app/components/ApplicationArea/ErrorMessage.test.js
PASS client/app/services/auth.test.js
[etc]
PASS client/app/pages/users/components/ReadOnlyUserProfile.test.js
PASS client/app/components/queries/ScheduleDialog.test.js
Test Suites: 13 passed, 13 total
Tests: 1 skipped, 77 passed, 78 total
Snapshots: 12 passed, 12 total
Time: 3.028s
Ran all test suites.
Done in 6.23s.
Again, this will likely show several things passing, and possibly some warnings. There should be no errors though.
Unit tests for the visualizations are run from the viz-lib directory:
$ (cd viz-lib; yarn test)
yarn run v1.22.19
$ jest
PASS src/visualizations/chart/plotly/prepareData.test.ts
PASS src/visualizations/chart/Editor/GeneralSettings.test.tsx
[etc]
PASS src/visualizations/chart/plotly/prepareLayout.test.ts
PASS src/visualizations/chart/getChartData.test.ts
Test Suites: 19 passed, 19 total
Tests: 103 passed, 103 total
Snapshots: 52 passed, 52 total
Time: 9.474s
Ran all test suites.
Done in 10.72s.
This quick test approach will leave a few Docker containers running which aren't useful for the next steps.
So, you might as well shut them down before continuing. This command does that:
$ make down
We use Cypress version 11.2.0 for End to End (E2E) testing. It mostly works ok, though can take a bit of time to get used to.
Cypress requires some GUI related libraries to be installed before it will run, so lets install those:
$ sudo apt -y install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 \
libxtst6 xauth xvfb
Also, before we run commands in Cypress (whether GUI or cli), we need to start Redash in the background with things set up for testing:
$ yarn cypress build
$ yarn cypress start
With that done, we can launch the Cypress GUI, or run the Cypress tests via the cli (a bit more reliable).
To run the tests via the cli, use this:
$ yarn cypress run
That will automatically run the full Cypress E2E tests, displaying progress as it goes. It generally takes around 1/2 an hour to run.
Alternatively, you can start the Cypress GUI using this command:
$ yarn cypress open > /dev/null 2>&1 &
The GUI looks like this:
With this version of the GUI you can't run all of the tests at once, so it's more useful for debugging of individual tests that are failing for some reason.
So, the standard approach is to do a full run of the tests from the command line, and use the GUI for investigating test failures and developing new tests.
To run an individual test file (eg for debugging purposes), then choose the E2E Testing option on the left hand side.
The window will then update to a "Browser selection" dialog:
We tend to use the Electron one by default, but testing in other browsers can be useful for compatibility purposes too.
With a browser selected, click the Start E2E Testing in Electron button to open the Cypress test selection window:
To run an individual test, select it in the list. When you click on it, Cypress will open a new window to run just that one. You can pause it, and also view the state of the in-built browser at any point in time during the run.
For debugging purposes, it is often useful to open the Developer Tools -> Console. A lot of useful debugging information is displayed there.
After the Cypress tests have completed, the docker containers it was using are still running.
To shut them down, use this:
$ yarn cypress stop