From 22e29868c56d65e51210865b2f6d3784873dec59 Mon Sep 17 00:00:00 2001 From: Ben Alkov Date: Fri, 15 Dec 2023 10:49:11 -0500 Subject: [PATCH 1/2] docs(test): writing integration tests also describes the local development process: - processing test data with cachi2 - running pytest Signed-off-by: Ben Alkov --- docs/writing_integration_tests.md | 51 +++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/writing_integration_tests.md diff --git a/docs/writing_integration_tests.md b/docs/writing_integration_tests.md new file mode 100644 index 000000000..6450808ff --- /dev/null +++ b/docs/writing_integration_tests.md @@ -0,0 +1,51 @@ +# Writing integration tests + +## Developing the test source + +If you are trying to write a cachi2 integration test, and wish to run cachi2 +against a local source repo, assuming that + +- the image "localhost/cachi2:latest" exists and is a valid cachi2 container + image +- your test source is in "~/temp/cachi2-test", **and it is a valid git repo** + (i.e. a valid ".git" directory is present) + +executing e.g. + +```bash +podman run --rm -ti -v "~/temp/cachi2-test:~/temp/cachi2-test:z" -w "~/temp/cachi2-test" localhost/cachi2:latest +``` + +*should* give you a way to properly process the test source with cachi2 + +## Running pytest locally + +Once you have working test sources, you'll need to commit and push them +somewhere that pytest can clone them from. + +We *strongly* recommend making a fork, specific to your test, from one of the +repos found under the [cachito test repos][] GitHub org (note that there +are *many* repos there, covering all of the package managers which cachi2 and +Cachito support) [^1]. + +Once you have a fork, push to a new branch named after your scenario - now +pytest will have a proper commit hash in a repo to which you have complete +access (once your test is complete and passing, you can simply open a PR against +the repo in the [cachito-testing][] org). + +At this point, you should be able to test locally. + +## Running the test suite + +It's a good idea to run the whole cachi2 integration test suite, just to make +sure everything still works properly. The command for this, *from inside the +cachi2 repo* is `tox -e integration`. You can also provide +`CACHI2_IMAGE=localhost/cachi2:latest` if you already have a current cachi2 +container. e.g. `CACHI2_IMAGE=localhost/cachi2:latest tox -e integration` - this +will save a lot of time, as otherwise the image will be rebuilt from scratch. + +[^1]: If you **really** can't find an existing repo which is related to your test +scenario, ping a maintainer and explain your situation. + +[cachito-testing]: https://github.com/orgs/cachito-testing +[cachito test repos]: https://github.com/orgs/cachito-testing/repositories?type%3Dsource From d89a07fdd85e9acb9b1460c32950ecd7c32ef3dc Mon Sep 17 00:00:00 2001 From: Ben Alkov Date: Fri, 22 Mar 2024 13:54:30 -0400 Subject: [PATCH 2/2] DRAFT: walkthrough To be squashed after review --- docs/writing_integration_tests.md | 46 +++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/writing_integration_tests.md b/docs/writing_integration_tests.md index 6450808ff..c63925b40 100644 --- a/docs/writing_integration_tests.md +++ b/docs/writing_integration_tests.md @@ -35,6 +35,52 @@ the repo in the [cachito-testing][] org). At this point, you should be able to test locally. +### A walkthrough + +1. Fork a test repo under [cachito-testing][] which uses the package manager you + need to test, and/or is otherwise related +1. Clone the new fork to your local machine +1. Create a new branch in the repo, named to reflect the purpose of your test(s) +1. The goal here is to create what looks like a real + Go/Javascript/Python/whichever project, but simplified enough to *only* include + the required files which cachi2 needs to find and resolve dependencies +1. Commit the test code, and push it to your fork in GitHub. Take note of the + commit hash. + +#### Testing your tests + +1. Add your pytest scenarios to the appropriate integration test source file + under 'cachi2/tests/integration'. You'll need to put the following in + `utils.TestParameters` for your test case + + - `repo="https://github.com/cachito-testing/cachito-pip-without-deps.git"` # this will be the name of the test repo you forked + - `ref="3fe2fc3cb8ffa36317cacbd9d356e35e17af2824"` # this will be the commit hash noted previously + - `packages=({"path": ".", "type": "pip"},)` # for pip, for example + - `check_vendor_checksums=False` # or `True`, depending on the scenario + +1. In your local **cachi2** repo, make sure you have 'pytest' and 'jsonschema' + pip-installed in the cachi2 venv, so that you can run pytest without tox. Without + `tox`, it's **much** easier to run pytest with *only* your selected test + scanarios, and get useful, accessible logs +1. Bonus: pip-install 'pytest-html' in the venv, and add (e.g.) + `--html=path_to_reports/pytest-report.html` to your `pytest` command line + for a very nicely formatted HTML report +1. To summarize + + - You're in the cachi2 repo directory + - Your test scenario's source is in 'tests/integration/test_foo.py' + - Your test scenario is called `test_foo_package`, and its pytest 'id' is + `foo_incorrect_checksum` + +1. Now run + + ```bash + CACHI2_IMAGE=localhost/cachi2:latest pytest -rA -vvvv --confcutdir=tests/integration --log-cli-level=DEBUG tests/integration/test_foo.py::test_foo_package[foo_incorrect_checksum] + ``` + + which will run *only* 'tests/integration/test_foo.py::test_foo_package' with + it's "foo_incorrect_checksum" ID'd parameter set with a pre-built cachi2 container + ## Running the test suite It's a good idea to run the whole cachi2 integration test suite, just to make