-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs(test): writing integration tests #430
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
# Writing integration tests | ||
|
||
## Developing the test source | ||
|
||
If you are trying to write a cachi2 integration test, and wish to run cachi2 | ||
against a local source repo, assuming that | ||
|
||
- the image "localhost/cachi2:latest" exists and is a valid cachi2 container | ||
image | ||
- your test source is in "~/temp/cachi2-test", **and it is a valid git repo** | ||
(i.e. a valid ".git" directory is present) | ||
|
||
executing e.g. | ||
|
||
```bash | ||
podman run --rm -ti -v "~/temp/cachi2-test:~/temp/cachi2-test:z" -w "~/temp/cachi2-test" localhost/cachi2:latest | ||
``` | ||
|
||
*should* give you a way to properly process the test source with cachi2 | ||
|
||
## Running pytest locally | ||
|
||
Once you have working test sources, you'll need to commit and push them | ||
somewhere that pytest can clone them from. | ||
|
||
We *strongly* recommend making a fork, specific to your test, from one of the | ||
repos found under the [cachito test repos][] GitHub org (note that there | ||
are *many* repos there, covering all of the package managers which cachi2 and | ||
Cachito support) [^1]. | ||
|
||
Once you have a fork, push to a new branch named after your scenario - now | ||
pytest will have a proper commit hash in a repo to which you have complete | ||
access (once your test is complete and passing, you can simply open a PR against | ||
the repo in the [cachito-testing][] org). | ||
eskultety marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+23
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is summarized later in the walkthrough in a step-by-step fashion. Can we merge this into the walkthrough so that it doesn't feel like repeating the same information with slightly different wording? |
||
|
||
At this point, you should be able to test locally. | ||
|
||
### A walkthrough | ||
|
||
1. Fork a test repo under [cachito-testing][] which uses the package manager you | ||
need to test, and/or is otherwise related | ||
1. Clone the new fork to your local machine | ||
1. Create a new branch in the repo, named to reflect the purpose of your test(s) | ||
1. The goal here is to create what looks like a real | ||
Go/Javascript/Python/whichever project, but simplified enough to *only* include | ||
the required files which cachi2 needs to find and resolve dependencies | ||
1. Commit the test code, and push it to your fork in GitHub. Take note of the | ||
commit hash. | ||
|
||
#### Testing your tests | ||
|
||
1. Add your pytest scenarios to the appropriate integration test source file | ||
under 'cachi2/tests/integration'. You'll need to put the following in | ||
`utils.TestParameters` for your test case | ||
|
||
- `repo="https://github.com/cachito-testing/cachito-pip-without-deps.git"` # this will be the name of the test repo you forked | ||
- `ref="3fe2fc3cb8ffa36317cacbd9d356e35e17af2824"` # this will be the commit hash noted previously | ||
- `packages=({"path": ".", "type": "pip"},)` # for pip, for example | ||
- `check_vendor_checksums=False` # or `True`, depending on the scenario | ||
|
||
1. In your local **cachi2** repo, make sure you have 'pytest' and 'jsonschema' | ||
pip-installed in the cachi2 venv, so that you can run pytest without tox. Without | ||
`tox`, it's **much** easier to run pytest with *only* your selected test | ||
scanarios, and get useful, accessible logs | ||
1. Bonus: pip-install 'pytest-html' in the venv, and add (e.g.) | ||
`--html=path_to_reports/pytest-report.html` to your `pytest` command line | ||
for a very nicely formatted HTML report | ||
1. To summarize | ||
|
||
- You're in the cachi2 repo directory | ||
- Your test scenario's source is in 'tests/integration/test_foo.py' | ||
- Your test scenario is called `test_foo_package`, and its pytest 'id' is | ||
`foo_incorrect_checksum` | ||
|
||
1. Now run | ||
|
||
```bash | ||
CACHI2_IMAGE=localhost/cachi2:latest pytest -rA -vvvv --confcutdir=tests/integration --log-cli-level=DEBUG tests/integration/test_foo.py::test_foo_package[foo_incorrect_checksum] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Running with an already built image IMO only makes sense when your debugging an issue, but from a developer's perspective you just added some code to cachi2 that you may have tested locally outside container environment, so you actually want the image to be rebuilt for tests. By all means we have to document the env variable, I just don't think that what you're proposing in the guideline is going to be the case for most invocations during test development. |
||
``` | ||
|
||
which will run *only* 'tests/integration/test_foo.py::test_foo_package' with | ||
it's "foo_incorrect_checksum" ID'd parameter set with a pre-built cachi2 container | ||
|
||
## Running the test suite | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we need another section to describe some general good practices when it comes to running our integration test suite, the contents can live happily in the previous one IMO. |
||
|
||
It's a good idea to run the whole cachi2 integration test suite, just to make | ||
sure everything still works properly. The command for this, *from inside the | ||
cachi2 repo* is `tox -e integration`. You can also provide | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we also want to mention the |
||
`CACHI2_IMAGE=localhost/cachi2:latest` if you already have a current cachi2 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's good you mention this, but it feels weird that you actually explain the variable after the walkthrough section (next commit) which comes before this one. |
||
container. e.g. `CACHI2_IMAGE=localhost/cachi2:latest tox -e integration` - this | ||
will save a lot of time, as otherwise the image will be rebuilt from scratch. | ||
|
||
[^1]: If you **really** can't find an existing repo which is related to your test | ||
scenario, ping a maintainer and explain your situation. | ||
|
||
[cachito-testing]: https://github.com/orgs/cachito-testing | ||
[cachito test repos]: https://github.com/orgs/cachito-testing/repositories?type%3Dsource |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need
-it
if we're only executing commands with cachi2 rather than trying to run an interactive session (which also would not work without overriding the entry point)?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copypasta from somewhere... I'll change it.