Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(test): writing integration tests #430

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions docs/writing_integration_tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Writing integration tests

## Developing the test source

If you are trying to write a cachi2 integration test, and wish to run cachi2
against a local source repo, assuming that

- the image "localhost/cachi2:latest" exists and is a valid cachi2 container
image
- your test source is in "~/temp/cachi2-test", **and it is a valid git repo**
(i.e. a valid ".git" directory is present)

executing e.g.

```bash
podman run --rm -ti -v "~/temp/cachi2-test:~/temp/cachi2-test:z" -w "~/temp/cachi2-test" localhost/cachi2:latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need -it if we're only executing commands with cachi2 rather than trying to run an interactive session (which also would not work without overriding the entry point)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copypasta from somewhere... I'll change it.

```

*should* give you a way to properly process the test source with cachi2

## Running pytest locally

Once you have working test sources, you'll need to commit and push them
somewhere that pytest can clone them from.

We *strongly* recommend making a fork, specific to your test, from one of the
repos found under the [cachito test repos][] GitHub org (note that there
are *many* repos there, covering all of the package managers which cachi2 and
Cachito support) [^1].

Once you have a fork, push to a new branch named after your scenario - now
pytest will have a proper commit hash in a repo to which you have complete
access (once your test is complete and passing, you can simply open a PR against
the repo in the [cachito-testing][] org).
eskultety marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +23 to +34
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is summarized later in the walkthrough in a step-by-step fashion. Can we merge this into the walkthrough so that it doesn't feel like repeating the same information with slightly different wording?


At this point, you should be able to test locally.

### A walkthrough

1. Fork a test repo under [cachito-testing][] which uses the package manager you
need to test, and/or is otherwise related
1. Clone the new fork to your local machine
1. Create a new branch in the repo, named to reflect the purpose of your test(s)
1. The goal here is to create what looks like a real
Go/Javascript/Python/whichever project, but simplified enough to *only* include
the required files which cachi2 needs to find and resolve dependencies
1. Commit the test code, and push it to your fork in GitHub. Take note of the
commit hash.

#### Testing your tests

1. Add your pytest scenarios to the appropriate integration test source file
under 'cachi2/tests/integration'. You'll need to put the following in
`utils.TestParameters` for your test case

- `repo="https://github.com/cachito-testing/cachito-pip-without-deps.git"` # this will be the name of the test repo you forked
- `ref="3fe2fc3cb8ffa36317cacbd9d356e35e17af2824"` # this will be the commit hash noted previously
- `packages=({"path": ".", "type": "pip"},)` # for pip, for example
- `check_vendor_checksums=False` # or `True`, depending on the scenario

1. In your local **cachi2** repo, make sure you have 'pytest' and 'jsonschema'
pip-installed in the cachi2 venv, so that you can run pytest without tox. Without
`tox`, it's **much** easier to run pytest with *only* your selected test
scanarios, and get useful, accessible logs
1. Bonus: pip-install 'pytest-html' in the venv, and add (e.g.)
`--html=path_to_reports/pytest-report.html` to your `pytest` command line
for a very nicely formatted HTML report
1. To summarize

- You're in the cachi2 repo directory
- Your test scenario's source is in 'tests/integration/test_foo.py'
- Your test scenario is called `test_foo_package`, and its pytest 'id' is
`foo_incorrect_checksum`

1. Now run

```bash
CACHI2_IMAGE=localhost/cachi2:latest pytest -rA -vvvv --confcutdir=tests/integration --log-cli-level=DEBUG tests/integration/test_foo.py::test_foo_package[foo_incorrect_checksum]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running with an already built image IMO only makes sense when your debugging an issue, but from a developer's perspective you just added some code to cachi2 that you may have tested locally outside container environment, so you actually want the image to be rebuilt for tests. By all means we have to document the env variable, I just don't think that what you're proposing in the guideline is going to be the case for most invocations during test development.

```

which will run *only* 'tests/integration/test_foo.py::test_foo_package' with
it's "foo_incorrect_checksum" ID'd parameter set with a pre-built cachi2 container

## Running the test suite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need another section to describe some general good practices when it comes to running our integration test suite, the contents can live happily in the previous one IMO.


It's a good idea to run the whole cachi2 integration test suite, just to make
sure everything still works properly. The command for this, *from inside the
cachi2 repo* is `tox -e integration`. You can also provide
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also want to mention the make test-integration shortcut or would that only add noise?

`CACHI2_IMAGE=localhost/cachi2:latest` if you already have a current cachi2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good you mention this, but it feels weird that you actually explain the variable after the walkthrough section (next commit) which comes before this one.

container. e.g. `CACHI2_IMAGE=localhost/cachi2:latest tox -e integration` - this
will save a lot of time, as otherwise the image will be rebuilt from scratch.

[^1]: If you **really** can't find an existing repo which is related to your test
scenario, ping a maintainer and explain your situation.

[cachito-testing]: https://github.com/orgs/cachito-testing
[cachito test repos]: https://github.com/orgs/cachito-testing/repositories?type%3Dsource