Cherry-pick to earlgrey_1.0.0: [ci] Introduce a test orchestrator written in bazel #24868
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
github-actions
bot
requested review from
rswarbrick and
cfrantz
as code owners
jwnrt
approved these changes
Oct 24, 2024
|
Blocked on #24871 |
jwnrt
force-pushed
the
backport-23505-to-earlgrey_1.0.0
branch
2 times, most recently
from
5b1b7a8
to
9005cb4
Compare
nbdd0121
reviewed
Nov 21, 2024
The commit fundamentally changes how tests are considered for running in CI. Previously, any test not marked as "manual", "broken" and "skip_in_ci" would be run in CI in all execution environment for which it is defined. We then came up with some clever bazel queries to try and avoid running them in multiple execution environment when not necessary but this is quite fragile. The approach is very different: when creating the tests using opentitan_test, we call a new function, called "ci_orchestrator" with the list of all exec_env for which it is defined. This function then returns a subset of that list that should be marked with skip_in_ci. The current implementation is that there is a list of exec_env that are mutually-exclusive (_ONLY_RUN_ONE_IN_CI_SORTED) and all but one will be skipped: the only one to run will the highest one in the list. With this approach, we are guaranteed to never run duplicate tests. If more complicated logic is required, it can always be added to this function. As a byproduct of this, we can significantly simplify the filtering logic when running FPGS jobs. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 9d846d6)
The --test_tag_filters option of `bazel test` is very convenient but is only available for testing and not for querying. This script takes as input a list of tags in the same fort as test_tag_filters and runs an equivalent bazel query to list all tests that match. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit e906405)
Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 072fd99)
Those tests can run in the ROM_EXT stage which should be preferred. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 8e828bf)
Previously we only tagged then with the exec_env name but for the CI it is convenient to have a global way of querying all FPGA tests. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 046356b)
With the filter tags filtering logic, it is very easily to create a situation where some tests are run in several jobs, or to forget running tests because of a non-exhaustive set of filters. This commit adds a final CI job that checks the above condition. This is achieved by downaloding the list of tests run by every job and then making sure that there are no duplicates. We also check for missing tests by running a query on all non-broken, non-manual, non-skip_in_ci FPGA tests. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit fb58259)
Those tests require the host interface to support serial break but the CW310 used in the CI do not support it so they always fail. These tests will need to be changed to the hyperdebug interface. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 8a211ea)
The test rv_dm_mem_access_dev_rv_dm_delayed_enabled is broken. Signed-off-by: Amaury Pouly <[email protected]> (cherry picked from commit 1c44deb)
This wasn't present in the cherry-picked commits for the test orchestrator, but was present on this branch before. Signed-off-by: James Wainwright <[email protected]> (commit is original to earlgrey_1.0.0)
jwnrt
force-pushed
the
backport-23505-to-earlgrey_1.0.0
branch
from
9005cb4
to
b284c17
Compare
|
CI failure looks like flakiness to me ( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an automatic cherry-pick of #23505 to branch
earlgrey_1.0.0.