ci: add workflows #34

JaeAeich · 2024-08-05T14:47:13Z

Summary by Sourcery

Introduce a new Task Execution Service (TES) module along with supporting modules for transport, configuration, and service information. Add comprehensive CI workflows for PR evaluation, code testing, and code quality checks. Implement build and test scripts, and add documentation and issue templates.

New Features:

Introduce a new module for Task Execution Service (TES) with functionalities to create, get, list, and cancel tasks.
Add a new Transport module to handle HTTP requests with support for GET, POST, PUT, and DELETE methods.
Implement a new Configuration module to manage API configurations including base path, user agent, and authentication details.
Add a ServiceInfo module to fetch service information using the new Transport module.

Enhancements:

Add a build script (build-models.sh) to generate models from OpenAPI specifications.
Introduce a pre-commit configuration (.pre-commit-config.yaml) to enforce code quality checks.
Add a markdown lint configuration (.markdownlint.yaml) to enforce consistent markdown styling.
Add a YAML lint configuration (.yamllint.yaml) to enforce consistent YAML styling.
Add a nextest configuration (nextest.toml) to manage test retries and setup commands.

Build:

Add a build script (build-models.sh) to automate the generation of models from OpenAPI specifications.

CI:

Add a CI workflow (.github/workflows/pr_evaluation.yaml) to evaluate pull requests with checks for merge conflicts, pre-commit hooks, semantic PR titles, and binary builds.
Add a CI workflow (.github/workflows/code_test.yaml) to run integration and unit tests with coverage reporting.
Add a CI workflow (.github/workflows/code_quality.yaml) to run formatting, linting, and spell check tasks.

Documentation:

Add a README.md file with instructions for building the project, running tests, and testing the CI/CD workflow locally.

Tests:

Add unit tests for the TES module to test task creation, status retrieval, cancellation, and listing.
Add unit tests for the Transport module to test HTTP request handling.
Add a test utility module (test_utils.rs) to setup the testing environment and ensure Funnel server is running.

Chores:

Add a general-purpose issue template (.github/ISSUE_TEMPLATE/general-purpose.md) for reporting issues.

sourcery-ai · 2024-08-05T14:47:21Z

Reviewer's Guide by Sourcery

This pull request introduces a new module for Task Execution Service (TES) with task management and async functions, a Transport module for HTTP requests, and a ServiceInfo module for service information retrieval. It also includes GitHub Actions workflows for PR evaluation, code testing, and code quality checks, along with configuration files for markdownlint, pre-commit hooks, and yamllint. Additionally, scripts and configuration for building OpenAPI models and running tests are added.

File-Level Changes

Files	Changes
`lib/src/tes/mod.rs` `lib/src/tes/model.rs`	Added a new module for TES (Task Execution Service) with models, task management, and async functions for task operations, including unit tests.
`lib/src/transport.rs` `lib/src/serviceinfo/mod.rs`	Added new modules for Transport and ServiceInfo with HTTP request handling and service information retrieval, including unit tests.
`.github/workflows/pr_evaluation.yaml` `.github/workflows/code_test.yaml` `.github/workflows/code_quality.yaml`	Added GitHub Actions workflows for PR evaluation, code testing, and code quality checks.
`.markdownlint.yaml` `.pre-commit-config.yaml` `.yamllint.yaml`	Added configuration files for markdownlint, pre-commit hooks, and yamllint.
`build-models.sh` `run-tests.sh` `nextest.toml`	Added scripts and configuration for building OpenAPI models, running tests, and ensuring the Funnel server is running.

Tips

Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
Continue your discussion with Sourcery by replying directly to review comments.
You can change your review settings at any time by accessing your dashboard:
- Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
- Change the review language;
You can always contact us if you have any questions or feedback.

uniqueg

Some minor comments, but generally the workflows look really good/useful. Of course needs some changes in the code to pass. Also, perhaps a more Rust-experienced person could look at it, because I don't know the toolchain really. @pavelnikonorov @vemonet

I have added CODECOV_TOKEN.

uniqueg · 2024-08-07T12:03:57Z

.github/ISSUE_TEMPLATE/general-purpose.md

Doesn't have anything to do with CI?

😭 yeah I just wanted to add most of the things that were left from cookiecutter thats it, not just the CI.

Fair. But let's make sure to remove this from this PR and add it to another (together with the PR template).

committed the removal of .github/ISSUE_TEMPLATE/general-purpose.md to the branch

uniqueg · 2024-08-07T12:05:58Z

.github/workflows/code_quality.yaml

+        with:
+          component: clippy
+
+      - name: Run unit tests


Run linting?

captions updated to the format "Running .... checks"

uniqueg · 2024-08-07T12:07:11Z

.github/workflows/code_test.yaml

+        run: cargo test --tests
+
+      - name: Install cargo-tarpaulin
+        run: cargo install cargo-tarpaulin


Add to Cargo.toml?

aah yes, didn't touch that as it wan't merged.

I think since it is a CLI tool it needs to be installed manually outside of the Cargo.toml, or maybe you can add it to the rust-toolchain file

Really? What's the difference? Aren't the deps in Cargo.toml installed via cargo install?

@uniqueg deps in Cargo.toml are installed differently – into the project's build cache, while the cargo install is only for standalone CLI tools such as rustfmt and cargo-tarpaulin.

as for cargo-tarpaulin, it can be installed via components like clippy, but only with the 'nightly' toolchain. So I’d suggest keeping it as a separate installation step.

Adding a rust-toolchain might be beneficial, so I've added an issue #45 (comment)

uniqueg · 2024-08-07T12:08:13Z

.github/workflows/pr_evaluation.yaml

+          poetry-install-options: "--only=misc --no-root"
+          poetry-export-options: "--only=misc"
+
+      - name: Check all the pre-commit hooks passed


Check that all pre-commit hooks passed

🤯 this is what I have in cookiecutter though!!!! Do I have change it there as well?

Why the need for poetry here?

Please ignore, was copying the structure from a python project, must have been left in. Will remove.

Yes, not a biggie of course. You can change it here, and sneak it in some other time in the cookiecutter template, if you remember

Ah, yes, overlooked the Poetry artifacts here - good catch @vemonet

committed the removal of Poetry

JaeAeich · 2024-08-07T15:29:41Z

@uniqueg either merge this after we have merged all other PRs, because resolving convo will mean me changing some code so it doesn't make sense to do that rn, or merge this into some PR take it from there which might not be ideal though.

.github/workflows/code_quality.yaml

vemonet · 2024-08-08T07:24:34Z

.github/workflows/code_test.yaml

+        run: cargo install cargo-tarpaulin
+
+      - name: Run unit tests with coverage
+        run: cargo tarpaulin --tests --out Xml


I think you should also add the --doc flag so that tarpaulin includes codeblocks from the docstrings

committed the adding of the --doc flag in both integration in unit tests

vemonet · 2024-08-08T07:25:25Z

Looking good, most things required seems to be there: tests, cov, fmt, linting, precommit

JaeAeich · 2024-08-08T08:12:36Z

Looking good, most things required seems to be there: tests, cov, fmt, linting, precommit

Thanks for the review, Ill make changes when the branch on which this is bases off of gets merged.

uniqueg · 2024-08-08T08:40:44Z

Looking good, most things required seems to be there: tests, cov, fmt, linting, precommit

Thanks for the review, Ill make changes when the branch on which this is bases off of gets merged.

"This branch has no conflicts with the base branch" <- I think you could already make the changes? Of course, it may need some other stuff until tests pass, but I do think you could work towards resolving the open issues at this point, @JaeAeich and @aaravm

…ity.yaml

pavelnikonorov

@uniqueg I've reviewed all conversations, and committed the discussed changes.
also, I have tested it locally using act, the workflows work, but the tests fail, which should be addressed separately and within the other PRs review.

Added two issues out of this PR: #44, #45

P.S. FYI, I've done a few push-force on my commits here to amend them, but it appears that it uses a different credentials source, which belongs to Grigorii who used to work on the same machine.

pavelnikonorov · 2024-10-26T16:01:36Z

.github/ISSUE_TEMPLATE/general-purpose.md

committed the removal of .github/ISSUE_TEMPLATE/general-purpose.md to the branch

.github/workflows/code_quality.yaml

pavelnikonorov · 2024-10-26T16:25:57Z

.github/workflows/code_quality.yaml

+        with:
+          component: clippy
+
+      - name: Run unit tests


captions updated to the format "Running .... checks"

pavelnikonorov · 2024-10-26T21:37:54Z

.github/workflows/code_test.yaml

+        run: cargo test --tests
+
+      - name: Install cargo-tarpaulin
+        run: cargo install cargo-tarpaulin


@uniqueg deps in Cargo.toml are installed differently – into the project's build cache, while the cargo install is only for standalone CLI tools such as rustfmt and cargo-tarpaulin.

as for cargo-tarpaulin, it can be installed via components like clippy, but only with the 'nightly' toolchain. So I’d suggest keeping it as a separate installation step.

Adding a rust-toolchain might be beneficial, so I've added an issue #45 (comment)

pavelnikonorov · 2024-10-26T21:48:03Z

.github/workflows/pr_evaluation.yaml

+          poetry-install-options: "--only=misc --no-root"
+          poetry-export-options: "--only=misc"
+
+      - name: Check all the pre-commit hooks passed


committed the removal of Poetry

pavelnikonorov · 2024-10-26T21:50:24Z

.github/workflows/code_test.yaml

+        run: cargo install cargo-tarpaulin
+
+      - name: Run unit tests with coverage
+        run: cargo tarpaulin --tests --out Xml


committed the adding of the --doc flag in both integration in unit tests

pavelnikonorov · 2024-10-28T11:19:03Z

all suggestions addressed, merging as discussed with @uniqueg

chore: add some ci

f1f5e59

JaeAeich changed the base branch from main to dev August 5, 2024 14:48

uniqueg changed the title ~~feat: cicd~~ ci: add workflows Aug 7, 2024

uniqueg requested changes Aug 7, 2024

View reviewed changes

uniqueg mentioned this pull request Aug 7, 2024

ci: workflows for local runs and GH actions #18

Closed

vemonet reviewed Aug 8, 2024

View reviewed changes

.github/workflows/code_quality.yaml Outdated Show resolved Hide resolved

vemonet reviewed Aug 8, 2024

View reviewed changes

pavelnikonorov added 4 commits October 26, 2024 17:00

remove .github/ISSUE_TEMPLATE/general-purpose.md

7666b76

remove unnecessary code autoformatting in .github/workflows/code_qual…

95c05d8

…ity.yaml

add toolchain key and componentS key name fix

6c8e65a

update steps captions

e1532be

Xforsa force-pushed the cicd branch 3 times, most recently from 792e8da to 7442c57 Compare October 26, 2024 21:26

add funnel into workflow as TES instance is required for tests

16004da

Xforsa force-pushed the cicd branch from 7442c57 to 16004da Compare October 26, 2024 21:28

pavelnikonorov mentioned this pull request Oct 26, 2024

ci/cd: consider adding rust-toolchain file #45

Open

pavelnikonorov added 2 commits October 26, 2024 21:46

remove unnecessary Poetry setup as it not Python project

db3c54c

add --doc flag in tarapulin for both integration and unit tests

1e2b109

pavelnikonorov approved these changes Oct 26, 2024

View reviewed changes

pavelnikonorov merged commit 057b5ea into dev Oct 28, 2024

pavelnikonorov deleted the cicd branch October 28, 2024 11:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add workflows #34

ci: add workflows #34

JaeAeich commented Aug 5, 2024 •

edited by sourcery-ai bot

Loading

sourcery-ai bot commented Aug 5, 2024 •

edited

Loading

uniqueg left a comment •

edited

Loading

uniqueg Aug 7, 2024

JaeAeich Aug 7, 2024

uniqueg Aug 8, 2024

pavelnikonorov Oct 26, 2024

uniqueg Aug 7, 2024

pavelnikonorov Oct 26, 2024

uniqueg Aug 7, 2024

JaeAeich Aug 7, 2024

vemonet Aug 8, 2024

uniqueg Aug 8, 2024

pavelnikonorov Oct 26, 2024

uniqueg Aug 7, 2024

JaeAeich Aug 7, 2024

vemonet Aug 8, 2024

JaeAeich Aug 8, 2024

uniqueg Aug 8, 2024

uniqueg Aug 8, 2024

pavelnikonorov Oct 26, 2024

JaeAeich commented Aug 7, 2024

vemonet Aug 8, 2024

pavelnikonorov Oct 26, 2024

vemonet commented Aug 8, 2024

JaeAeich commented Aug 8, 2024

uniqueg commented Aug 8, 2024

pavelnikonorov left a comment

pavelnikonorov Oct 26, 2024

pavelnikonorov Oct 26, 2024

pavelnikonorov Oct 26, 2024

pavelnikonorov Oct 26, 2024

pavelnikonorov Oct 26, 2024

pavelnikonorov commented Oct 28, 2024

ci: add workflows #34

ci: add workflows #34

Conversation

JaeAeich commented Aug 5, 2024 • edited by sourcery-ai bot Loading

Summary by Sourcery

sourcery-ai bot commented Aug 5, 2024 • edited Loading

Reviewer's Guide by Sourcery

File-Level Changes

uniqueg left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JaeAeich commented Aug 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vemonet commented Aug 8, 2024

JaeAeich commented Aug 8, 2024

uniqueg commented Aug 8, 2024

pavelnikonorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavelnikonorov commented Oct 28, 2024

JaeAeich commented Aug 5, 2024 •

edited by sourcery-ai bot

Loading

sourcery-ai bot commented Aug 5, 2024 •

edited

Loading

uniqueg left a comment •

edited

Loading