[Feat] gunittest: option to define order for integration tests

[ninsbl]

Is your feature request related to a problem? Please describe.
Some modules are regularly applied in a specific order. Here I think for example of addons i.sentinel.download and i.sentinel.import or i.sentinel.preproc.

It would be nice to be able to specify the order of unit tests so integration between modules can be tested too. One would have to delay "teardown" until integration test is complete.

Would that be possible? And what would be needed?

[echoix]

I think that it is generally not the way to go. If a test passes or fails because of the order it run, there's a real problem with the test.

If the test meant to have specific actions run before, it should be written as so, so that the test on himself starts from a fresh state, arranges what it needs, does his things, checks the results, then cleans up (or at the end of the module). If we are speaking of integration tests, this would mean that the test wants to test a bit more steps than what is currently defined.

Some tools exists to make sure the tests are run in a random order, to highlight these kinds of of issues when they aren't caught before, or since they were written.

Furthermore, the less interdependencies between tests (meaning they fully define what they test), the more options we have to make them run faster and split them across cores or jobs.

[ninsbl]

Thanks, @echoix ! Good points! So, the approach would be to rather write integration tests that operate independently?!

Since auch tests are not limited to single modules, where would we put them? And naturally, they would be expected to run a bit longer... So, timeoutneeds to be handled...

[echoix]

Just another test file anywhere near what you'd test. With a different file name, it makes it easier to filter out.

And as marisn noted, it probably shouldn't be the download itself that should be tested end-to-end, as apart from being long, it would probably be flaky.

[marisn]

I hope you are not trying to download files from internet with i.sentinel.download. Although this would be a good test to see if the server side API hasn't changed, it might cause problems on its own (e.g. running offline, server down, etc.). Mock is your friend.

I would go with writing simple tests for each of modules and for one (last) of modules just write a full integration test. Yes, it will fail also if any of other executed modules fails, but that's life.