Using Guibot for GUI assertions and operations

[pevogam]

Hi OpenQA community,

This is just if it gets someone here interested but perhaps you would find the Guibot project useful as a potential backend for the GUI assertions and operations. Let me elaborate.

We at Intra2net AG have developed a lot of GUI tests for our products and various packages like the Horde groupware, AD/LDAP, and more. More often than now we needed to simulate a real user to operate on our systems (encapsulated for testing as virtual machines), looking at the VNC screen and performing assertions about certain subregions of the screen. I heard about your project and since I am using OpenSUSE on my home desktop I was surprised to see that you do a lot of this validation too. Clearly, I am not fully aware of all capabilities of your GUI operator but I would still like to recommend Guibot because of its rich choice of CV (computer vision) backends - from template and feature matching to OCR and DL - and its rich choice of desktop controller backends like VNCDoTool for VNC screens, XDoTool, AutoPy, PyAutoGUI, etc. The CV backends are especially important since through 7 years of GUI test development we have had plenty of corner cases where we need more advanced forms of screen region matching. The OCR backends (we have three options) are also helpful in recent tests where we want to read out text from the screen and convert it back to string for some more elaborate (e.g. regex or else) assertions.

If you want to know more just let me know and I hope we can be of your assistance,
Plamen

[okurz]

Hi @pevogam , nice that you found openQA. Thank you for your article. It is very interesting to see Guibot. It is great to see other approaches. openQA of course also uses openCV as well as Tesseract OCR for the OCR backend although the latter is not very prominent, likely due to the normal screen matching approach being easy enough to maintain :) A big benefit of openQA for example for openSUSE distribution testing was being completely independant of the operating system on a machine that we want to run the tests on. Is Guibot running within the same operating system that runs the applications you want to test or can that be independant?

[pevogam]

Hi @okurz,

Hi @pevogam , nice that you found openQA. Thank you for your article. It is very interesting to see Guibot. It is great to see other approaches. openQA of course also uses openCV as well as Tesseract OCR for the OCR backend although the latter is not very prominent, likely due to the normal screen matching approach being easy enough to maintain :)

Very interesting to know this!

Yes, in fact the point for us maintaining Guibot by itself separately from a larger ecosystem of testing components relying on virtual machines, containers, etc. is that we maintain a lot of CV backends (e.g. template, feature, and cascade matching just in OpenCV but also PyTorch for DL backends, different types of text detection from contours to CNNs, etc.) which takes the larger percentage of the maintenance costs for the project. However this is justified for us in two ways:

1. Making the project open source will increase the benefit for multiple parties centralizing the maintenance cost in one place and thus giving greater returns per maintenance cost.

2. All the years of GUI testing have shown us that a basic 1:1 matching (or just slightly more sophisticated in the case of OpenCV's standard template matching) is simply not enough for real life GUI testing scenarios. Having more CV backends to choose from or switch among is thus useful to handle more of the practical situations we enter with each new test we develop.

A big benefit of openQA for example for openSUSE distribution testing was being completely independant of the operating system on a machine that we want to run the tests on. Is Guibot running within the same operating system that runs the applications you want to test or can that be independant?

The short answer is both, again due to the higher range of DC (display controlling) backends we have. We have backends that can run directly on an operating system which could be all of Linux/MacOS/Windows but we also have a VNC desktop controller that simply looks at a VNC screen of a remote or virtual machine without being influenced by reboots or any other OS-dependent events and requirements.

[Martchus]

openQA and its backend os-autoinst serve multiple purposes:

1. Dealing with big assets such as ISO images of a distribution and HDDs of an installation.
2. Spinning up an environment to run a full system test, e.g. starting a VM via QEMU (but there are various other backend).
3. Providing an API for testing applications including graphical interfaces and sound output.
4. Providing a web UI and REST API to view test results, create reference images, manage assets and more.

I assume Guibot would only help with point 3.

Most of the functions of the test API (3.) can be ignored if not wanted. For instance kernel tests usually rely on upstream testsuites. The test results can be imported if they use a common format like JUnit. Other test tools could be used in the same way. So instead of doing the clicking via openQA's API one might just launch a Guibot test instead. In that sense it could already be used right now.

I'd like to note that the graphics processing hasn't seen much changes lately except for migrating to newer OpenCV versions. So we're most likely lacking behind Guibot a lot. That means we could indeed benefit from integrating the tool. Like I said, it could theoretically already be used like other external testsuites. I'm saying "theoretically" here because it leads to lots of questions, especially how well it will integrate with the existing features of openQA's web UI (4.).

[pevogam]

Hi @Martchus,

openQA and its backend os-autoinst serve multiple purposes:

1. Dealing with big assets such as ISO images of a distribution and HDDs of an installation.

2. Spinning up an environment to run a full system test, e.g. starting a VM via QEMU (but there are various other backend).

3. Providing an API for testing applications including graphical interfaces and sound output.

4. Providing a web UI and REST API to view test results, create reference images, manage assets and more.

I assume Guibot would only help with point 3.

Precisely, this is what I meant with "potential backend for the GUI assertions and operations" above. We do make use of various non-GUI tests as well and rely on Qemu just as much but at present we use the Avocado Framework (prev. Autotest) for these. Guibot here is viewed only as a component of any such potential ecosystem.

Most of the functions of the test API (3.) can be ignored if not wanted. For instance kernel tests usually rely on upstream testsuites. The test results can be imported if they use a common format like JUnit. Other test tools could be used in the same way. So instead of doing the clicking via openQA's API one might just launch a Guibot test instead. In that sense it could already be used right now.

I'd like to note that the graphics processing hasn't seen much changes lately except for migrating to newer OpenCV versions. So we're most likely lacking behind Guibot a lot. That means we could indeed benefit from integrating the tool. Like I said, it could theoretically already be used like other external testsuites. I'm saying "theoretically" here because it leads to lots of questions, especially how well it will integrate with the existing features of openQA's web UI (4.).

Yes, I guess this would be the key step here and the main potential source of problems and sink of effort. Three brief examples of API usage and main interfaces are shown here in case you want to take a quick look and conclusion on how difficult integration will be. I presume the general API will be more or less the same with region and subregion calls to click/type/etc methods. But the main point of this discussion is mostly to let you know of this possibility. I am not aware how heavy GUI test usage you have in the OpenQA's user base, any potential integration effort could be justified by sufficiently large usage.