How to use the System Test Matrix

[brdji]

In this guide I will demonstrate how to define a new behavior, analyze it, and cover it, so that it is shown as tested in the System Test Matrix (STM) dashboard.

Ideally, all behaviors of a technical project should be covered by tests, but realistically this is not always possible. Therefore, we should first focus on covering the most important (and often, most used) behaviors in every module. For the purposes of this tutorial, I have selected a simple behavior that is already covered by tests: wallet/wallet.go - SignBytes. We will add this behavior to the catalog, add annotations to a test that is covering this behavior, and we will see how the catalog is updated to reflect the state of the tests.

To follow this guide, you will need the following:

• The System Test Matrix project cloned and setup on your local machine

1. To start with, clone the venus project in the repo-to-crawl directory of the STM folder. This directory is used by the crawler to build the behavior catalog. All files and folders will be searched for stm annotations, and matched to defined behaviors.

2. To define a new behavior, we need to select a system and subsystem to nest it under. Since we do not have any, we first need to define one.
   All systems and behaviors are defined in the following folder: frontend/apps/web/scripts/data/systems. Let's create a new folder, named wallet, and a file wallet.yml inside of it, where we can store wallet behaviors. The final directory structure should therefore look like this: frontend/apps/web/scripts/data/systems/wallet/wallet.yml
   (Note: The folder and file do not have to share the same name! Feel free to use a different naming scheme).

3. All behavior files need a header, which is used by the crawler and dashboard. This is how the header should look like in our wallet.yml file:

system: wallet
subsystem: Wallet
features:
  - TODO

The naming scheme we used matches the system name (wallet) with the folder name, and the file name (wallet.go) with the subsystem name (Wallet). For example, the file wallet/dsbackend.go would have the following header in its behavior file:

system: wallet
subsystem: DsBackend
features:
  - TODO

Note: As stated above, feel free to change the naming scheme to better suit your needs.

4. Now that we have a file where we can list behaviors, we can add our first behavior definition:

system: wallet
subsystem: Wallet
features:
  - name: Sign bytes
    behaviors:
      - id: WALLET_WALLET_SIGN_BYTES_001
        description: |
          Cryptographically sign 'data' using the corresponding private key 'addr'.

Let's examine this behavior definition in detail:

• The name of the behavior is usually the name of the function we are describing, in this case SignBytes.
• The ID naming scheme is SYSTEM_SUBSYSTEM_BEHAVIOR_NAME_001. We have found this to be very useful when debugging and working on the behavior catalog, as you can easily figure out what file (and system and subsystem) the behavior can be found in. Of course, the behavior ID can be any string, as long as it is unique!
• The description can be anything, but the most useful description is a summary of what the function does. Because the catalog can be used by people who aren't familiar with the codebase, the behavior's description can be very helpful to them.

5. The next step is to annotate a test that is using this behavior. Fortunately, the tests in the file wallet/wallet_test.go already test this behavior, so we don't have to write a new test. To let the test crawler know that this file contains behaviors for the catalog, we first need to add the following annotation at the very top of the file: //stm: #unit
   Since the test file is a unit test file, we declare the file as such to the crawler. If the file contained integration tests, then the annotation would be //stm: #integration.

Then we add an annotation to a test that is referencing this behavior:

//stm: @WALLET_WALLET_SIGN_BYTES_001
sig, err := fs.SignBytes(context.Background(), data, addr)

If the following code line (ie. sig, err := ... ) tests more than one behavior, then the annotation should list all of the behaviors, delimited by a comma. For example:

//stm: @WALLET_WALLET_SIGN_BYTES_001, @WALLET_WALLET_SIGN_BYTES_002, @CHAIN_MESSAGE_DO_SOMETHING_003
wallet.DoComplexOperation(context.Background(), data)

If the annotation would be too long, then it is also possible to split it into several lines, simply starting each one with //stm: . For example:

//stm: @WALLET_WALLET_SIGN_BYTES_001, @WALLET_WALLET_SIGN_BYTES_002 
//stm: @CHAIN_MESSAGE_DO_SOMETHING_003
wallet.DoComplexOperation(context.Background(), data)

6. We can now run the test crawler, and it should update the catalog. Navigate your terminal to the test-crawler directory, and run:
   make copy-output
   This will cause the crawler to go through all files in the repo-to-crawl directory, and populate the catalog with any found behaviors. It will also copy the output .json file to the directory where the dashboard expects to find it.

7. Once the test crawler has finished, navigate your terminal in the frontend directory, and run yarn serve. Once the server has booted up, you can navigate your browser to localhost:8080. You should see the wallet system, subsystem, and finally, the behavior Sign bytes marked as partially tested.

That's it! We now have our first behavior defined and annotated in a test. You now understand how to define behaviors in the catalog, update the test files, and use the dashboard to get an overview of the behavior coverage.

If you have any questions, please leave a comment below.
Thank you!

[hunjixin]

i will try it this afternoon.

[hunjixin]

@brdji cool tool, i known how to to add an annocation and test behavior in stm.