[Feature Request] Add queuing process on Testkube to trigger the testsuite.

[trangpkaxon]

Is your feature request related to a problem? Please describe.

• Regarding Test Triggers, we don't have queue mode for the same automation test run, which causes our test to run in parallel and fails because of a data conflict.

Step reproduce

• For example, we have test suite A

  1. A is running (A-1st)
  2. Others are still using the test suite A (A-2nd)

• Actual:

  • A-2nd can be executed concurrently with A-1st.
    => In this case, the testscript may have conflicting data that is utilized to perform

• Expectation:

  • Run A-2nd only after A-1st has completed.

Additional context
https://github.com/kubeshop/testkube/assets/162653239/53276f6d-8383-46dc-a920-99cc229b070d

[vsukhin]

thank you @trangpkaxon looks like an intersting feature. for @TheBrunoLopes and @jmorante-ks to analyze and prioritize

[jmorante-ks]

@trangpkaxon , I can give an update on this topic.

TestSuites are being deprecated in favor of Test Workflows execute feature.

For Test Workflows we have in roadmap enhancements that could cover your queue needs. We do not have ETA for them yet, we will let you know once we have it confirmed.

[vsukhin]

@trangpkaxon For Test Workflows we have a workaround - a special testworkflow template, that should be included in such queuing test workflows. All test worrkflows will be launched, but only one will be active, when other will wait for a free slot - @rangoo94

[rangoo94]

Hi, the template @vsukhin mentioned basically looks like this:

kind: TestWorkflowTemplate
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: limit-concurrency
spec:
  config:
    concurrency:
      type: integer
      default: 1
    delay:
      type: number
      description: delay seconds
      default: 2
  setup:
  - name: Waiting for free slot
    run:
      image: kubeshop/testkube-cli
      shell: |
        echo "Concurrency:      {{ config.concurrency }}"
        while true
        do
          ALL="$({{ shellquote(
            "testkube", "get", "twe", "-c", "direct", "--api-uri", "http://testkube-api-server:8088",
            "--testworkflow", workflow.name, "-o", "go", "--go-template", "{{ .Number }} {{ .Result.Status }} {{ .Id }}\n"
          ) }} | grep -E 'queued|running|paused' | sort -h | sed -E 's/([^ ]* )*//')"
          RUNNABLE="$(echo "$ALL" | head -{{ int(config.concurrency) }})"
          ORDER="$(echo "$ALL" | grep -n "{{ execution.id }}" | sed -E "s/:.*//")"
          if [ -z "$(echo "$RUNNABLE" | grep "{{ execution.id }}")" ]; then
            echo "Current position: $ORDER ($(( ORDER - {{ int(config.concurrency) }} )) needs to finish to continue)"
            sleep {{ float(config.delay) }}
          else
            echo "Current position: $ORDER... Starting!"
            exit 0
          fi
        done

It's quite simple - it's calling the API Server to get information if there are any other executions in progress, and advancing to the next step when it's ready.

The deprecated Test Suites are represented by execute step in the Test Workflow. You can use it like this:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: example-no-concurrency
spec:
  use:
  - name: limit-concurrency
    config:
      concurrency: 1 # maximum workflows to run at once
      delay: 1       # seconds between consecutive checks
  steps:
  # regular steps, i.e. the test-suite like scheduler
  - execute:
      # parallelism: 3 # optionally, limit how many of the workflows below could be run in parallel
      workflows:
      - name: some-name
      - name: some-name-2
      - name: some-name-3

The execute syntax is not available in the Open Source version though. In the Open Source, you can try to i.e. call with Testkube CLI, like:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: example-no-concurrency
spec:
  use:
  - name: limit-concurrency
    config:
      concurrency: 1 # maximum workflows to run at once
      delay: 1       # seconds between consecutive checks
  steps:
  - run:
      image: kubeshop/testkube-cli:latest
      shell: |
        testkube -c direct --api-uri "http://testkube-api-server:8088" run tw -f some-name
        testkube -c direct --api-uri "http://testkube-api-server:8088" run tw -f some-name-2
        testkube -c direct --api-uri "http://testkube-api-server:8088" run tw -f some-name-3