Rainforest QA GitHub Action

Marketplace homepage: rainforestapp/github-action

This is the Rainforest QA GitHub Action, it allows you to easily kick off a Rainforest run from your GitHub workflows, to make sure that every release passes your Rainforest integration tests.

Sections

• Prerequisites
• Base Usage
• Optional Parameters
• Release Process

Prerequisites

A Rainforest QA account

If you don't already have one, you can talk to us about setting up an account here.

A Rainforest QA API token

You can find yours on the Integrations setting page. Do not expose this token in your .github/workflows/<workflow>.yml file. Instead, use a GitHub encrypted secret. You may name this secret as you wish (as long as it's a name allowed by GitHub), e.g. RAINFOREST_API_TOKEN.

A run group with at least one test

Run groups are a way to group tests that should be run together (for example, a smoke suite you might want to run on every deploy). For more information on run groups, see this help article.

Once you have a run group which contains at least one enabled test, you can run it in GitHub workflows using this Action. You will need its ID (visible at the end of the run group URL: https://app.rainforestqa.com/run_groups/<ID>).

Base usage

This is a simple workflow file that will start a Rainforest run using run group #1234 every time a commit is pushed to the main branch. To use it, change the run group ID to the one you wish to run, ensure your Rainforest API token is properly configured in your GitHub secrets, and commit this file in your repo at .github/workflows/rainforest.yml.

on:
  push:
    branches:
      - main

jobs:
  rainforest:
    runs-on: ubuntu-latest
    name: Run Rainforest
    steps:
      - name: Rainforest
        uses: rainforestapp/github-action@master
        with:
          token: ${{ secrets.RAINFOREST_API_TOKEN }}
          run_group_id: 1234

Optional Parameters

description

An arbitrary string to associate with the run.

Default behavior

The default value :

"${GITHUB_REPOSITORY} - ${GITHUB_REF_NAME} ${GITHUB_JOB} $(date -u +'%FT%TZ')"

This means that if no description parameter is passed in and your repository is named my_org/my_repo, the GitHub job is #42 on the my_feature_branch branch, and the current time (in UTC) is noon on December 31st, 2021; then the created run's description will be:

my_org/my_repo - my_feature_branch 42 2021-12-31T12:00:00Z

environment_id

Use a specific environment for this run. This parameter will be ignored if the custom_url parameter is also passed in.

Default behavior

If no environment_id parameter is passed in, the created run will use the Run Group's default environment.

custom_url

Use a specific URL (via a temporary environment) for this run.

Default behavior

If no custom_url parameter is passed in, the created run will use the Run Group's default environment.

conflict

How we should handle currently active runs.

Allowed values

Value	Behavior
cancel	Cancel all other runs in the same environment.
cancel-all	Cancel all other runs, regardless of environment.

Default behavior

If no conflict parameter is passed in, then no active runs will be canceled.

execution_method

The execution method to use for this run.

Allowed values

Value	Behavior	Requirements
automation	Run against our automation agent.	- All tests in the run group are written with the Visual Editor. - No tests use a Tester Instruction/Confirmation action.
crowd	Run against our global crowd of testers.
automation_and_crowd	Run against our automation agent where possible, fall back to the crowd of testers.
on_premise	Run against your internal testers.	- On-premise is enabled for your account.

Default behavior

If no execution_parameter parameter is passed in, the created run will run against the run group's default execution method.

release

A string used to link a run to a release (for example, a git SHA or tag, a version number, a code name)

Default behavior

If no release parameter is passed in, the SHA1 hash of the latest commit of the current workflow (obtained via the GITHUB_SHA environment variable) will be used.

automation_max_retries

Set to a value larger than 0 to retry failed tests excuted by our automation agent in the same run up to that number of times.

Default behavior

If no automation_max_retries parameter is passed in, the default from your account or run group is used.

branch

Use a specific Rainforest branch for this run.

Default behavior

If no branch parameter is passed in, the main branch will be used.

background

Set to true to immediately complete the GitHub workflow job without waiting for the Rainforest run to complete

Default behavior

By default we wait for the run to complete in order to pass or fail the workflow job based on the run's result.

dry_run

Set to true to run parameter validations without actually starting a run in Rainforest.

Default behavior

If no dry_run parameter is passed in, the run will be started in Rainforest.

Rerunning failed tests

If your Rainforest run fails due to a "non-bug" (your testing environment might have had a hiccup, or a test might have needed to be tweaked, etc), then rather than make code changes and then run your full testing suite once more, you'll instead want to rerun just the tests that failed. The Rainforest QA GitHub Action uses GitHub caching to know when a workflow is being rerun. It will then automatically rerun only the tests which failed in the previous run.

Running multiple Rainforest runs from a single workflow using workflow_call

The Rainforest QA GitHub Action cannot detect on its own when it is being called multiple times in a single workflow through an intermediary reusable workflow. This means that the second invocation will attempt to rerun the run created by the first invocation, which will fail if that run has no failed tests to rerun. In order to avoid this situation, set the cache_key parameter to a distinct value in each invocation:

# reusable workflow
on:
  workflow_call:
    inputs:
      cache_key:
        type: string

jobs:
  rainforest:
    runs-on: ubuntu-latest
    name: Run Rainforest
    steps:
      - name: Rainforest
        uses: rainforestapp/github-action@master
        with:
          token: ${{ secrets.RAINFOREST_API_TOKEN }}
          run_group_id: 1234
          cache_key: ${{ inputs.cache_key }}

# calling workflow
jobs:
  first-invocation:
    uses: ./.github/workflows/reusable.yml
    with:
      cache_key: first-invocation

  second-invocation:
    uses: ./.github/workflows/reusable.yml
    with:
      cache_key: second-invocation

GitHub Action Release Process

This section describes the release process for the Action itself:

1. Create a feature branch and do your work.
2. Update the version in action.yml.
3. Get review and merge to master.
4. Create a GitHub Release with the proper v-prefixed version tag (i.e. v0.0.1). List Bugfixes, Breaking changes, and New features (if present), with links to the PRs. See previous releases for an idea of the format we're using.
5. The release.yml workflow will then run to update the major release tag. E.g. if your release was for v1.2.3, then it will automatically update the v1 tag.

If you want to run an integration test, create a new branch in a repo of your choice and add a new workflow to.github/workflows/ with the action pointing to your commit.