iframe-proxy | Sunbelt Computer Software

Integrating code coverage reporting into your CI pipeline ensures that coverage data is consistently collected and reported for every build. Qlty is compatible with every major CI provider. Qlty offers CI-specific packages for GitHub Actions (GitHub Action) and CircleCI (CircleCI Orb), but works just as well without them. Qlty provides environment detection for most CI providers, making integration simpler, but offers simple instructions for those without it.

CI Provider	Package	Environment Detection	OIDC
GitHub Actions	qlty-action	Yes	Yes
CircleCI	qlty-orb	Yes	No
Buildkite		Yes	No
Jenkins		Yes	No
CodeFresh		Yes	No
SemaphoreCI		Yes	No
TravisCI		Yes	No
bitrise		Yes	No
Other CI Providers		No	No

CI Provider Integration

GitHub Actions

Integrate with Qlty using our GitHub Action. While integration can be done without our GitHub Action, using it allows you to take advantage of OIDC support.

# .github/workflows/test.yml
name: Test and Coverage

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read
  id-token: write # Required for OIDC

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Run your tests with coverage
      - name: Run tests
        run: npm test -- --coverage

      # Upload coverage to Qlty
      - uses: qltysh/qlty-action/coverage@v2
        with:
          oidc: true
          files: coverage/lcov.info

# .github/workflows/test.yml
name: Test and Coverage

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Run your tests with coverage
      - name: Run tests
        run: npm test -- --coverage

      # Upload coverage to Qlty
      - uses: qltysh/qlty-action/coverage@v2
        with:
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
          files: coverage/lcov.info

How the action verifies the Qlty CLI

Before it runs the Qlty CLI, the action checks the CLI’s attestation with the GitHub CLI (gh). If the check rejects the CLI, the step fails, even with skip-errors: true, and the CLI never runs. If the check can’t finish because GitHub is unavailable, the CLI doesn’t run either, and skip-errors decides whether the step fails. skip-errors, which is on by default, keeps a problem with Qlty or with a service the action depends on from failing your build: a failed upload, a CLI error, or an attestation check that can’t finish. It never lets an unverified Qlty CLI run. The check needs gh 2.67.0 or later on the machine that runs the step. GitHub-hosted runners include it. Self-hosted runners and container: jobs often don’t. Without a usable gh, the action logs a warning that the run is unverified and continues.

Pin versions deliberately: if you pin the action by commit SHA or exact tag, or set cli-version, update those pins when a security release ships. qltysh/qlty-action/coverage@v2 follows the latest v2 release.
Self-hosted runners: install gh, keep the operating system’s ca-certificates package current, and keep the runner itself up to date, since actions run on the Node version bundled with the runner.

CircleCI

Qlty offers a CircleCI Orb for easy integration.

# Add the Qlty orb to your `.circleci/config.yml`:
version: 2.1

orbs:
  qlty-orb: qltysh/qlty-orb@0.0

jobs:
  test:
    docker:
      - image: cimg/node:16.13
    steps:
      # Checkout is required
      - checkout
      - run:
          name: Install dependencies
          command: npm ci
      - run:
          name: Run tests with coverage
          command: npm test
      - qlty-orb/coverage_publish:
          files: coverage/lcov.info

workflows:
  main:
    jobs:
      - test

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:16.13
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: npm ci
      - run:
          name: Run tests with coverage
          command: npm test
      - run:
          name: Install Qlty CLI
          command: curl https://qlty.sh | sh
      - run:
          name: Upload coverage
          command: qlty coverage publish coverage/lcov.info
          environment:
            QLTY_COVERAGE_TOKEN: ${QLTY_COVERAGE_TOKEN}

workflows:
  main:
    jobs:
      - test

Travis CI

Add Qlty coverage reporting to your .travis.yml:

language: node_js
node_js:
    - 16

script:
    - npm test -- --coverage

after_success:
    - curl https://qlty.sh | sh
    - qlty coverage publish coverage/lcov.info

Make sure to add your QLTY_COVERAGE_TOKEN in the Travis CI repository settings.

On Travis CI, qlty coverage publish detects the branch for push builds (merge commits and direct pushes) as well as Pull Request builds, so no --override-branch is required.

GitLab CI

Add coverage reporting to your .gitlab-ci.yml:

test:
    stage: test
    script:
        - npm ci
        - npm test -- --coverage
        - curl https://qlty.sh | sh
        - qlty coverage publish coverage/lcov.info
    artifacts:
        paths:
            - coverage/

Add your QLTY_COVERAGE_TOKEN as a CI/CD variable in your GitLab project settings.

Buildkite

Export your QLTY_COVERAGE_TOKEN as an environment variable. (See Buildkite’s documentation on managing secrets. One option is to export QLTY_COVERAGE_TOKEN in an environment agent hook):

#!/usr/bin/env bash
set -e

export QLTY_COVERAGE_TOKEN="YOUR_COVERAGE_TOKEN"

Then update your pipeline.yaml appropriately. Here’s an example with Ruby test coverage:

steps:
    - label: ":ruby: Ruby 3.1 Tests"
      key: "ruby-tests"
      command: |
          bundle install
          bundle exec rspec --require spec_helper
      artifact_paths:
          - "coverage/coverage.json"

    - wait

    - label: ":chart_with_upwards_trend: Coverage Report"
      command: |
          # Download the coverage artifact
          buildkite-agent artifact download "coverage/coverage.json" .

          curl https://qlty.sh | sh
          qlty coverage publish coverage/coverage.json
      depends_on: "ruby-tests"

Jenkins

Jenkins requires a credentials entry for your QLTY_COVERAGE_TOKEN. To add this:

Navigate to Manage Jenkins → Manage Credentials
Select the appropriate credentials store (usually Jenkins → Global credentials)
Click Add Credentials
Set the Kind to Secret text
Enter your Qlty coverage token as the Secret
Set the ID to qlty-coverage-token
Add a Description (e.g., “Qlty Coverage Token”)
Click Create

Here’s an example Jenkinsfile with Ruby test coverage:

pipeline {
    agent any

    environment {
        GEM_HOME = "${WORKSPACE}/.gem"
        PATH = "${WORKSPACE}/.gem/bin:${env.PATH}"
        BUNDLE_PATH = "${WORKSPACE}/vendor/bundle"
        QLTY_COVERAGE_TOKEN = credentials('qlty-coverage-token')
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Install Dependencies') {
            steps {
                sh 'bundle install --jobs 4 --retry 3'
            }
        }

        stage('Run Tests') {
            steps {
                sh 'bundle exec rspec --require spec_helper'
            }
        }

        stage('Upload Coverage') {
            steps {
                sh '''
                    curl -sSL https://qlty.sh | sh
                    ~/.qlty/bin/qlty coverage publish coverage/coverage.json
                '''
            }
        }
    }

    post {
        always {
            cleanWs()
        }
    }
}

The credentials('qlty-coverage-token') function retrieves the secret text credential you created and makes it available as the QLTY_COVERAGE_TOKEN environment variable.

bitrise

Configure your QLTY_COVERAGE_TOKEN as a project level token. Check “Exposed to pull requests” in order to see diff coverage and total coverage in pull requests. The coverage data file(s) must be placed somewhere accessible to future steps. bitwrise’s documentation lists several options for managing files across steps. Assuming a coverage data file located at coverage.lcov, the following step uploads coverage to qlty.sh:

- script@1:
      inputs:
          - content: |-
                #!/usr/bin/env bash
                set -e
                set -o pipefail
                set -x

                curl https://qlty.sh | sh
                ~/.qlty/bin/qlty coverage publish coverage.lcov

Other CI Providers

As long as your CI environment meets the system requirements to install the Qlt CLI, you can publish coverage. Publishing coverage on any CI provider follows the same procedure:

Ensure a QLTY_COVERAGE_TOKEN is available in the environment
Install the Qlty CLI
Execute qlty coverage publish command with the correct arguments

For example:

# Assumes $QLTY_COVERAGE_TOKEN is set in the environment
curl https://qlty.sh | sh
qlty coverage publish coverage/lcov.info # Replace with path to your coverage data file(s)

For security-sensitive environments, you can verify the CLI’s integrity before installation.

The Qlty CLI coverage commands leverage as much information from its environment as possible to ease integration. This includes environment variables as well as git metadata from the current working directory. However, sometimes, such as when attempting to publish coverage from a CI provider not listed above, Qlty is unable to detect this information from its environment. In these cases, you will need to provide the missing metadata to qlty coverage publish as additional arguments. Because usually no more than 2 - 4 arguments are needed, we recommend a trial and error process, following the instructions provided by the command when it errors. Below are some common scenarios you may run into.

Path Validation Failures

qlty coverage publish validates that the source files specified in your coverage data files match the source files in your repository. This ensures that we’re able to properly associate coverage with your source files in Qlty Cloud. If we’re unable to validate that a majority (up to a configurable threshold) of the source files referenced exist in your repository, qlty coverage publish will error. Usually these validate failures this can be addressed with the --add-prefix and/or --strip-prefix arguments. See Path Fixing for more information.

Commit SHA Missing

A commit sha is always required when sending coverage data to Qlty.sh. The commit sha can be manually specified with: --override-commit-sha <COMMIT_SHA>

Reference Missing

When publishing coverage, Qlty associates the coverage data with a single reference. The reference can be one of:

a branch
a pull request
a git tag

If not inferred, provide one of the following:

pull request: --override-pr-number <PR_NUMBER>
branch: --override-branch <BRANCH_NAME> (can be used alongside --override-pr-number)
git tag: --override-git-tag <TAG_NAME> (cannot be combined with either --override-pr-number or --override-branch)

Commit Time Missing

A timestamp associated with the commit is also required. This can be specified with: --override-commit-time <OVERRIDE_COMMIT_TIME> The commit time can be a Unix timestamp (seconds since epoch) or a date in RFC3339/ISO8601 format.

Examples

QLTY_COVERAGE_TOKEN=QLTY_TOKEN qlty coverage publish --override-branch=main --override-commit-sha=e83c5163316f89bfbde7d9ab23ca2e25604af290 --override-commit=time=1704067200 lcov.info

QLTY_COVERAGE_TOKEN=QLTY_TOKEN qlty coverage publish --override-pr-number=34 --override-branch=feature-1 --override-commit-sha=e83c5163316f89bfbde7d9ab23ca2e25604af290 --override-commit=time=1704067200 lcov.info

QLTY_COVERAGE_TOKEN=QLTY_TOKEN qlty coverage publish --override-git-tag=v0.2.2 --override-commit-sha=e83c5163316f89bfbde7d9ab23ca2e25604af290 --override-commit=time=1704067200 lcov.info

Advanced CI Configuration

Parallelized Tests

When running tests in parallel, you’ll need to merge the coverage reports before uploading. Qlty supports both client-side and server-side merging.

Client-side Merging

Collect all coverage reports into a single location and upload them together:

qlty coverage publish report1.lcov report2.lcov report3.lcov

Server-side Merging

Upload each part separately and specify the total number of parts:

# Part 1
qlty coverage publish --total-parts-count=3 report1.lcov

# Part 2
qlty coverage publish --total-parts-count=3 report2.lcov

# Part 3
qlty coverage publish --total-parts-count=3 report3.lcov

Learn more about Coverage Merging.

Multiple Test Suites

If you have different types of tests (unit, integration, e2e), you can track them separately using coverage tags:

# Upload unit test coverage
qlty coverage publish --tag=unit unit-coverage.lcov

# Upload integration test coverage
qlty coverage publish --tag=integration integration-coverage.lcov

Learn more about Coverage Tags.

Environment Detection

What is environment detection?

Environment detection is Qlty CLI’s ability to infer, without user input, metadata about your build from CI provider-specific environment variables. For example, CircleCI exposes the commit being built in an environment variable CIRCLE_SHA1. Without specific environment detection, you can still use our coverage publisher with a few extra command line arguments. See our documentation on Other CI Providers.

Adding Environment Detection

If you want automatic environment detection:

Double check our Qlty repository for the most up to date list of supported CI providers.
Our coverage publisher is open source and you can add support by following the other provider examples in under an hour. (Provide an LLM with the other examples and a link to your CI provider’s list of available environment variables)

Troubleshooting

Common Issues

Authentication failures: Check that your token is correctly set as an environment variable
Report format not recognized: Make sure you’re using a supported coverage format
Path mismatches: Use the --strip-prefix and --add-prefix options to fix file paths
Missing metadata: Use --override-branch, --override-commit and --override-build-id if CI detection fails

Sunbelt Computer Software

PL/B Language Development and Support

Getting Started

Coverage In Your Workflow

Concepts

Features

Advanced Configuration

Reference

CI Integration / Uploader

CI Provider Integration

GitHub Actions

How the action verifies the Qlty CLI

CircleCI

Travis CI

GitLab CI

Buildkite

Jenkins

bitrise

Other CI Providers

Path Validation Failures

Commit SHA Missing

Reference Missing

Commit Time Missing

Examples

Advanced CI Configuration

Parallelized Tests

Client-side Merging

Server-side Merging

Multiple Test Suites

Environment Detection

What is environment detection?

Adding Environment Detection

Troubleshooting

Common Issues

See Also

Sunbelt Computer Software

PL/B Language Development and Support

​CI Provider Integration

​GitHub Actions

​How the action verifies the Qlty CLI

​CircleCI

​Travis CI

​GitLab CI

​Buildkite

​Jenkins

​bitrise

​Other CI Providers

​Path Validation Failures

​Commit SHA Missing

​Reference Missing

​Commit Time Missing

​Examples

​Advanced CI Configuration

​Parallelized Tests

​Client-side Merging

​Server-side Merging

​Multiple Test Suites

​Environment Detection

​What is environment detection?

​Adding Environment Detection

​Troubleshooting

​Common Issues

​See Also

CI Provider Integration

GitHub Actions

How the action verifies the Qlty CLI

CircleCI

Travis CI

GitLab CI

Buildkite

Jenkins

bitrise

Other CI Providers

Path Validation Failures

Commit SHA Missing

Reference Missing

Commit Time Missing

Examples

Advanced CI Configuration

Parallelized Tests

Client-side Merging

Server-side Merging

Multiple Test Suites

Environment Detection

What is environment detection?

Adding Environment Detection

Troubleshooting

Common Issues

See Also