Integrating Buildkite | Cloudsmith Docs
Documentation
Book a demo

CI/CD

Integrating Buildkite

Buildkite is a platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure. Use Buildkite to orchestrate and manage your own fleet of build hosts, from containers, to cloud instances, to bare metal servers.

No Code Uploading

The Cloudsmith CLI gives you full control when connecting to any CI/CD process, allowing you to upload any of our support formats or query your repositories. Just configure your API Key, install the CLI, and you'll be all set.

Getting started

Integrating Cloudsmith into your Buildkite pipeline is as simple as installing the Cloudsmith CLI during one of your pipeline steps and then using the cloudsmith push command to upload the artifacts from your pipeline to your Cloudsmith repository.

Adding your API key to your Buildkite environment

For details on obtaining your Cloudsmith API-Key see:

In order to use the Cloudsmith CLI within a Buildkite pipeline, you'll need to configure the environment on your build host(s) with your Cloudsmith API Key. Your Cloudsmith API Key should be considered as a secret and should not be committed into any source repositories or stored as a plain text variable in the Buildkite pipeline.

One of the easiest ways to set up your environment is to use Buildkite environment hooks, the environment hooks are scripts on a build host that run before each command in a pipeline.
A simple example to set up your Cloudsmith API key would be:

shell
set -euo pipefail
if [[ "$BUILDKITE_PIPELINE_SLUG" == "your-pipeline-name" ]]; then
       export CLOUDSMITH_API_KEY="abcdefghijklmnop1234567890"
fi

For more information on hooks, please see the Buildkite documentation here. There are several other methods of handling secrets in a Buildkite pipeline, you can find more details here

Authenticating with OpenID Connect (OIDC)

OpenID Connect (OIDC) removes the need to store a Cloudsmith API key on your build hosts. In each job, the Cloudsmith CLI requests an OIDC token from the Buildkite agent and exchanges it for a short-lived Cloudsmith access token. There is no long-lived credential to rotate or protect.

To authenticate with OIDC:

  1. Create a Cloudsmith service account and give it access to the target repository. The CLI authenticates as this service account. Note its slug.

  2. Create an OIDC provider in your Cloudsmith workspace with these settings:

    • Provider URL: https://agent.buildkite.com
    • Required OpenID token claims: an organization_slug claim set to the slug of your Buildkite organization. To allow only one pipeline, you can also add a pipeline_slug claim.
    • Service accounts: the service account from step 1.

    For more information, see Buildkite in the OpenID Connect documentation.

  3. Remove CLOUDSMITH_API_KEY from the agent environment, for example, from the environment hook. The CLI uses an API key before it tries OIDC.

  4. Set the CLOUDSMITH_WORKSPACE and CLOUDSMITH_SERVICE_SLUG environment variables, then run the CLI:

    yaml
    env:
  CLOUDSMITH_WORKSPACE: "your-workspace"
  CLOUDSMITH_SERVICE_SLUG: "your-service-account"

steps:
  - label: "Push package"
    commands:
      - cloudsmith whoami --verbose
      - cloudsmith push FORMAT WORKSPACE/REPOSITORY FILENAME

    The step must run on an agent that has the CLI installed. For more information, see Adding the Cloudsmith CLI to your Buildkite pipeline.

  5. Check the output of cloudsmith whoami --verbose. When OIDC authentication succeeds, the output includes lines similar to the following:

    text
    Authentication Method: OIDC Auto-Discovery
  Source: OIDC via Buildkite (org: your-workspace, service: your-service-account)

    If the CLI reuses a cached Cloudsmith token, the source shows OIDC [cached] instead.

The CLI requests the cloudsmith audience by default. To request a different audience, set the --oidc-audience option or the CLOUDSMITH_OIDC_AUDIENCE environment variable. For more information about how the CLI detects a Buildkite job, see Automatically discover OIDC credentials.

Adding the Cloudsmith CLI to your Buildkite pipeline

The Cloudsmith CLI ships as a standalone binary, so no Python or other runtime is required on the build host. The following examples use a Linux build host; the install script requires curl (or GNU wget), tar, and gzip. To add the Cloudsmith CLI to your pipeline, include the install script in one of your pipeline steps in your pipeline.yml file. The install script prints the installation result as key=value lines; linking the parsed executable into /usr/local/bin makes the cloudsmith command available to later pipeline steps that run on the same agent:

yaml
steps:
  - label: "Install Cloudsmith CLI"
    commands:
      - install_output="$(curl -fsSL https://install.cloudsmith.com/raw/versions/latest/cli.sh | sh -s -- --version 1.20.1)"
      - executable="$(printf '%s\n' "$install_output" | sed -n 's/^executable=//p')"
      - sudo ln -sf "$executable" /usr/local/bin/cloudsmith

An example of a setup step that installs Ruby, some tooling, and the Cloudsmith CLI would look like:

yaml
steps:
  - label: "PreBuild"
    commands:
      - sudo apt update
      - sudo apt-get install ruby ruby-dev rubygems build-essential ca-certificates curl -y
      - sudo gem install --no-document fpm
      - install_output="$(curl -fsSL https://install.cloudsmith.com/raw/versions/latest/cli.sh | sh -s -- --version 1.20.1)"
      - executable="$(printf '%s\n' "$install_output" | sed -n 's/^executable=//p')"
      - sudo ln -sf "$executable" /usr/local/bin/cloudsmith

You can also add the install commands to a pipeline step using the Buildkite Web UI, if not using a pipeline.yml file:

Pushing a build artifact to a Cloudsmith repository

To push an artifact from your Buildkite pipeline to a Cloudsmith repository, you use the cloudsmith push command in one of your pipeline steps:

yaml
steps:
  - label: "Push package"
    commands:
      - cloudsmith push FORMAT WORKSPACE/REPOSITORY FILENAME

Please see the Cloudsmith CLI documentation for more details of the syntax of the cloudsmith push command and the Supported Formats page for examples of the cloudsmith push command for each supported format.

An example of this command in a step that builds a Debian package would look like:

yaml
steps:
  - label: "BuildAndPushPackage"
    commands:
      - make
      - fpm -f -s dir -t deb -v 1.0.0 -n cloudsmith-buildkite-test .
      - cloudsmith push deb demo/buildkite-demo/debian/buster cloudsmith-buildkite-test_1.0.0_amd64.deb

Again, you can also add the cloudsmith push command to a pipeline step using the Buildkite Web UI, if not using a pipeline.yml file:

Buildkite Pipeline Steps

Pipeline steps in Buildkite are stateless and as a result, if you have a fleet of agents then each step is not guaranteed to run on the same agent. This means that if you build an artifact in one step, and want to push it to a Cloudsmith repository in a subsequent step, the artifact will need to be stored temporarily and then retrieved by the push step. Buildkite provides temporary artifact storage that you can use for this purpose (see here for more details).