Conforma E2E Tests

Description

These tests validate Conforma (Enterprise Contract) end-to-end functionality -- covering Tekton Chains signing, attestation verification, and enterprise contract policy evaluation. They are ported from konflux-ci/e2e-tests and run against an upstream Konflux instance deployed with Tekton Chains.

The test suite uses the Ginkgo framework and is labeled with ec for selective execution.

Prerequisites

• An OpenShift or KinD cluster with Konflux deployed (including Tekton Chains and Enterprise Contract)
• QUAY_TOKEN environment variable set (base64-encoded Docker config for Quay.io registry access)
• The following CRD APIs available on the cluster:
  • Snapshot (Application API)
  • PipelineRun (Tekton)
  • EnterpriseContractPolicy (Conforma CRDs)

What the tests cover

1. Infrastructure checks

   • Tekton Chains controller is running
   • Cosign signing secret (signing-secrets) is present with cosign.key, cosign.pub, and cosign.password

2. Image build, signing, and attestation

   • A buildah-demo pipeline builds and pushes a container image
   • Tekton Chains creates a cosign signature (.sig) and attestation (.att) for the image

3. Enterprise Contract verification (verify-enterprise-contract task)

   • Succeeds when the SLSA provenance policy is met
   • Reports FAILURE (non-strict mode) when test policies are not satisfied
   • Fails (strict mode) when test policies are not satisfied
   • Fails when an unexpected/wrong signing key is used

4. EC CLI validation

   • Error handling: verifies proper failure message when attestation doesn't match the public key
   • Multi-image validation: accepts a list of image references for batch verification

5. Release policy

   • Red Hat products pass the full Red Hat policy rule collection
   • Untrusted task references are detected and rejected
   • Unpinned task bundle references produce a WARNING

How to run

Option A: Full pipeline (provisions a new cluster)

1. Provision an OpenShift cluster

   Use cluster-bot or a similar tool:

   workflow-launch hypershift-hostedcluster-workflow 4.15
   

2. Install the OpenShift Pipelines operator

   kubectl apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-pipelines-operator
     namespace: openshift-operators
   spec:
     channel: latest
     name: openshift-pipelines-operator-rh
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

3. Create required secrets

   The following secrets must exist in the pipeline namespace:

   Secret	Purpose
   mapt-kind-secret	AWS credentials for KinD cluster provisioning/deprovisioning
   konflux-e2e-secrets	E2E test secrets (e.g., quay-token)
   konflux-test-infra	OCI registry credentials for artifact storage
   konflux-operator-e2e-credentials	Operator-level credentials for E2E

4. Apply the pipeline definition

   kubectl apply -f ./.tekton/pipelines/conforma-e2e/pipeline.yaml

5. Start the pipeline

   tkn pipeline start conforma-e2e-pipeline \
     --param git-url=https://github.com/conforma/e2e-tests.git \
     --param revision=main \
     --param oci-container-repo=quay.io/conforma/e2e-tests \
     --param oci-container-repo-credentials-secret=konflux-test-infra \
     --use-param-defaults \
     --showlog

   The pipeline will:

   • Provision a KinD cluster on AWS
   • Deploy Konflux with Tekton Chains via the Konflux operator
   • Run the Ginkgo test suite
   • Collect artifacts and push to OCI
   • Deprovision the cluster

Testing a custom CLI build

To test a conforma/cli PR against the e2e suite, pass the custom CLI params:

tkn pipeline start conforma-e2e-pipeline \
  --param git-url=https://github.com/conforma/e2e-tests.git \
  --param revision=main \
  --param oci-container-repo=quay.io/conforma/e2e-tests \
  --param oci-container-repo-credentials-secret=konflux-test-infra \
  --param custom-ec-cli-url=https://github.com/conforma/cli.git \
  --param custom-ec-cli-revision=<commit-sha-or-branch> \
  --use-param-defaults \
  --showlog

The pipeline clones the CLI repo, builds the ec binary, layers it onto quay.io/conforma/cli:latest using crane append, and pushes a throwaway image tagged cli-pr-<short-sha>. All verify-enterprise-contract test cases then run against the custom image instead of the released one.

When both params are empty (the default), the pipeline uses the standard task bundle and released CLI image.

Option B: Run directly against an existing Konflux cluster

If you already have a Konflux cluster running:

export KUBECONFIG=/path/to/your/kubeconfig
export QUAY_TOKEN="$(base64 -w0 < ~/.docker/config.json)"
export TEST_ENVIRONMENT=upstream
go run github.com/onsi/ginkgo/v2/ginkgo -v --label-filter="ec" ./cmd

Or using the Makefile:

export KUBECONFIG=/path/to/your/kubeconfig
export QUAY_TOKEN="$(base64 -w0 < ~/.docker/config.json)"
export TEST_ENVIRONMENT=upstream
make test-e2e

Configuration

Environment variable	Required	Description
KUBECONFIG	Yes	Path to kubeconfig for the target cluster
QUAY_TOKEN	Yes	Base64-encoded Docker config for Quay.io registry
TEST_ENVIRONMENT	No	Set to upstream for upstream Konflux deployments
QUAY_E2E_ORGANIZATION_ENV	No	Quay.io organization for test images (defaults to redhat-appstudio-qe)
E2E_APPLICATIONS_NAMESPACE	No	Override the generated test namespace
CUSTOM_EC_CLI_IMAGE	No	Custom CLI container image (set automatically by the pipeline when custom-ec-cli-url is provided)
CUSTOM_EC_TASK_YAML	No	Path to task YAML to patch with the custom image (set automatically by the pipeline)
KLOG_VERBOSITY	No	Kubernetes client logging verbosity (default: 1)

Project structure

cmd/e2e_test.go                        # Test entrypoint and BeforeSuite setup
tests/contract/contract.go             # Enterprise Contract test scenarios
pkg/
  clients/
    common/controller.go               # Kubernetes helper operations
    kubernetes/client.go               # K8s client initialization
    tekton/                            # Tekton-specific clients (bundles, chains, ECP, pipelines, signing)
  constants/constants.go               # Shared constants and timeouts
  framework/                           # Test framework (namespace creation, RBAC, reporting)
  utils/
    contract/policy.go                 # ECP policy helpers
    tekton/                            # Pipeline generators, matchers, cosign utilities, task image patching
.tekton/
  pipelines/conforma-e2e/pipeline.yaml # Tekton Pipeline for full CI execution
  conforma-e2e-pull-request.yaml       # PipelineRun trigger for pull requests