Antivirus Scanning for Docker images

Posted on June 27, 2026July 21, 2026 by vzurczak

It has been a very long time since my last post here.
Anyway, I had to dig into a topic I never really explored and I thought sharing these results may help someone, one day, maybe.

The Requirement

One of my customers delivers a solution for its on-premises customers as a set of Docker images and Helm charts. The security team of one of these cutomers has asked for an antivirus scan report for every version. We are not talking about CVEs (with tools like Trivy or Grype). No, the focus here is about antivirus scans.

This is something we never do for Docker images.
Most of images are based on Linux filesystems and run on Linux systems, for which very few virus exist. It is not that it is not feasible, but Linux virus are rare (such systems face other threats). And in the Docker ecosystem, if such an infected file had to be spotted, the image itself would quickly be removed and fixed.

So, I have been looking for an antivirus for Linux that we can use in an automated pipeline to generate such reports. ClamAV quickly emerged as the best (to not say the only) choice, even though this tool appears in some comparisons as not as efficient as more well-known antivirus solutions. In fact, these other solutions provide additional services (such as real-time system monitoring). Therefore, the comparison delta is not really on antivirus scans but about features that ClamAV does not provide. It fact, ClamAV is often used on Linux systems to scan e-mail attachments. This tool comes with a CLI, which is very convenient for a CI integration.

Now, the question is… how do we scan a Docker image with ClamAV?
Unlike Trivy as an example, there is no “docker mode”. Sure, ClamAV is available as a Docker image, but its CLI has no dedicated option to scan a Docker image or to interact with a container registry. The real interrogation was: if we save the Docker image as a tar, is the scan as efficient as if we had mounted the ClamAV binary directly in the file system? Let’s compare the two approaches.

Scan a Docker image as a TAR file

First, let’s consider this Dockerfile:

			
FROM alpine:3.23
ADD https://secure.eicar.org/eicar.com.txt /opt/ops/eicar.com.txt
ADD https://secure.eicar.org/eicar.com.txt /home/lambda/to.tmp

This is a basic Alpine image where we copy the EICAR file twice in two different locations.
The EICAR file is a special file whose signature is detected by antivirus Software and allows to perform some tests.

Now, let’s build the image and export as a TAR.
With this format, you only have blobs (or layers), in the same way you can see them in the file system of a container registry. Then, let’s scan it with ClamAV.

			
# Update the ClamAV image
docker pull clamav/clamav:stable
# Cache the virus database
docker volume create clam_db
# Create the image to scan
docker build -t test-av .
# Save it as a single file
docker save -o /tmp/test-av.tar test-av
# The file system is a set of blobs, with a directory
# /blobs/sha256 and various files.
# Update the database
docker run -ti --rm \
	--name "clamav" \
	-v /tmp/test-av.tar:/scan/test-av.tar \
	--mount source=clam_db,target=/var/lib/clamav \
	clamav/clamav:stable \
	freshclam
# Mount everything and launch the scan
docker run -ti --rm \
	--name "clamav" \
	-v /tmp/test-av.tar:/scan/test-av.tar \
	--mount source=clam_db,target=/var/lib/clamav \
	clamav/clamav:stable \
	clamscan /scan

		

Here is the result:

			
/scan/test-av.tar: Eicar-Test-Signature FOUND
----------- SCAN SUMMARY -----------
Known viruses: 3627885
Engine version: 1.5.2
Scanned directories: 1
Scanned files: 1
Infected files: 1
Data scanned: 68 B
Data read: 8.31 MiB (ratio 0.00:1)
Time: 11.504 sec (0 m 11 s)
Start Date: 2026:06:27 13:49:40
End Date:   2026:06:27 13:49:51

		

The signature is recognized.
However, you can notice that it does not say which file in the TAR file contains the signature, nor how many times it was found. The scan result will always contain a single line. Maybe not exactly the kind of details expected by the security team of our customer.

Scan a Docker image as an exported file system

Originally, I thought the best approach was mounting the ClamAV binary into a running container created from the image to scan and then run it. It appeared to not cover all the cases. Indeed, ClamAV is not available as a single binary. Instead, it has dependencies to libraries. And some libraries may change depending on the base image (consider Debian VS. Alpine images with musl / libc). Let’s not even mention distroless or scratch images.

So, the best solution is to extract the file system of the image, no matter what it is, launch ClamAV as previously, with the extracted file system in a mount point.

			
# Update the ClamAV image
docker pull clamav/clamav:stable
# Cache the virus database
docker volume create clam_db
# Create a container from the image but do not start it
docker create --name temporary_scan test-av
# Export the file system of the container
# (and fix permission issues for deletion)
mkdir /tmp/test-av-fs
docker export temporary_scan | tar \
	--no-same-owner \
	--no-same-permissions \
	--exclude='dev/*' \
	-xf - \
	-C /tmp/dg-fstar
# Here, we really have a Linux file system
# /bin, /dev, /etc, /home...
# Delete the container
docker rm temporary_scan
# Update the database
docker run -ti --rm \
	--name "clamav" \
	-v /tmp/test-av.tar:/scan/test-av.tar \
	--mount source=clam_db,target=/var/lib/clamav \
	clamav/clamav:stable \
	freshclam
# Mount everything and launch the scan
# -r: recursive scan
# -i: only show infected files
docker run -ti --rm \
	--name "clamav" \
	-v /tmp/test-av-fs:/scan \
	--mount source=clam_db,target=/var/lib/clamav \
	clamav/clamav:stable \
	clamscan -r -i /scan

		

Notice I used the “-i” option to only display the infected files.
Without this option, you get the whole list of scanned files (which may be good for a complete report).

Here is the result:

			
/scan/home/lambda/to.tmp: Eicar-Test-Signature FOUND
/scan/opt/ops/eicar.com.txt: Eicar-Test-Signature FOUND
----------- SCAN SUMMARY -----------
Known viruses: 3627885
Engine version: 1.5.2
Scanned directories: 102
Scanned files: 84
Infected files: 2
Data scanned: 8.73 MiB
Data read: 8.02 MiB (ratio 1.09:1)
Time: 11.924 sec (0 m 11 s)
Start Date: 2026:06:27 14:01:27
End Date:   2026:06:27 14:01:39

		

The good part here is that it shows exactly which files are infected (and thus how many times). This information is maybe not useful for the report reader but it is for the development team if it wants to understand the source and fix it.

I found few comments indicating the second approach could be more efficient from a memory point of view, as files are already extracted, while in the first method, they are extracted on the fly before being analyzed. The scan time is almost the same here, so this is not relevant in this precise case. But there may be cases where it makes a difference.

Tricky situation

Let’s now consider this Dockerfile:

			
FROM alpine:3.23
ADD https://secure.eicar.org/eicar.com.txt /opt/ops/eicar.com.txt
RUN rm /opt/ops/eicar.com.txt

Almost the same file, except the EICAR file will appear in the Docker layers but not in the resulting file system mounted in the container. Let’s verify our scan process. By exporting the image (first method), we get…

			
/scan/test-av.tar: Eicar-Test-Signature FOUND
----------- SCAN SUMMARY -----------
Known viruses: 3627917
Engine version: 1.5.3
Scanned directories: 1
Scanned files: 1
Infected files: 1
Data scanned: 17.56 MiB
Data read: 8.31 MiB (ratio 2.11:1)
Time: 12.330 sec (0 m 12 s)
Start Date: 2026:07:17 14:43:42
End Date: 2026:07:17 14:43:54

		

This is not surprising, since the file is present in one of the layers.
Let’s see with the second method, that scans the resulting file system.

			
----------- SCAN SUMMARY -----------
Known viruses: 3627917
Engine version: 1.5.3
Scanned directories: 100
Scanned files: 82
Infected files: 0
Data scanned: 8.73 MiB
Data read: 8.02 MiB (ratio 1.09:1)
Time: 11.545 sec (0 m 11 s)
Start Date: 2026:07:17 14:44:44
End Date:   2026:07:17 14:44:55

		

As expected, the file does not appear in the resulting file system and is thus not detected.

Conclusion

To be honest, I was not sure the first method would work.
An exported Docker image is a set of layers: when you open the first TAR, it does not have the same file structure than in the second attempt. So, I was curious to see if it could detect infected files among the Docker layers. And it did.

First, I had a preference for the second option.
I then thought about the case of a virus somewhere in a layer but not in the resulting file system. And what you mind is: will my servers and image repository host something that contain a virus? The first method is definitely more reliable. The second one should be reserved to obtain more precise reports and identify which files are affected (some sort of debug or deeper analysis).

Running ClamAV in a Docker container is also the most simple solution.
It updates the database on the first startup but needs to be updated manually then. To save time, in particular in a CI system, it is better to cache its database in a separate volume. You can just reuse the commands given above.

Combining Jenkins’ Job DSL and Shared Libraries for Docker Images Pipelines

Posted on April 17, 2020 by vzurczak

This is another article about Docker pipelines with Jenkins.
I had already written about Jenkins libraries to standardize Jenkins pipelines in an organization. This can be seen as a complement in an OpenShift context.

OpenShift is a solution based on Kubernetes and that brings additional features such as a portal, a catalog of solutions (including Jenkins), many security features and new K8s resources. Among these new resources, there is the notion of Build Config. A build config object allows to associate a source configuration with a Jenkins project. So, basically, you write a YAML file, indicating as an example the location of a git repository, the branch you want to build, etc. You then pass it to the oc client (oc apply -f my.yaml) and it will create a Jenkins project. You will find more information about build configurations on the OpenShift project.

The problem I got with build configs was that the generated project in Jenkins is a simple one.
It does not support multiple branches. Generating a multi-branch pipeline would be much better, but it is not feasible. I was then advised to look at Jenkins’ Job DSL. It relies on a Jenkins plug-in that from a seed project can populate and configure Jenkins projects. Thanks to this plug-in, I quickly got a Jenkins configuration as code.

Using the Job DSL

The first thing to do is to create a seed project in Jenkins.
It is a free-style project with a build step that executes a Job DSL script. Everything is documented here.

The following script is stored in a git repository.
I copy it in my seed’s configuration and that’s it. Here is what it looks like…

def gitBaseUrl = "https://some.gitlab/path/to/a/my/gitlab/group"
def gitRepos = [
	["docker-project-name-1":"path-to-project-1"],
	["docker-project-name-2":"path-to-project-2"]
]

for (gitRepo in gitRepos) {
	for ( e in gitRepo ) {

		// Create Jenkins folders and reference shared libraries
		folder("${e.value}") {
			properties {
			folderLibraries {
				libraries {
					libraryConfiguration {
						name("my-library-for-docker")
						defaultVersion('master')
						implicit(false)
						allowVersionOverride(true)
						includeInChangesets(true)
						retriever {
							modernSCM {
							scm {
							git {
								remote('https://some.gitlab/path/to/a/jenkins/library')
								credentialsId('my-credentials-id')
							}}}
						}
					}
				}
			}}
		}

		// Create multi-branch pipeline projects
		multibranchPipelineJob("${e.value}/${e.key}") {
			branchSources {
				branchSource {
					source { git {
						id("${e.value}-${e.key}")
						remote("${gitBaseUrl}/${e.value}/${e.key}.git")
						credentialsId('middleware-jenkins-gittoken')
					}}
					
					strategy {
						defaultBranchPropertyStrategy {
							props {
								// Do not trigger build on branch scan
								noTriggerBranchProperty()
							}
						}
					}
				}
			}
			
			// Listen to changes in branches and new tags
			configure { node ->
				node / sources / data / 'jenkins.branch.BranchSource' / source / traits {
					'jenkins.plugins.git.traits.BranchDiscoveryTrait'()
					'jenkins.plugins.git.traits.TagDiscoveryTrait'()
				}
			}

			// Verify new branches and new tags everyday
			triggers {
				cron('@daily')
			}

			// What to do with old builds?
			orphanedItemStrategy {
				discardOldItems {
					numToKeep(10)
					daysToKeep(-1)
				}
			}
		}
	}
}

Here, I explicitly reference the projects I want to generate.
I reuse the git’s folder structure and project it in Jenkins. Shared libraries can be defined globally in Jenkins, but also on folders. The Job DSL does not allow to configure global stuff. But it allows to do it on folders. The multi-branch pipelines are quite easy to understand. We consider both branches and tags.

If you need to customize the script, you can find examples on Github and most of all in your Jenkins instance.

Once the script is set in your seed project, just build it in Jenkins and it will update your Jenkins projects. The build is idem-potent. You can run it as many times as you want. It will overwrite the current settings. So, if you need to update the DSL script, just do it and run a new build, everything will be updated. This is useful if you add new shared libraries. In the same way, the Jenkins plug-in tracks the projects it has created. So, if I remove a project from my list, with the default behavior, it will not delete the Jenkins related project but only disable it (it will be read-only).

I have investigated about directly referencing the script instead of copying it.
It seems you cannot automatically propagate changes from the sources, you have to validate the changes first. I guess this is a security feature. I have not searched a lot of about this, this is not a big matter for us, for the moment.

Normalized Pipeline for Docker Images

The job DSL defines a shared library at the folder level.
Here is a simple pipeline library (myDockerPipeline.groovy) for our Docker images.

1. Checkout the sources.
2. Verify some assertions on our Dockerfile.
3. Build the image.
4. Publish it in the right repository (not the same for branches and tags).
5. Perform an AQUA analysis of development images. We assume another pipeline handles images built from a tag (not shown here).

There is no test in this pipeline, although we could add some.

In Jenkins, it is achieved with…

// Shared library that defines the generic pipeline for Docker images.
def call( Map pipelineParams ) {

	// Basic properties
	def tokenString = pipelineParams.gitRepoPath.replace('/', '-') + "--" + pipelineParams.gitRepoName
	def imageName = pipelineParams.gitRepoName.replace('-dockerfile', '')
	def label = 'base'

	// Complex properties (configure build trigger through an URL and a token)
	properties([
		pipelineTriggers([
		[$class: 'GenericTrigger',
			genericVariables: [
				[key: 'ref', value: '$.ref'],
				[
					key: 'before',
					value: '$.before',
					expressionType: 'JSONPath',
					regexpFilter: '',
					defaultValue: ''
				]
			],
			genericRequestVariables: [
				[key: 'requestWithNumber', regexpFilter: '[^0-9]'],
				[key: 'requestWithString', regexpFilter: '']
			],
			genericHeaderVariables: [
				[key: 'headerWithNumber', regexpFilter: '[^0-9]'],
				[key: 'headerWithString', regexpFilter: '']
			],
			
			causeString: 'Triggered after a change on $ref',
			token: "${tokenString}",
			printContributedVariables: true,
			printPostContent: true,
			regexpFilterText: '$ref',
			regexpFilterExpression: 'refs/heads/' + BRANCH_NAME
		]
		])
	])

	podTemplate(label: label, cloud: 'openshift', containers: [
		containerTemplate(
			name: "jnlp",
			image: "my-jenkins-jnlp:v3.11",
			envVars: [
				envVar(key: 'ENV_DOCKER_HOST', value: 'remote-docker-engine'),
				envVar(key: 'ENV_LOCAL_IMG_NAME', value: 'my-team/' + imageName),
				envVar(key: 'ENV_DEV_IMG_NAME', value: 'my-team/dev/' + imageName),
				envVar(key: 'ENV_RELEASE_IMG_NAME', value: 'my-team/releases/' + imageName)
			]
		),
		containerTemplate(
			name: "aqua",
			image: "our-aqua-image:v3.11",
			command: 'cat', 
			ttyEnabled: true,
			envVars: [
				envVar(key: 'ENV_DEV_IMG_NAME', value: 'my-team/dev/' + imageName)
			]
		)
	],
	serviceAccount: "jenkins") {
		node(label) {
			container(name: 'jnlp') {

				// Checkout
				stage('Checkout') {
					checkout scm
				}
				
				// Lint
				stage('Linting') {
					// Do we have the right labels in the Dockerfile?
					verifyDockerfile()
				}

				// Build
				stage('Build') {
					sh 'docker -H "${ENV_DOCKER_HOST}" build -t "$ENV_LOCAL_IMG_NAME" .'
				}

				// Stages executed for a TAG
				if(env.TAG_NAME) {
					stage('Publish') {

						sh'''#!/bin/bash
						# Push to BUILD
						docker -H "${ENV_DOCKER_HOST}" tag \
							"$ENV_LOCAL_IMG_NAME" \
							"${ENV_RELEASE_IMG_NAME}":"${TAG_NAME}"

						docker -H "${ENV_DOCKER_HOST}" push \
							"${ENV_RELEASE_IMG_NAME}":"${TAG_NAME}"

						# Push to releases
						docker -H "${ENV_DOCKER_HOST}" tag \
							"$ENV_LOCAL_IMG_NAME" \
							"${ENV_RELEASE_IMG_NAME}":"${TAG_NAME}"

						docker -H "${ENV_DOCKER_HOST}" push \
							"${ENV_RELEASE_IMG_NAME}":"${TAG_NAME}"
						'''

					}
				}

				// Simple branch
				else if(env.BRANCH_NAME) {
					stage('Publish') {

						sh'''#!/bin/bash
						# Push to BUILD
						docker -H "${ENV_DOCKER_HOST}" tag \
							"$ENV_LOCAL_IMG_NAME" \
							"${ENV_DEV_IMG_NAME}":"${BRANCH_NAME}"

						docker -H "${ENV_DOCKER_HOST}" push \
							"${ENV_DEV_IMG_NAME}":"${BRANCH_NAME}"
						'''

					}
				}
			}

			// Here, we use the AQUA plug-in to scan images
			// (we reference a remote AQUA installation).
			container(name: 'aqua') {
				if(env.BRANCH_NAME) {
					stage("Hosted : Aqua CI/CD Scan Image") {
						ansiColor('css') {
							aqua customFlags: '',
							hideBase: false,
							hostedImage: '"${ENV_DEV_IMG_NAME}":"${BRANCH_NAME}"',
							localImage: '',
							locationType: 'hosted',
							notCompliesCmd: 'echo "The AQUA has failed."',
							onDisallowed: 'fail',
							showNegligible: true, 
							registry: 'our-registry-id',
							register: true
						}
					}
				}
			}
		}
	}
}

The main thing to notice is what is commented as the complex properties.
By default, our Jenkins installation has a global listener activated: https://our-jenkins-url/generic-webhook-trigger/invoke
So, anyone sending a HTTP notification to this address could trigger something. The question is how to use it to trigger a specific job, for a specific branch? Well, we use a simple token for that. Here, the token is based on the repository name and location. As an example, our git repo “some-path/some-project” will be associated with the following token: some-path–some-project

So, if someone notifies https://our-jenkins-url/generic-webhook-trigger/invoke?token=some-path–some-project, then the job’s configuration will catch it. The other properties allow to filter the right branch and only trigger the right Jenkins job.

Another element to notice is the custom verifyDockerfile library.
Here is its code (verifyDockerfile.groovy).

def call() {
	def dockerfileContent = readFile('Dockerfile')
	assert dockerfileContent.contains('LABEL maintainer=') : "No maintainer was found."
	assert dockerfileContent.contains('"common-mailbox@our-domain.com"') : "The maintainer must be common-mailbox@our-domain.com"
	// OK, the check is somehow basic
}

It allows to verify some parts of the Dockerfile.
Eventually, here is an example of our pipeline (Jenkinsfile).

@Library('my-library-for-docker') _

myDockerPipeline(
    gitRepoPath: 'repo-path',
    gitRepoName: 'repo-name'
)

This way, the content of our Jenkinsfile is minimalist.
We can update our library at any time without having to update the Jenkinsfile. No matter how many Docker images you maintain, you are sure all of them follow a same pipeline.

As a reminder, all the groovy libraries must be located under the vars directory in your project.

About the Source Branch Plug-ins

You must have noticed I referenced the projects by hand at the beginning of the seed’s script.
It is possible to avoid this and to use a plug-in to scan directly your sources. There are existing ones for Github, Bitbucket and GitLAB.

You have to define an organization folder and its properties.
Once the seed is built, it will scan the Git forge and create multi-branch pipeline projects (for the branches that have a Jenkinsfile). Here is a sample for GitLAB.

organizationFolder('GitLab Organization Folder') {
    displayName('GitLAB')

    // "Projects"
    organizations {
        gitLabSCMNavigator {
            projectOwner("my-gitlab-group")
            credentialsId("personal-token")
            serverName("my-gitlab-url")
            traits {
                subGroupProjectDiscoveryTrait() // discover projects inside subgroups
                gitLabBranchDiscovery {
                    strategyId(3) // discover all branches
                }
            }
        }
    }
	
    // "Project Recognizers"
    projectFactories {
        workflowMultiBranchProjectFactory {
            scriptPath 'Jenkinsfile'
        }
    }
	
    // "Orphaned Item Strategy"
    orphanedItemStrategy {
        discardOldItems {
            daysToKeep(10)
            numToKeep(5)
        }
    }
	
    // "Scan Organization Folder Triggers" 
    triggers {
        periodicFolderTrigger {
            interval('60')
        }
    }
}

As you can see, it is a little bit less verbose.
We have not chosen this approach though. Overall, the manual declaration is suitable for now. We also noticed some glitches with the GitLAB plug-in, mainly about character encoding and avatars. This is not a big issue by itself, fixes will come for that.

End-to-end tests for applications in Kubernetes

Posted on July 9, 2019April 17, 2020 by vzurczak

This article aims at introducing a small library to ease end-to-end testing of applications in Kubernetes environments.

An overview of what already existed

When typing “kubernetes e2e” or “kubernetes end to end” on Google, the first result I got was about testing a K8s cluster or component. It is what the project’s team is using to test the development of the Kubernetes project. This is not what I wanted. My goal was to test an application I packaged for K8s, not K8s itself.

Terratest is another solution I found. We have the same goal, but viewing this project made me realize I did not want a solution involving advanced programming. We have DevOps that can develop and maintain operative aspects. But they are not so many and most hardly know the Go language. All the team members learned kubectl and Helm commands easily. A scripting solution would be better. This would avoid the choice of a programming language (Go, Java…) and a thus a lot of arguing / reinventing.

Since we mostly had Helm packages, used by several internal projects, I tried to focus on Helm. I immediately found the solution used by the official Helm project. There are interesting parts, such as the linting and version checks. This can be convenient if you setup a internal Helm repository and that you want to make every chart use the same rules. There are also commands to check an installation. Anyway, this is tailored for a collection of Helm charts and still not adapted to what I wanted.

I then found the unit test plug-in for Helm. The principle is to create a YAML file that contains a set of tests. A Helm chart has to be deployed in the environment. The YAML files are passed to the chart that verifies them. This is an interesting solution, but it mostly tests the templating of your chart. Not the applicative behavior.

Testing an application in a Kubernetes environment means being able to deploy it, adapt the topology (scale replicas), verify everything works, execute scenarios and check assertions at various stages. The solution that fit the best this requirement was EUFT. This small project relies on BATS (Bash Automated Testing System), a script framework that allows to write and execute unit tests by using scripts. EUFT is in fact a solution to deploy a small K8s cluster and run BATS tests inside. Examples of tests are available in this repository. I also found out afterwards that Hashicorp was using the same technique for some of their Helm packages.

If I liked the principle of BATS, all the tests used by EUFT and Hashicorp are a little bit complex to maintain. Not everyone in our project is a script god. Besides, we do not want to deploy a K8s cluster in our tests: we want to use an existing one, with the same settings than our production one. This is important because of permissions and network policies. Running e2e tests in a ephemeral K8s installation is too limited. However, EUFT gave me a direction since I have not found anything else.

The DETIK library

I was not really inspired for a name…
DETIK stands for « DevOps End-to-End Testing In Kubernetes ». The idea is to write tests by using scripts, running them with BATS, and having a simple syntax, almost in natural language, to write assertions about resources in Kubernetes. With kubectl or Helm commands, a few knowledge in scripts (BASH, Ruby, Python, whatever…) and this library, anyone should be able to write applicative tests and be able to maintain them with very few efforts.

In addition to performing actions on the cluster, I also wanted to support the execution of scenarios. Scenarios can imply topology adaptations, but also user actions. BATS can integrate with many solutions, such as Selenium or Cypress for end-user scenarios, or Gatling for performance tests. With all these tools, it becomes possible to test an application from end-to-end in a K8s environment.

Example

The following example is taken from the Git repository.
It show the test of a Helm package. A part of the syntax comes from BATS.

#!/usr/bin/env bats

###########################################
# An example of tests for a Helm package
# that deploys Drupal and Varnish
# instances in a K8s cluster.
###########################################

load "/home/testing/lib/detik.bash"
DETIK_CLIENT_NAME="kubectl"
pck_version="1.0.1"

function setup() {
	cd $BATS_TEST_DIRNAME
}

function verify_helm() {
 	helm template ../drupal | kubectl apply --dry-run -f -
}


@test "verify the linting of the chart" {

	run helm lint ../drupal
	[ "$status" -eq 0 ]
}


@test "verify the deployment of the chart in dry-run mode" {

	run verify_helm
	[ "$status" -eq 0 ]	
}


@test "package the project" {

	run helm -d /tmp package ../drupal
	# Verifying the file was created is enough
	[ -f /tmp/drupal-${pck_version}.tgz ]
}


@test "verify a real deployment" {

	[ -f /tmp/drupal-${pck_version}.tgz ]

	run helm install --name my-test \
		--set varnish.ingressHost=varnish.test.local \
		--set db.ip=10.234.121.117 \
		--set db.port=44320 \
		--tiller-namespace my-test-namespace \
		/tmp/drupal-${pck_version}.tgz

	[ "$status" -eq 0 ]
	sleep 10

	# PODs
	run verify "there is 1 pod named 'my-test-drupal'"
	[ "$status" -eq 0 ]

	run verify "there is 1 pod named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	# Postgres specifics
	run verify "there is 1 service named 'my-test-postgres'"
	[ "$status" -eq 0 ]

	run verify "there is 1 ep named 'my-test-postgres'"
	[ "$status" -eq 0 ]

	run verify \
		"'.subsets[*].ports[*].port' is '44320' " \
		"for endpoints named 'my-test-postgres'"
	[ "$status" -eq 0 ]

	run verify \
		"'.subsets[*].addresses[*].ip' is '10.234.121.117' " \
		"for endpoints named 'my-test-postgres'"
	[ "$status" -eq 0 ]

	# Services
	run verify "there is 1 service named 'my-test-drupal'"
	[ "$status" -eq 0 ]

	run verify "there is 1 service named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	run verify "'port' is '80' for services named 'my-test-drupal'"
	[ "$status" -eq 0 ]

	run verify "'port' is '80' for services named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	# Deployments
	run verify "there is 1 deployment named 'my-test-drupal'"
	[ "$status" -eq 0 ]

	run verify "there is 1 deployment named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	# Ingress
	run verify "there is 1 ingress named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	run verify \
		"'.spec.rules[*].host' is 'varnish.test.local' " \
		"for ingress named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	run verify \
		"'.spec.rules[*].http.paths[*].backend.serviceName' " \
		"is 'my-test-varnish' for ingress named 'my-test-varnish'"
	[ "$status" -eq 0 ]

	# PODs should be started
	run try "at most 5 times every 30s " \
		"to get pods named 'my-test-drupal' " \
		"and verify that 'status' is 'running'"
	[ "$status" -eq 0 ]

	run try "at most 5 times every 30s " \
		"to get pods named 'my-test-varnish' " \
		"and verify that 'status' is 'running'"
	[ "$status" -eq 0 ]

	# Indicate to other tests that the deployment succeeded
	echo "started" > tests.status.tmp
}


@test "verify the deployed application" {

	if [[ ! -f tests.status.tmp ]]; then
		skip " The application was not correctly deployed... "
	fi

	rm -rf /tmp.drupal.html
	curl -sL http://varnish.test.local -o /tmp/drupal.html
	[ -f ${BATS_TMPDIR}/drupal.html ]

	grep -q "<title>Choose language | Drupal</title>" /tmp/drupal.html
	grep -q "Set up database" /tmp/drupal.html
	grep -q "Install site" /tmp/drupal.html
	grep -q "Save and continue" /tmp/drupal.html
}


@test "verify the undeployment" {

	run helm del --purge my-test --tiller-namespace my-test-namespace
	[ "$status" -eq 0 ]
	[ "$output" == "release \"my-test\" deleted" ]

	run verify "there is 0 service named 'my-test'"
	[ "$status" -eq 0 ]

	run verify "there is 0 deployment named 'my-test'"
	[ "$status" -eq 0 ]

	sleep 60
	run verify "there is 0 pod named 'my-test'"
	[ "$status" -eq 0 ]
}


@test "clean the test environment" {
	rm -rf tests.status.tmp
}

These unit tests include the linting of the chart, a dry-run deployment, but also a real deployment with a basic topology. After deploying it, we verify assertions on K8s resources. Once the application (a simple Drupal) is started, we get the content of the web site and make sure it contains some expected words and sentences. We could replace it by a Selenium scenario.

Executing the bats my-test-file.bats command would start the execution.
A successful run would show the following output:

bats my-test-file.bats
1..7

✓ 1 verify the linting of the chart
✓ 2 verify the deployment of the chart in dry-run mode
✓ 3 package the project
✓ 4 verify a real deployment
✓ 5 verify the deployed application
✓ 6 verify the undeployment
✓ 7 clean the test environment

The command "bats my-test-file.bats" exited with 0.

Errors appear like below.

...

✗ 1 verify the linting of the chart
    (in test file my-test(file.bats, line 14)
     `[ "$status" -eq 0 ]' failed

...

Library Principles

Assertions are used to generate kubectl queries.
The output is extracted and compared to the values given as parameters.

There are very few queries in fact.
However, they work with all the kinds of resources of Kubernetes. That includes native K8s objects (POD, services….) but also OpenShift elements (routes, templates…) or custom resources (e.g. the upcoming Helm v3 objects).

Queries can be run with kubectl or with oc (the OpenShift client).
You only have to specify the client name in the DETIK_CLIENT_NAME variable (and make sure the client is available in the path).

With this, you can verify pre and post-conditions when using a Kubernetes client, Helm or even operators.

Usage

The library is available as a single file.
It can be donwloaded from this Github repository. The syntax is documented in the readme of the project.

A Dockerfile is provided as a basis in the project.
It embeds a kubectl client, a Helm client, BATS and the DETIK library. Depending on your cluster configuration, you might want to add other items (e.g. to log into your cluster).

Continuous Integration

The project is documented and explains how to execute (and write) such tests on your own machine. But the real interest of such tests is to be run in the last parts of an automated pipeline.

Here is a simple Jenkinsfile (for a Jenkins pipeline).

def label = "${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
podTemplate(label: label, containers: [
	containerTemplate(
			name: 'jnlp',
			image: 'jnlp-slave-alpine:3.27-1-alpine'), 
	containerTemplate(
			name: 'detik',
			image: 'detik:LATEST',
			ttyEnabled: true,
			alwaysPullImage: true, 
			envVars: [
				envVar(key: 'http_proxy', value: 'http://proxy.local:3128'),
				envVar(key: 'https_proxy', value: 'http://proxy.local:3128'),
				envVar(key: 'TILLER_NAMESPACE', value: 'my-test-namespace')
			])
]) {

	node(label) {
		container(name: 'jnlp') {
			stage('Checkout') {
				checkout scm
			}
		}

		container(name: 'ci-docker') {
			stage('Login') {
				withCredentials([usernamePassword(
						credentialsId: 'k8s-credentials',
						passwordVariable: 'K8S_PASSWORD',
						usernameVariable: 'K8S_USERNAME')]) {

					echo 'log into the cluster...'
					// TODO: it depends on your cluster configuration
				}
			}

			stage('Build and Test') {
				sh 'bats tests/main.bats'
			}
		}
	}
}

It can easily be adapted for Travis or GitLab CI.
You will find more examples on Github.

News from April 2020: the project has joined the BATS Core organization on Github.

Using Graylog for Centralized Logs in K8s platforms and Permissions Management

Posted on February 13, 2019February 14, 2019 by vzurczak

This article explains how to centralize logs from a Kubernetes cluster and manage permissions and partitionning of project logs thanks to Graylog (instead of ELK). The fact is that Graylog allows to build a multi-tenant platform to manage logs. Let’s take a look at this.

Reminders about logging in Kubernetes

As it is stated in Kubernetes documentation, there are 3 options to centralize logs in Kubernetes environements.

The first one is about letting applications directly output their traces in other systems (e.g. databases). This approach always works, even outside Docker. However, it requires more work than other solutions. Not all the applications have the right log appenders. It can also become complex with heteregenous Software (consider something less trivial than N-tier applications). Eventually, log appenders must be implemented carefully: they should indeed handle network failures without impacting or blocking the application that use them, while using as less resources as possible. So, althouth it is a possible option, it is not the first choice in general.

The second solution is specific to Kubernetes: it consists in having a side-car container that embeds a logging agent. This agent consumes the logs of the application it completes and sends them to a store (e.g. a database or a queue). This approach is better because any application can output logs to a file (that can be consumed by the agent) and also because the application and the agent have their own resources (they run in the same POD, but in different containers). Side-car containers also gives the possibility to any project to collect logs without depending on the K8s infrastructure and its configuration. However, if all the projets of an organization use this approach, then half of the running containers will be collecting agents. Even though log agents can use few resources (depending on the retained solution), this is a waste of resources. Besides, it represents additional work for the project (more YAML manifests, more Docker images, more stuff to upgrade, a potential log store to administrate…). A global log collector would be better.

That’s the third option: centralized logging. Rather than having the projects dealing with the collect of logs, the infrastructure could set it up directly. The idea is that each K8s minion would have a single log agent and would collect the logs of all the containers that run on the node. This is possible because all the logs of the containers (no matter if they were started by Kubernetes or by using the Docker command) are put into the same file. What kubectl log does, is reading the Docker logs, filtering the entries by POD / container, and displaying them. This approach is the best one in terms of performances. What is difficult is managing permissions: how to guarantee a given team will only access its own logs. Not all the organizations need it. Small ones, in particular, have few projects and can restrict access to the logging platform, rather than doing it IN the platform. Anyway, beyond performances, centralized logging makes this feature available to all the projects directly. They do not have to deal with logs exploitation and can focus on the applicative part.

Centralized Logging in K8s

Centralized logging in K8s consists in having a daemon set for a logging agent, that dispatches Docker logs in one or several stores. The most famous solution is ELK (Elastic Search, Logstash and Kibana). Logstash is considered to be greedy in resources, and many alternative exist (FileBeat, Fluentd, Fluent Bit…). The daemon agent collects the logs and sends them to Elastic Search. Dashboards are managed in Kibana.

Things become less convenient when it comes to partition data and dashboards. Elastic Search has the notion of index, and indexes can be associated with permissions. So, there is no trouble here. But Kibana, in its current version, does not support anything equivalent. All the dashboards can be accessed by anyone. Even though you manage to define permissions in Elastic Search, a user would see all the dashboards in Kibana, even though many could be empty (due to invalid permissions on the ES indexes). Some suggest to use NGinx as a front-end for Kibana to manage authentication and permissions. It seems to be what Red Hat did in Openshift (as it offers user permissions with ELK). What I present here is an alternative to ELK, that both scales and manage user permissions, and fully open source. This relies on Graylog.

Here is what Graylog web sites says: « Graylog is a leading centralized log management solution built to open standards for capturing, storing, and enabling real-time analysis of terabytes of machine data. We deliver a better user experience by making analysis ridiculously fast, efficient, cost-effective, and flexible. »

I heard about this solution while working on another topic with a client who attended a conference few weeks ago. And indeed, Graylog is the solution used by OVH’s commercial solution of « Log as a Service » (in its data platform products). This article explains how to configure it. It is assumed you already have a Kubernetes installation (otherwise, you can use Minikube). To make things convenient, I document how to run things locally.

Architecture

Graylog is a Java server that uses Elastic Search to store log entries.
It also relies on MongoDB, to store metadata (Graylog users, permissions, dashboards, etc).

What is important is that only Graylog interacts with the logging agents. There is no Kibana to install. Graylog manages the storage in Elastic Search, the dashboards and user permissions. Elastic Search should not be accessed directly. Graylog provides a web console and a REST API. So, everything feasible in the console can be done with a REST client.

Deploying Graylog, MongoDB and Elastic Search

Obviously, a production-grade deployment would require a highly-available cluster, for both ES, MongoDB and Graylog. But for this article, a local installation is enough. A docker-compose file was written to start everything. As ES requires specific configuration of the host, here is the sequence to start it:

sudo sysctl -w vm.max_map_count=262144
docker-compose -f compose-graylog.yml up

You can then log into Graylog’s web console at http://localhost:9000 with admin/admin as credentials. Those who want to create a highly available installation can take a look on Graylog’s web site.

Deploying the Collecting Agent in K8s

As discussed before, there are many options to collect logs.
I chose Fluent Bit, which was developed by the same team than Fluentd, but it is more performant and has a very low footprint. There are also less plug-ins than Fluentd, but those available are enough.

What we need to is get Docker logs, find for each entry to which POD the container is associated, enrich the log entry with K8s metadata and forward it to our store. Indeed, Docker logs are not aware of Kubernetes metadata. We therefore use a Fluent Bit plug-in to get K8s meta-data. I saved on Github all the configuration to create the logging agent. It gets logs entries, adds Kubernetes metadata and then filters or transforms entries before sending them to our store.

The message format we use is GELF (which a normalized JSON message supported by many log platforms). Notice there is a GELF plug-in for Fluent Bit. However, I encountered issues with it. As it is not documented (but available in the code), I guess it is not considered as mature yet. Instead, I used the HTTP output plug-in and built a GELF message by hand. Here is what it looks like before it is sent to Graylog.

{
"short_message":"2019/01/13 17:27:34 Metric client health check failed...",
"_stream":"stdout",
"_timestamp":"2019-01-13T17:27:34.567260271Z",
"_k8s_pod_name":"kubernetes-dashboard-6f4cfc5d87-xrz5k",
"_k8s_namespace_name":"test1",
"_k8s_pod_id":"af8d3a86-fe23-11e8-b7f0-080027482556",
"_k8s_labels":{},
"host":"minikube",
"_k8s_container_name":"kubernetes-dashboard",
"_docker_id":"6964c18a267280f0bbd452b531f7b17fcb214f1de14e88cd9befdc6cb192784f",
"version":"1.1"
}

Eventually, we need a service account to access the K8s API.
Indeed, to resolve to which POD a container is associated, the fluent-bit-k8s-metadata plug-in needs to query the K8s API. So, it requires an access for this.

You can find the files in this Git repository. The service account and daemon set are quite usual. What really matters is the configmap file. It contains all the configuration for Fluent Bit: we read Docker logs (inputs), add K8s metadata, build a GELF message (filters) and sends it to Graylog (output). Take a look at the Fluent Bit documentation for additionnal information.

Configuring Graylog

There many notions and features in Graylog.
Only few of them are necessary to manage user permissions from a K8s cluster. First, we consider every project lives in its own K8s namespace. If there are several versions of the project in the same cluster (e.g. dev, pre-prod, prod) or if they live in different clusters does not matter. What is important is to identify a routing property in the GELF message. So, when Fluent Bit sends a GELF message, we know we have a property (or a set of properties) that indicate(s) to which project (and which environment) it is associated with. In the configmap stored on Github, we consider it is the _k8s_namespace property.

Now, we can focus on Graylog concepts.
We need…

An input

An input is a listener to receive GELF messages.
You can create one by using the System > Inputs menu. In this example, we create a global one for GELF HTTP (port 12201). There are many options in the creation dialog, including the use of SSL certificates to secure the connection.

Indices

Graylog indices are abstractions of Elastic indexes. They designate where log entries will be stored. You can associate sharding properties (logical partition of the data), retention delay, replica number (how many instances for every shard) and other stuff to a given index. Every projet should have its own index: this allows to separate logs from different projects. Use the System > Indices to manage them.

A project in production will have its own index, with a bigger retention delay and several replicas, while a developement one will have shorter retention and a single replica (it is not a big issue if these logs are lost).

Streams

A stream is a routing rule. They can be defined in the Streams menu. When a (GELF) message is received by the input, it tries to match it against a stream. If a match is found, the message is redirected into a given index.

When you create a stream for a project, make sure to check the Remove matches from ‘All messages’ stream option. This way, the log entry will only be present in a single stream. Otherwise, it will be present in both the specific stream and the default (global) one.

The stream needs a single rule, with an exact match on the K8s namespace (in our example).
Again, this information is contained in the GELF message. Notice that the field is _k8s_namespace in the GELF message, but Graylog only displays k8s_namespace in the proposals. The initial underscore is in fact present, even if not displayed.

Do not forget to start the stream once it is complete.

Dashboards

Graylog’s web console allows to build and display dashboards.
Make sure to restrict a dashboard to a given stream (and thus index). Like for the stream, there should be a dashboard per namespace. Using the K8s namespace as a prefix is a good option.

Graylog provides several widgets…
Take a look at the documentation for further details.

Roles

Graylog allows to define roles.
A role is a simple name, coupled to permissions (roles are a group of permissions). You can thus allow a given role to access (read) or modify (write) streams and dashboards. For a project, we need read permissions on the stream, and write permissions on the dashboard. This way, users with this role will be able to view dashboards with their data, and potentially modifying them if they want.

Roles and users can be managed in the System > Authentication menu.

Users

Apart the global administrators, all the users should be attached to roles.
These roles will define which projects they can access. You can consider them as groups. When a user logs in, Graylog’s web console displays the right things, based on their permissions.

There are two predefined roles: admin and viewer.
Any user must have one of these two roles. He (or she) may have other ones as well. When a user logs in, and that he is not an administrator, then he only has access to what his roles covers.

Clicking the stream allows to search for log entries.

Notice that there are many authentication mechanisms available in Graylog, including LDAP.

Summary

Graylog uses MongoDB to store metadata (stream, dashboards, roles, etc) and Elastic Search to store log entries. We define an input in Graylog to receive GELF messages on a HTTP(S) end-point. These messages are sent by Fluent Bit in the cluster.

When such a message is received, the k8s_namespace_name property is verified against all the streams.
When one matches this namespace, the message is redirected in a specific Graylog index (which is an abstraction of ES indexes). Only the corresponding streams and dashboards will be able to show this entry.

Eventually, only the users with the right role will be able to read data from a given stream, and access and manage dashboards associated with it. Logs are not mixed amongst projects. Isolation is guaranteed and permissions are managed trough Graylog.

In short : 1 project in an environment = 1 K8s namespace = 1 Graylog index = 1 Graylog stream = 1 Graylog role = 1 Graylog dashboard. This makes things pretty simple. You can obviously make more complex, if you want…

Testing Graylog

You can send sample requests to Graylog’s API.

# Found on Graylog's web site
curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://192.168.1.18:12201/gelf'

This one is a little more complex.

# Home made
curl -X POST -H 'Content-Type: application/json' -d '{"short_message":"2019/01/13 17:27:34 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.","_stream":"stdout","_timestamp":"2019-01-13T17:27:34.567260271Z","_k8s_pod_name":"kubernetes-dashboard-6f4cfc5d87-xrz5k","_k8s_namespace_name":"test1","_k8s_pod_id":"af8d3a86-fe23-11e8-b7f0-080027482556","_k8s_labels":{},"host":"minikube","_k8s_container_name":"kubernetes-dashboard","_docker_id":"6964c18a267280f0bbd452b531f7b17fcb214f1de14e88cd9befdc6cb192784f","version":"1.1"}' http://localhost:12201/gelf

Feel free to invent other ones…

Automating stuff

Every features of Graylog’s web console is available in the REST API.
It means everything could be automated. Every time a namespace is created in K8s, all the Graylog stuff could be created directly. Project users could directly access their logs and edit their dashboards.

Check Graylog’s web site for more details about he API. When you run Graylog, you can also access the Swagger definition of the API at http://localhost:9000/api/api-browser/

Going further

The resources in this article use Graylog 2.5.
The next major version (3.x) brings new features and improvements, in particular for dashboards. There should be a new feature that allows to create dashboards associated with several streams at the same time (which is not possible in version 2.5, a dashboard being associated with a single stream – and so a single index). That would allow to have transverse teams, with dashboards that span across several projects.

See https://www.graylog.org/products/latestversion for more details.

Hints for Tests

If you do local tests with the provided compose, you can purge the logs by stopping the compose stack and deleting the ES container (docker rm graylogdec2018_elasticsearch_1). Then restart the stack.

If you remove the MongoDB container, make sure to reindex the ES indexes.
Or delete the Elastic container too.

Sunbelt Computer Software

PL/B Language Development and Support

Vincent Zurczak's Blog

My technical blog