Container technologies have had a very strong influence in application development. In addition to the well documented value of containers related to deployment and management of applications, one of the biggest impacts has been the ability to create sets of full integration tests which can run anywhere.
The traditional way to test applications was heavily rooted in mocking due to the inability for the developers to run external services in their local development environment. The reason for mocking is to be able to programmatically replicate expected results from an external service, such as a database or messaging broker. Developers would create mock implementations using framework like Mockito and substitute these mock implementations when running in an environment where the services are not available. The problems pop up due to the limitations associated with mocking such as the inability to properly test database schema changes or transactions.
With containers, these service availability constraints can be eliminated. Developers can now spin up all kinds of infrastructure and services quickly and easily. Developers can use tools like docker or podman to start the external services in whatever environment they like and use those services for testing. For example, instead of writing a bunch of mock implementations for data access, which require their own care and maintenance, developers can now run suites of tests which can cover the full stack, not just the application code.
Enter Testcontainers
The next step in the evolution was to bring container orchestration into the testing lifecycle. The first iteration was based on the build tool lifecycles. For example, in maven, a developer could use a maven plugin to create and destroy resources as needed in the maven lifecycle. However, these quickly evolved and were replaced by more agile java libraries like Testcontainers. The Testcontainers library provided a lightweight, easily customizable set of apis to plug-in containers of all types directly into JUnit. At first glance, Testcontainers seems to be the ultimate enabler for building a very powerful and thorough set of testing capabilities for any project. Developers can install docker on their local machine and run full integration tests using a wide array of industry standard services.
The Issue: CICD
The problem comes from Testcontainers native integration with Docker. Testcontainers expects a docker daemon to be generally available on the host in which it’s running, typically located at /var/run/docker.sock. If you are running some legacy CICD products which run on a single host, this is generally not a problem. However, when you want to use something that’s container based, such as Tekton, then the issue rears it’s ugly head. The containers used to perform builds do not expose a docker daemon and therefore, the Testcontainers library will not be able to do its job.
Docker-in-Docker
For the solution, we need to be able to run docker. However, because we are running a Tekton pipeline, we are going to be running in containers so the solution is to be able to run docker in a container. This is where docker-in-docker comes in handy. According to the docker blog:
What’s special in mydind? Almost nothing! It is built with a regular Dockerfile. Let’s see what is in that Dockerfile. First, it installs a few packages:lxcandiptables(because Docker needs them), andca-certificates(because when communicating with the Docker index and registry, Docker needs to validate their SSL certificates). The Dockerfile also indicates that/var/lib/dockershould be a volume. This is important, because the filesystem of a container is an AUFS mountpoint, composed of multiple branches; and those branches have to be “normal” filesystems (i.e. not AUFS mountpoints). In other words,/var/lib/docker, the place where Docker stores its containers, cannot be an AUFS filesystem. Therefore, we instruct Docker that this path should be a volume. Volumes have many purposes, but in this scenario, we use them as a pass-through to the “normal” filesystem of the host machine. The/var/lib/dockerdirectory of the nested Docker will live somewhere in/var/lib/docker/volumeson the host system.
Now that we have a solution for being able to run docker inside of a container, we need to figure out how to plug this into the task.
Tekton Sidecar
Luckily, Tekton has an answer for this. Sidecars allow additional containers to be configured and spun up on a task pod. For our example, we would be looking for a maven container to execute the appropriate goals but we need to have a running docker daemon located at /var/run/docker.sock and available to the JUnit test lifecycle.
The Task
Here’s the full yaml for an example maven build task.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: maven-build-task
spec:
workspaces:
- name: source
- name: maven-settings
params:
- name: MAVEN_IMAGE
type: string
description: Maven base image
default: gcr.io/cloud-builders/mvn@sha256:57523fc43394d6d9d2414ee8d1c85ed7a13460cbb268c3cd16d28cfb3859e641 #tag: latest
- name: GOALS
description: maven goals to run
type: array
default:
- "package"
- name: MAVEN_MIRROR_URL
description: The Maven repository mirror url
type: string
default: ""
- name: SERVER_USER
description: The username for the server
type: string
default: ""
- name: SERVER_PASSWORD
description: The password for the server
type: string
default: ""
- name: PROXY_USER
description: The username for the proxy server
type: string
default: ""
- name: PROXY_PASSWORD
description: The password for the proxy server
type: string
default: ""
- name: PROXY_PORT
description: Port number for the proxy server
type: string
default: ""
- name: PROXY_HOST
description: Proxy server Host
type: string
default: ""
- name: PROXY_NON_PROXY_HOSTS
description: Non proxy server host
type: string
default: ""
- name: PROXY_PROTOCOL
description: Protocol for the proxy ie http or https
type: string
default: "http"
- name: CONTEXT_DIR
type: string
description: >-
The context directory within the repository for sources on
which we want to execute maven goals.
default: "."
steps:
- name: mvn-settings
image: registry.access.redhat.com/ubi8/ubi-minimal:8.2
script: |
#!/usr/bin/env bash
[[ -f $(workspaces.maven-settings.path)/settings.xml ]] && \
echo 'using existing $(workspaces.maven-settings.path)/settings.xml' && exit 0
cat > $(workspaces.maven-settings.path)/settings.xml <<EOF
<settings>
<servers>
<!-- The servers added here are generated from environment variables. Don't change. -->
<!-- ### SERVER's USER INFO from ENV ### -->
</servers>
<mirrors>
<!-- The mirrors added here are generated from environment variables. Don't change. -->
<!-- ### mirrors from ENV ### -->
</mirrors>
<proxies>
<!-- The proxies added here are generated from environment variables. Don't change. -->
<!-- ### HTTP proxy from ENV ### -->
</proxies>
</settings>
EOF
xml=""
if [ -n "$(params.PROXY_HOST)" -a -n "$(params.PROXY_PORT)" ]; then
xml="<proxy>\
<id>genproxy</id>\
<active>true</active>\
<protocol>$(params.PROXY_PROTOCOL)</protocol>\
<host>$(params.PROXY_HOST)</host>\
<port>$(params.PROXY_PORT)</port>"
if [ -n "$(params.PROXY_USER)" -a -n "$(params.PROXY_PASSWORD)" ]; then
xml="$xml\
<username>$(params.PROXY_USER)</username>\
<password>$(params.PROXY_PASSWORD)</password>"
fi
if [ -n "$(params.PROXY_NON_PROXY_HOSTS)" ]; then
xml="$xml\
<nonProxyHosts>$(params.PROXY_NON_PROXY_HOSTS)</nonProxyHosts>"
fi
xml="$xml\
</proxy>"
sed -i "s|<!-- ### HTTP proxy from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
fi
if [ -n "$(params.SERVER_USER)" -a -n "$(params.SERVER_PASSWORD)" ]; then
xml="<server>\
<id>serverid</id>"
xml="$xml\
<username>$(params.SERVER_USER)</username>\
<password>$(params.SERVER_PASSWORD)</password>"
xml="$xml\
</server>"
sed -i "s|<!-- ### SERVER's USER INFO from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
fi
if [ -n "$(params.MAVEN_MIRROR_URL)" ]; then
xml=" <mirror>\
<id>mirror.default</id>\
<url>$(params.MAVEN_MIRROR_URL)</url>\
<mirrorOf>central</mirrorOf>\
</mirror>"
sed -i "s|<!-- ### mirrors from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
fi
- name: mvn-goals
image: $(params.MAVEN_IMAGE)
workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
command: ["/usr/bin/mvn"]
args:
- -s
- $(workspaces.maven-settings.path)/settings.xml
- "$(params.GOALS)"
- -ntp
volumeMounts:
- mountPath: /var/run/
name: dind-socket
sidecars:
- image: docker:20.10-dind
name: docker
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dind-storage
- mountPath: /var/run/
name: dind-socket
volumes:
- name: dind-storage
emptyDir: {}
- name: dind-socket
emptyDir: {}In this task, the docker sidecar is started and the docker daemon is mounted on a shared volume, dind-socket. That same volume is mounted in the maven container on the mount path expected by Testcontainers, namely /var/run. This gives us access to the running docker daemon and makes it available for use by Testcontainers.
Adding in the Proper Permissions
If you try to run this task in OpenShift without any additional configuration, it will not work. OpenShift, by default, does not allow containers to be run in privileged mode which enhances the overall platform security. But in this case, we need to run in privileged mode because of docker. Therefore, we need to update the security for the service account running our tasks.
We want to isolate the permissions to a single privileged service account to the cicd namespace. Of course, this is assuming all of your pipelines run in a single namespace and are generally managed as a centralized service. If this isn’t the case, then each pipeline administrator will need to setup their own privileged service account for their own namespace. To accomplish this, let’s create a new service account and provide the correct scc to it.
oc project cicd
oc create sa dind
oc adm policy add-scc-to-user privileged -z dindThis serviceAccountName can be specified in the TriggerTemplate for the pipeline and can be narrowed to only apply to the maven build step that needs the access.
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: maven-build-trigger-template
spec:
params:
- name: git-repo-url
description: The git repository url
- name: git-repo-name
description: The name of the rep
- name: git-revision
description: The git revision
- name: git-ref
description: The name of the ref
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: maven-build-pipeline-$(tt.params.git-repo-name)-
spec:
serviceAccountName: pipeline
serviceAccountNames:
- taskName: maven-build-task
serviceAccountName: dind
pipelineRef:
name: maven-build-pipeline
params:
- name: git-repo-url
value: $(tt.params.git-repo-url)
- name: git-repo-name
value: $(tt.params.git-repo-name)
- name: git-revision
value: $(tt.params.git-revision)
- name: git-ref
value: $(tt.params.git-ref)
workspaces:
- name: workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500MiHappy Building!