Running Testcontainers in OpenShift Pipelines with Docker-in-Docker

Container technologies have had a very strong influence in application development. In addition to the well documented value of containers related to deployment and management of applications, one of the biggest impacts has been the ability to create sets of full integration tests which can run anywhere.

The traditional way to test applications was heavily rooted in mocking due to the inability for the developers to run external services in their local development environment. The reason for mocking is to be able to programmatically replicate expected results from an external service, such as a database or messaging broker. Developers would create mock implementations using framework like Mockito and substitute these mock implementations when running in an environment where the services are not available. The problems pop up due to the limitations associated with mocking such as the inability to properly test database schema changes or transactions.

With containers, these service availability constraints can be eliminated. Developers can now spin up all kinds of infrastructure and services quickly and easily. Developers can use tools like docker or podman to start the external services in whatever environment they like and use those services for testing. For example, instead of writing a bunch of mock implementations for data access, which require their own care and maintenance, developers can now run suites of tests which can cover the full stack, not just the application code.

Enter Testcontainers

The next step in the evolution was to bring container orchestration into the testing lifecycle. The first iteration was based on the build tool lifecycles. For example, in maven, a developer could use a maven plugin to create and destroy resources as needed in the maven lifecycle. However, these quickly evolved and were replaced by more agile java libraries like Testcontainers. The Testcontainers library provided a lightweight, easily customizable set of apis to plug-in containers of all types directly into JUnit. At first glance, Testcontainers seems to be the ultimate enabler for building a very powerful and thorough set of testing capabilities for any project. Developers can install docker on their local machine and run full integration tests using a wide array of industry standard services.

Testcontainers finds my local daemon managed by Docker Desktop

The Issue: CICD

The problem comes from Testcontainers native integration with Docker. Testcontainers expects a docker daemon to be generally available on the host in which it’s running, typically located at /var/run/docker.sock. If you are running some legacy CICD products which run on a single host, this is generally not a problem. However, when you want to use something that’s container based, such as Tekton, then the issue rears it’s ugly head. The containers used to perform builds do not expose a docker daemon and therefore, the Testcontainers library will not be able to do its job.

Where is my unix domain socket for Docker? It’s not there.


For the solution, we need to be able to run docker. However, because we are running a Tekton pipeline, we are going to be running in containers so the solution is to be able to run docker in a container. This is where docker-in-docker comes in handy. According to the docker blog:

What’s special in my dind? Almost nothing! It is built with a regular Dockerfile. Let’s see what is in that Dockerfile.

First, it installs a few packages: lxc and iptables (because Docker needs them), and ca-certificates (because when communicating with the Docker index and registry, Docker needs to validate their SSL certificates).

The Dockerfile also indicates that /var/lib/docker should be a volume. This is important, because the filesystem of a container is an AUFS mountpoint, composed of multiple branches; and those branches have to be “normal” filesystems (i.e. not AUFS mountpoints). In other words, /var/lib/docker, the place where Docker stores its containers, cannot be an AUFS filesystem. Therefore, we instruct Docker that this path should be a volume. Volumes have many purposes, but in this scenario, we use them as a pass-through to the “normal” filesystem of the host machine. The /var/lib/docker directory of the nested Docker will live somewhere in /var/lib/docker/volumes on the host system.

Now that we have a solution for being able to run docker inside of a container, we need to figure out how to plug this into the task.

Tekton Sidecar

Luckily, Tekton has an answer for this. Sidecars allow additional containers to be configured and spun up on a task pod. For our example, we would be looking for a maven container to execute the appropriate goals but we need to have a running docker daemon located at /var/run/docker.sock and available to the JUnit test lifecycle.

The Task

Here’s the full yaml for an example maven build task.

kind: Task
  name: maven-build-task
    - name: source
    - name: maven-settings
    - name: MAVEN_IMAGE
      type: string
      description: Maven base image
      default: #tag: latest
    - name: GOALS
      description: maven goals to run
      type: array
        - "package"
    - name: MAVEN_MIRROR_URL
      description: The Maven repository mirror url
      type: string
      default: ""
    - name: SERVER_USER
      description: The username for the server
      type: string
      default: ""
      description: The password for the server
      type: string
      default: ""
    - name: PROXY_USER
      description: The username for the proxy server
      type: string
      default: ""
    - name: PROXY_PASSWORD
      description: The password for the proxy server
      type: string
      default: ""
    - name: PROXY_PORT
      description: Port number for the proxy server
      type: string
      default: ""
    - name: PROXY_HOST
      description: Proxy server Host
      type: string
      default: ""
      description: Non proxy server host
      type: string
      default: ""
    - name: PROXY_PROTOCOL
      description: Protocol for the proxy ie http or https
      type: string
      default: "http"
    - name: CONTEXT_DIR
      type: string
      description: >-
        The context directory within the repository for sources on
        which we want to execute maven goals.
      default: "."
    - name: mvn-settings
      script: |
        #!/usr/bin/env bash
        [[ -f $(workspaces.maven-settings.path)/settings.xml ]] && \
        echo 'using existing $(workspaces.maven-settings.path)/settings.xml' && exit 0
        cat > $(workspaces.maven-settings.path)/settings.xml <<EOF
            <!-- The servers added here are generated from environment variables. Don't change. -->
            <!-- ### SERVER's USER INFO from ENV ### -->
            <!-- The mirrors added here are generated from environment variables. Don't change. -->
            <!-- ### mirrors from ENV ### -->
            <!-- The proxies added here are generated from environment variables. Don't change. -->
            <!-- ### HTTP proxy from ENV ### -->
        if [ -n "$(params.PROXY_HOST)" -a -n "$(params.PROXY_PORT)" ]; then
          if [ -n "$(params.PROXY_USER)" -a -n "$(params.PROXY_PASSWORD)" ]; then
          if [ -n "$(params.PROXY_NON_PROXY_HOSTS)" ]; then
          sed -i "s|<!-- ### HTTP proxy from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
        if [ -n "$(params.SERVER_USER)" -a -n "$(params.SERVER_PASSWORD)" ]; then
          sed -i "s|<!-- ### SERVER's USER INFO from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
        if [ -n "$(params.MAVEN_MIRROR_URL)" ]; then
          xml="    <mirror>\
          sed -i "s|<!-- ### mirrors from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
    - name: mvn-goals
      image: $(params.MAVEN_IMAGE)
      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
      command: ["/usr/bin/mvn"]
        - -s
        - $(workspaces.maven-settings.path)/settings.xml
        - "$(params.GOALS)"
        - -ntp
        - mountPath: /var/run/
          name: dind-socket
    - image: docker:20.10-dind
      name: docker
        privileged: true
        - mountPath: /var/lib/docker
          name: dind-storage
        - mountPath: /var/run/
          name: dind-socket
    - name: dind-storage
      emptyDir: {}
    - name: dind-socket
      emptyDir: {}

In this task, the docker sidecar is started and the docker daemon is mounted on a shared volume, dind-socket. That same volume is mounted in the maven container on the mount path expected by Testcontainers, namely /var/run. This gives us access to the running docker daemon and makes it available for use by Testcontainers.

Adding in the Proper Permissions

If you try to run this task in OpenShift without any additional configuration, it will not work. OpenShift, by default, does not allow containers to be run in privileged mode which enhances the overall platform security. But in this case, we need to run in privileged mode because of docker. Therefore, we need to update the security for the service account running our tasks.

We want to isolate the permissions to a single privileged service account to the cicd namespace. Of course, this is assuming all of your pipelines run in a single namespace and are generally managed as a centralized service. If this isn’t the case, then each pipeline administrator will need to setup their own privileged service account for their own namespace. To accomplish this, let’s create a new service account and provide the correct scc to it.

oc project cicd
oc create sa dind
oc adm policy add-scc-to-user privileged -z dind

This serviceAccountName can be specified in the TriggerTemplate for the pipeline and can be narrowed to only apply to the maven build step that needs the access.

kind: TriggerTemplate
  name: maven-build-trigger-template
    - name: git-repo-url
      description: The git repository url
    - name: git-repo-name
      description: The name of the rep
    - name: git-revision
      description: The git revision
    - name: git-ref
      description: The name of the ref

  - apiVersion:
    kind: PipelineRun
      generateName: maven-build-pipeline-$(tt.params.git-repo-name)-
      serviceAccountName: pipeline
        - taskName: maven-build-task
           serviceAccountName: dind
        name: maven-build-pipeline
        - name: git-repo-url
          value: $(tt.params.git-repo-url)
        - name: git-repo-name
          value: $(tt.params.git-repo-name)
        - name: git-revision
          value: $(tt.params.git-revision)
        - name: git-ref
          value: $(tt.params.git-ref)
      - name: workspace
              - ReadWriteOnce
                storage: 500Mi

Happy Building!

Leave a Reply

Your email address will not be published. Required fields are marked *