In a previous post I outlined setting up an NFS mount in kubernetes by taking advantage of new local storage provisioner feature in order to have storage over NFS, in this post I will be going over creating a docker image for the gitlab runner (for ARM), deploying it to your kubernetes cluster, and finally using that to automate building containers via a CI/CD pipeline on your raspberry pi. But first I’ll answer the question of why might you want this, which is a valid question. Say you have an application that you are going to build frequently or even somewhat frequently and you want to deploy it to docker in kubnernetes, well you could do that all by hand, or you could setup a CI pipeline in GitLab and associate private runner(s) which will compile the code, build the docker image, and push it to your registry automatically. You could also automate deploying the change but that is something I will leave to you to investigate. Finally one nice thing you could do to speed it up would be to build the binaries on on x86/amd64 doing a cross compile, then publish those artifacts and build the docker image on your raspberry pi runner in order to take advantage of the increased processor speed during compilation and only use the slower ARMv6 or ARMv7 processor when needed.

Building the image

Now that you’re on board, let’s build a docker image! The good news is that GitLab already creates an ARM binary for the GitLab runner, they just don’t create a docker image. I will be using the 1.8 binary in my case but you are free to use a newer/older version, you may need an older version if you are running it locally, or a newer version depending on when you read this. You could get the latest version binary here and include it in your repo, but instead what I suggest is grab it using your Dockerfile. Below is what I came up with for a Dockerfile:

FROM arm32v7/ubuntu:16.04

RUN apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y ca-certificates wget apt-transport-https vim nano tzdata python-pip && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN mkdir -p /etc/gitlab-runner/certs && \
    mkdir -p /home/gitlab-runner && \
    chmod -R 700 /etc/gitlab-runner && \
    wget -q https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-Linux-armhf -O /usr/bin/docker-machine && \
    chmod +x /usr/bin/docker-machine && \
    docker-machine --version && \
    pip install dumb-init


COPY entrypoint /
ADD https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-arm /usr/bin/gitlab-runner
RUN chmod +x /entrypoint
RUN chmod +x /usr/bin/gitlab-runner
ENTRYPOINT ["dumb-init", "/entrypoint"]
CMD ["run", "--user=gitlab-runner", "--working-directory=/home/gitlab-runner"]

This may seem quite complex if you are new to docker but I will explain it step by step,

1. First we get Ubuntu version 16.04 as our base image, this is what the official gitlab runners are using so I chose to use this for consistency, you could use a different one but your Dockerfile will likely vary if you do.
2. Next we install various dependencies like the official gitlab runner, but on top of those we also setup python-pip which will be important later. And of course we clean up after ourselves to keep the layer size as small as possible.
3. The second RUN statement comes almost directly out of the official docker container as well with a few exceptions:
   • We are not checking the md5 hashes because I like to live life on the wild side
   • We are installing an ARM version of docker machine vs x86_64 (duh)
   • We are installing dumb-init using pip instead of getting it from it’s releases in github. The reason for this is because the releases don’t have an ARM compatible version of it but by doing this it’s compiled on our ARM processor so we now have an ARM compatible version. The drawback of this is that it makes the image a bit bigger because you have to include python.
4. Finally comes the standard stuff of setting up the entrypoint, grabbing the latest gitlab runner binary (you could make this a build parameter and grab a specific version), and make sure permissions are set correctly.

At this point you have a valid docker image that should work for running the gitlab-ci runner in kubernetes on a raspberry pi. You should tag it properly and push it to your docker registry of choice, this will make it accessible across your cluster, as well as be useful further down the line once this is automated. If you would like to see my repo, including the entrypoint script (which I got from the official repo) you can check that out here on gitlab.

Deploying to Kubernetes

Having a docker image is no good if you don’t use it anywhere, this already assumes you have an ARM based kubernetes cluster configured, thankfully the gitlab-runner is pretty easy to setup. I’ve got the kubernetes deployment objects in my git repo here. There is only one changes you’ll need to make a little bit later to make this work but for now you can just apply these configs, however I do recommend you look at all the options for the ConfigMap at some point as some may be useful for you, especially the global environment variables.

Once they’re applied and the gitlab runner pod is running, it’s time to register it. This could be done automatically via a script but because this is a one time process I chose to do it by hand. In order to do this you’ll need to get your registration token by going to your projects Settings->CI / CD->Runner Settings. Once you have that kubectl exec into the pod and run the following commands:

[ssmith@localhost ~]$ kubectl exec --namespace=gitlab -it POD_NAME bash
root@gitlab-runner-5868bb44cd-wfj77:/# gitlab-runner register       
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.com
Please enter the gitlab-ci token for this runner:
YOUR_TOKEN_HERE
Please enter the gitlab-ci description for this runner:
[gitlab-runner-5868bb44cd-wfj77]: arm-k8s-runner
Please enter the gitlab-ci tags for this runner (comma separated):
ARMv7
Whether to run untagged builds [true/false]:
[false]: 
Whether to lock the Runner to current project [true/false]:
[true]: false
Registering runner... succeeded                     runner=t9m2P363
Please enter the executor: kubernetes, docker, parallels, ssh, virtualbox, docker+machine, docker-ssh+machine, docker-ssh, shell:
kubernetes
Runner registered successfully.

At this point your runner should be registered, though we aren’t done yet. If you go to Settings->CI / CD->Runner Settings you’ll probably see a warning icon on your runner stating it hasn’t connected yet. Click the edit button on your runner and from there you can get the token for your runner to talk to gitlab, put it in your ConfigMap under the token setting. Once that’s done apply the updated ConfigMap then delete the running pod so it starts a new one using the updated config and is able to talk to gitlab. Once it starts running you should be able to see that your new custom runner is online.

Now that everything appears to be working, if you didn’t lock the runner to the current project, and you want to use this runner with other projects you can go to the other projects Runner Settings and click “Enable for this project” and they will start building using that runner (assuming that the CI jobs have the correct tags). If you did lock the runner you can easily change that by unchecking the “Lock to current projects” box on the runner’s edit page.

Building and tagging from GitLab CI

Ok, now that you have a gitlab runner in your kubernetes cluster, time to build images with it. Why not start with the gitlab-runner itself, that way in the future when newer versions of GitLab come out, you simply need to add a gitlab-ci.yml to the repo that you store the Dockerfile in and you have a new gitlab-runner built and pushed to your registry of choice. Below is the gitlab-ci file I use:

---
variables:
  DOCKER_IMAGE: quay.io/wwsean08/gitlab-runner
  VERSION: v10.8.0

stages:
  - build

build:
  stage: build
  image: arm32v6/docker
  tags:
    - ARMv7
  script:
    - docker build -t ${DOCKER_IMAGE}:latest .
    - docker tag ${DOCKER_IMAGE}:latest ${DOCKER_IMAGE}:${VERSION}-${CI_JOB_ID}
    - docker login -u=${DOCKER_USER} -p=${DOCKER_PASS} quay.io
    - docker push ${DOCKER_IMAGE}:latest
    - docker push ${DOCKER_IMAGE}:${VERSION}-${CI_JOB_ID}
    - 'if [ ! -z ${CI_BUILD_TAG} ]; then
       docker tag ${DOCKER_IMAGE}:${VERSION}-${CI_JOB_ID} ${DOCKER_IMAGE}:${CI_BUILD_TAG};
       docker push  ${DOCKER_IMAGE}:${CI_BUILD_TAG};
       fi'

What this does is when a commit is pushed to my gitlab repo like say updating the version variable to v10.9.0 it causes the docker image to be built and pushed to quay.io. Finally if a build is happening because of a tag then tag whatever docker image is created with the same tag and push that as well. If you’re just getting start with GitLab you’ll noice that this will fail at first, That is because you need to configure two secret variables, DOCKER_USER and DOCKER_PASS in your project settings. With all that work done you now have an end to end ARM docker runtime where you can create the code, commit the code, have the code be built on ARM and if you want pushed to an artifact managmenet/docker registry like artifactory or quay and used down the line in things like kubernetes or other CI jobs.