Introduction

If you already worked a bit with Docker, maybe did you end-up in a situation were you needed to launch/create/manage a new container from an already existing one. For example, when you are running a dockerized Jenkins instance and need to create ephemeral containers in your pipeline to perform build tasks.

In such case, a quick Google search will offers you two main options :

• Docker inside Docker (DinD) : you perform a full Docker installation within you container. Basically, you run Docker on top of another Docker. I’m not a big fan of this approach and some people much more knowledgeable than me have explained why it might be a bad idea (spoiler: filesystem issues and risks of corruption). This options however seems to remain relevant in some cases, for instance if you are using Kubernetes.

• Docker outside Docker (DooD) : you share the Docker daemon socket of the host with your hosted container using the --volume or -v flags :

  docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock alpine

  Once inside the container, performing a request to the Docker API become as simple as installing curl and doing :

  curl --unix-socket /var/run/docker.sock http://localhost/containers/json

Demo

In my case, I’m running my containers on a small dedicated server using Docker compose. Mainly to perform per commit build jobs on a couple of open-source projects I follow. So DinD doesn’t seems like a good option.

Container escape using the Docker daemon socket

This left us with the DooD approach. However, as a former “offensive security” guy, I can’t accept to break the isolation between my containers and my host so easily. Indeed, as stated by the Docker documentation : “Docker’s out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker’s Engine API to contact the daemon.”

Which mean that in case of compromising of a container with access to the Docker socket, getting root access on the host can be easily achieved. All an attacker would have to do is to download the standalone docker client, launch a new busybox container mounting the root of the host to a directory, and then chroot into it :

docker run -u 0 -ti --rm -v /:/media/host/ busybox chroot /media/host

Demo

Note : This is the exact same reason why being in the docker group is considered equal having root privilege. It works from the host too :

Demo

Docker’s built-in hardening options

Docker allow to protect the daemon socket using :

1. Mutual client-server authentication through TLS and x509.
2. The Docker Engine managed plugin system, allowing to create an authorization plugin.

Regarding the first point, it’s a great feature. However, if a container with access to the Docker socket is compromised, the attacker will also most likely have access to the client certificate. In such case, the scenario presented previously is still relevant.

Regarding the second point, no official authorization plugin is provided. Multiple implementation exist, more or less functional and more or less maintained. I’m not a big fan of Docker plugins, since they are a bit anti-idiomatic with the “Build Once, Deploy Anywhere” philosophy.

Securing the Docker socket using Nginx ?

When I saw that the Docker socket give access to a Web API, my first though was “reverse-proxy”. If you are not familiar with this concept, a reverse-proxy stand between a web server and its clients and forward their requests to it. A reverse-proxy also allows to perform different tasks such as caching, load-balancing and… requests filtering ! To solve our issue, we can mount our Docker socket into a revers-proxy container, configure our Docker client to work with a remote socket and filter its HTTP requests to the API to only allow what we want.

One of the most commonly used reverse proxy is Nginx, which happens to have a ready to use official image on Dockerhub. Neat right ? No.

By doing so, we basically move our risk of compromising from the Jenkins container to the Nginx one. The attack surface is reduced (since Jenkins is not really the most secure software ever), but we still end-up with a full-sized Docker container, either based on alpine or debian, running a service with much more features than actually needed.

In some (most ?) cases it may seems like an acceptable solution, but let’s get paranoid and see how we can do it better.

Scratch images and Golang to the rescue

A neat way to have an as empty as possible Docker image is to build it “FROM scratch”, which basically means that your image will “only” contains what you put in it. No sh, no /bin, no /lib, no thing. The first question that should come up to your mind is : What about dependencies ?

If you are coding in Java you need the JVM, if you are coding in Python you need the Python interpreter, and if you are coding in C/C++ you need your original Jurassic Park t-shirt and Stranger Things in the background for maximum efficiency. We don’t want that.

Luckily, Golang is a language offering just the level of abstraction we need to code a reverse-proxy painlessly while offering all the capabilities of a compiled language, including static compilation.

Static compilation is a compilation process resulting in a standalone binary embedding all its dependencies. For instance, a software developed in C and statically compiled won’t rely on the system libc but the one it as been compiled with. The same thing can easily be achieved in Go :

CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' src/reverse-proxy.go

We got a plan !

With this information in mind, all we now have to do is apply the classic “make it work, then make it better” method :

1. Create a basic Go reverse-proxy and move it to a scratch image
2. Add filtering capabilities
3. Add HTTPS support, client authentication and per client filtering policies

Each of those points will be the subject of dedicated blog posts in this series.

Alternative solutions

If you are looking for a production ready solution and are not really interested by the DIY part, you may want to have a look at :

• Open Policy Agent, a general-purpose policy engine which happens to have a tutorial for this exact use case. However it requires to install a Docker plugin and doesn’t offer actual client authentication. Alternatively, the possibility to use it as an API Authorization engine might be interesting.
• Tyk, a lightweight API Gateway coded in Go, offering filtering capabilities. However, it doesn’t seem to filter parameters.
• Your current API gateway or reverse-proxy that may offer some kind of ACL capabilities for API requests.

References

• https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
• https://applatix.com/case-docker-docker-kubernetes-part/
• https://justi.cz/security/2018/09/13/alpine-apk-rce.html
• http://dance.csc.ncsu.edu/papers/codaspy17.pdf
• https://embano1.github.io/post/scratch/
• https://www.bogotobogo.com/cplusplus/libraries.php
• https://github.com/TykTechnologies/tyk
• https://www.openpolicyagent.org/docs/docker-authorization.html