Templating Config Files In Docker Containers

Configuration management does many things, and it does most of those quite poorly. I could write a long essay on how by solving software problems with more software we’re simply creating more software problems, however I will attempt to resist that urge and instead focus on how Docker and the Docker ecosystem can help make configuration management less sucky.

Ignoring volume mounts (which are an abomination for which I hold @gabrtv wholly responsible for), Docker has two main ways to configure your application: firstly by creating the dockerfile in which you explicitly declare your dependencies and insert any configuration files, and secondly at run time where you pass commands and environment variables to be used inside the container to start your application.

We’re going to ignore the dockerfile here and assume that you have at least a passing familiarity with them;  instead we’re going to focus on how to configure your application at run time.

A true Docker Native app would have a very small config file of which some or all settings could be overridden by environment variables or CLI options that can be set at run time to modify the appropriate configuration option (say, pointing it at a MySQL server at 10.2.2.55 ).

Very few applications are written in this way, and unless you’re starting from scratch or are willing to heavily re-factor your existing applications you’ll find that building and configuring your applications to run in “the docker way” is not always an easy or particularly pleasant thing to have to do. Thankfully there are ways to fake it.

To save writing out a bunch of CLI arguments the cleanest ( in my opinion ) way to pass values into docker containers is via environment variables like so:

 $ docker run -ti --rm -e hello=world busybox env
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 HOSTNAME=8cb5546f1ec4
 TERM=xterm
 hello=world
 HOME=/root

We can then write an application to read that environment variable and use it as a configuration directive like so:

 #!/bin/sh
 echo hello $hello

No prizes for guessing our output, we run a docker container from the image containing this script:

 $ docker run -ti --rm -e hello=world helloworld
 hello world

Now this was a pretty asinine demo, and apart from showing how passing environment variables into a docker container works doesn’t really do anything useful.  Let’s look at a slightly more realistic application.  Take a python app that reads a configuration file and when asked renders a web page using the contents of that configuration file:

note:  these examples are abbreviated sections of the example factorish app.

example.py

 import ConfigParser
 import os

 from flask import Flask
 app = Flask(__name__)

 @app.route('/')
 def hello():
 Config = ConfigParser.ConfigParser()
 Config.read("example.conf")
 return 'Luke, I am your {}'.format(
 Config.get("example", "text"))

 if __name__ == '__main__':
 app.run(host='0.0.0.0', port=80)

example.conf

 [example]
 text: father

Now when we run this application we get the following:

$ docker run -d -p 8080:8080 -e text=mother example
$ curl localhost:8080
 Luke, I am your father

Obviously the application reads from the config file and thus passing in the environment variable `text` is meaningless. We need a way to take that environment variable and embed it in the config file before running the actual `example.py` application.  Chances are the first thing that popped into your head would be to use `sed` or a similar linux tool to rewrite the config file like so:

run.sh

 #!/bin/bash
 sed -i "s/^text:.*$/text: ${text}" example.conf
 exec gunicorn -b 0.0.0.0:8080 app:app

Now we can run it again with run.sh set as the starting command and the config should be rewritten.

$ docker run -d -p 8080:8080 -e text=mother example ./run.sh
$ curl localhost:8080
Luke, I am your mother

This might be fine for a really simple application like this example, however for a complicated app with many configuration options it becomes quite cumbersome and offers plenty of opportunity for human error to slip in. Fortunately there are now several good tools written specifically for templating files in the docker ecosystem,  my favourite being confd by Kelsey Hightower which is a slick tool written in golang that can take key-pairs from various sources ( the simplest being environment variables ) and render templates with them.

Using confd we would write out a template file using the `getv` directive which simply retrieves the value of a key.  You’ll notice that the key itself is lowercase,  this is because confd also supports retrieving key-pairs from tools such as etcd and confd which use this format.  When set to use environment variables it is translated into reading the variable “SERVICES_EXAMPLE_TEXT”.

example.conf
 [example]
 text: {{ getenv "/services/example/text" }}

We would accompany this with a metadata file that tells confd how to handle that template:

example.conf.toml
 [template]
 src   = "example.conf"
 dest  = "/app/example/example.conf"
 owner = "app"
 group = "app"
 mode  = "0644"
 keys = [
 "/services/example",
 ]
 check_cmd = "/app/bin/check {{ .src }}"
 reload_cmd = "service restart example"

The last piece of this puzzle is a executable command in the form of a shell script that docker will run which will call confd to render the template and then start the python application:

boot.sh

 #!/bin/bash
 # read 'text' env var and export it as confd expected value
 # set it to 'father' if it does not exist
 export SERVICES_EXAMPLE_TEXT=${SERVICES_EXAMPLE_TEXT:-"father"}
 # run confd to render out the config
 confd -onetime -backend env
 # run app
 exec gunicorn -b 0.0.0.0:8080 app:app

Now let’s run it, first without any environment variables:

 $ docker run -d -p 8080:8080 --name example factorish/example
 $ curl localhost:8080
 Luke, I am your father
 $ docker exec example cat /app/example/example.conf
 [example]
 text: father

As you can see the server is responding using the default value of `father` that we set in the export command above.   Let’s run it again but set the variable in the docker run command:

 $ docker run -d -e SERVICES_EXAMPLE_TEXT=mother -p 8080:8080 --name example factorish/example
 $ curl localhost:8080
 Luke, I am your mother
 $ docker exec example cat /app/example/example.conf
 [example]
 text: mother

We see that because we set the environment variable it is be available to `confd` which renders it out into the config file.

Now if you go and look at the full example app you’ll see there a bunch of extra stuff going on.   Let’s see some more advanced usage of confd by starting a coreos cluster running etcd in Vagrant. etcd is a distributed key-value store that can be used to externalize application configuration and retrieve it as a service.

 $ git clone https://github.com/factorish/factorish.git
 $ cd factorish
 $ vagrant up

This will take a few minutes as the servers come online and build/run the application.   Once they’re up we can log into one and play with our application:

 $ vagrant ssh core-01
 core@core-01 ~ $  docker ps
 CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
 ee80f89d2565        registry            "docker-registry"   25 seconds ago      Up 25 seconds       0.0.0.0:5000->5000/tcp   factorish-registry
 c763ed34b182        factorish/example   "/app/bin/boot"     52 seconds ago      Up 51 seconds       0.0.0.0:8080->8080/tcp   factorish-example
 core@core-01 ~ $ docker logs factorish-example
 ==> ETCD_HOST set.  starting example etcd support.
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /app/example/example.conf out of sync
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/confd/run out of sync
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/confd/run has been updated
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/example/run out of sync
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/example/run has been updated
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/healthcheck/run out of sync
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: INFO Target config /etc/service/healthcheck/run has been updated
 echo ==> example: waiting for confd to write initial templates...
 2015-11-08T21:35:16Z c763ed34b182 confd[23]: ERROR exit status 1
 Starting example
 *** Booting runit daemon...
 *** Runit started as PID 51
 2015-11-08 21:35:22 [56] [INFO] Starting gunicorn 0.17.2
 2015-11-08 21:35:22 [56] [INFO] Listening at: http://0.0.0.0:8080 (56)
 2015-11-08 21:35:22 [56] [INFO] Using worker: sync
 2015-11-08 21:35:22 [67] [INFO] Booting worker with pid: 67
 core@core-01 ~ $ curl localhost:8080
 Luke, I am your father

You can see here that we’ve started the example app,  but notice at the top where it says “starting example etcd support”.  This is because we’ve actually started it with some environment variables that makes it aware that etcd exists. It uses these to configure `confd` to run in the background and watch an etcd key for templated config value.

We can see this and modify the config setting using etcd commands:

 core@core-01 ~ $ etcdctl get /services/example/text
 father
 core@core-01 ~ $ etcdctl set /services/example/text mother
 mother
 core@core-01 ~ $ curl localhost:8080
 Luke, I am your mother
 core@core-01 ~ $ exit
 $ vagrant ssh core-02
 core@core-02 ~ $ curl localhost:8080
 Luke, I am your mother

With confd aware of etcd it is able to notice values being changed and react accordingly,  in this case it rewrites the templated config file and then restarts the example application.   If you look at the template’s metadata from earlier you’ll see it is instructed to watch a certain key and rewrite the template if it changes.  It also has two directives `check_cmd` which is used to ensure the created template would be syntactically correct and `reload_cmd` which it runs any time the template is successfully written,  in this case to reload our application.

You’ll also notice that we were able to connect to the other coreos nodes each of which was also running the example application and because etcd was clustered across the three nodes all three applications registered the changed and updated themselves.

So now, not only do we have good clean templating in our container, we also even have the ability to change some of those config settings on the fly by connecting it to etcd.

From this very simple building block we are only a short hop away from being able to automatically configure complicated stacks that react to changes in the infrastructure instantaneously.

Pretty cool huh?

This article is part of our Docker and the Future of Configuration Management blog roundup running this November.  If you have an opinion or experience on the topic you can contribute as well

1 Comment

Filed under DevOps

One response to “Templating Config Files In Docker Containers

  1. Mike H

    An excellent article, wish I’d stumbled upon it earlier. Specifically I was looking for that common nightmare: secure distribution of config files that contain authentication info, including passwords. Now, with that extra Docker flavour! Especially when you use a CI tool such as Jenkins, you’re face with with having to ponder where the get those config files from: you don’t want them with your code base, you don’t want them in your templates (not even the placeholders which you’d then set through the environment). Cert based auth isn’t always an option. However, you’ve given me some ideas to research and play with!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s