Monthly Archives: March 2013

Chef your haproxy load balancer and add encryption

As of last September, HAProxy supports ssl so you no longer have to put stud/stunnel/nginx in front of HAProxy and it can also connect to SSL on backend servers so you can have encrypted traffic the whole way to the app server.  Most people decrypt on the load balancer and then pass it to their app servers unencrypted but I am not a big fan of that architecture.  This post shows you how to set up HAProxy with chef and we will be setting up ssl all the way to the app servers. Big thanks to @jtimberman for his post on encrypted data bags which helped me figure this out.

Setup your Chef encrypted data bag to store your ssl cert

The first step is to create a secret key for your data bag to use. This will be used to encrypt your data bag and later by chef nodes to decrypt the data bag so that they can read from the data bag. Do not store the encrypted_data_bag_secret in source control as-is. Instead, you can put this into a keepass database and then store that in source control if you want to.

openssl rand -base64 512 > ~/.chef/encrypted_data_bag_secret

Next you have to create the databag which we have aptly called secrets

knife data bag create secrets

Now we can store our wildcard cert in the secrets databag. This will open an editor and you can copy and paste your cert and key into it. This will go to the chef server and not on local disk. I set these id, cert and key.

knife data bag create secrets wildcard --secret-file ~/.chef/encrypted_data_bag_secret

The last step uploaded your wildcard cert to the chef server and encrypted it.   The next step allows us to save off the json export of our encrypted wildcard cert which we can check into source control and version.  Later if we get in a bind we can tell chef to import the databag using this json export.

mkdir data_bags/secrets
knife data bag show secrets wildcard -Fj > data_bags/secrets/wildcard.json

This next step is to just do a sanity check to make sure the databag export looks good.  It should look like this:

cat data_bags/secrets/wildcard.json { "id": "wildcard", "cert": "encrypted string here", "key": "encrypted string here" }

Create your own wrapper cookbook

Now in your chef cookbook you can access this wildcard cert. The next step requires you to write your own wrapper cookbook which doesn’t do very much other than set default attributes, pull the wildcard cert from the databag, write it to a file and then call the haproxy cookbook to do the install.  (My cookbook for this is in a private github repo because we do some custom steps and set some settings that don’t apply to everyone, but if you create a new cookbook and follow these steps, you should be set.)
Create a cookbook

knife cookbook create my-loadbalancer

Next, change the recipes/default.rb to look like this:

# Pull the certs from the encrypted databag
wildcard_cert = Chef::EncryptedDataBagItem.load("secrets","wildcard")
my_cert = wildcard['cert'].chomp # you may not need this chomp, but I did
my_key = wildcard['key'].chomp # you may not need this chomp, but I did
# feed the cert and key into the chef template
template "/etc/ssl/private/haproxy.pem" do
source "haproxy_pem.erb"
owner "root"
group "root"
mode 0400
variables(:wildcard_key => my_key,
:wildcard_crt => my_cert)
# Install haproxy and we are using a forked version of haproxy to install 1.5-17 from source and add SSL
include_recipe "haproxy::app_lb"

Add this template to your cookbook. The template we used for this file haproxy.pem is pretty basic. Here are the contents of templates/default/haproxy_pem.erb

<%= @wildcard_crt %>
<%= @wildcard_key %>

The line that calls include_recipe “haproxy::app_lb” is actually installing our forked version of the haproxy cookbook which adds the below line to the file templates/default/haproxy-app_lb.cfg.erb to setup ssl binding.

bind ssl crt /etc/ssl/private/haproxy.pem

You can check out our fork for the chef-haproxy cookbook to see how we install from source, what default attributes you can set and how we have our haproxy.cfg template using the ssl certs.

To recap, we uploaded our cert to an encrypted databag, added a recipe to pull that out and put it in a file (haproxy.pem) and we changed the haproxy cookbook to use that file to handle ssl certs. Hope this helps and if you run into any problems let me know.


Filed under DevOps, Security