How Blackfire leverages Docker

As you may know, Blackfire was represented at the SymfonyLive conference in Paris. During this event, several people came to us and asked how we use Docker at Blackfire.io.

One of our goals is to make profiling straightforward for anyone, and it means that we need to be able to easily test our product on a lot of different platforms. And Docker gives us the ability to spin up new containers in milliseconds.

Moreover, our website relies a lot on different tools, so containers can also help us reach an iso-production development environment.

But Docker is only available on Linux and a big part of the Blackfire’s team is using MacOS X. So how one using MacOS X can use the best of both worlds?

Boot2docker

As you know, Docker is developed in Go and it is therefore quite easy to port to the MacOS platform. Docker is composed of two pieces: the client and the daemon, the latter being Linux specific. But as many client-server communications, changing the transport is often simple and one can easily imagine a Docker client running on MacOS X speaking to a Docker server running on Linux (in a Virtual Machine for example).

Enter boot2docker! boot2docker works quite well, install the package and you have a running Docker setup on MacOS X in minutes.

Hurray, we are done! Well not too fast.

Boot2docker and developers

As a DevOps, you probably want to be able to create containers to run a few commands to try something, or pull a pre-built image to try a piece of software; and those use cases work great.

But as a developer, you need to be able to tweak the stack, share the DB, have nice DNS names, and much more. The rest of this post is the result of several weeks of trying to improve our Docker usage to ease the development of Blackfire.

Networking

The first annoying thing you may notice when using boot2docker is that the IP of the virtual machine changes very often. The first tweak is to tell boot2docker to always use the same IP for the Docker virtual machine; that will also allow for more advanced tweaks.

boot2docker uses a default DHCP range going from 192.168.59.103 to 192.168.59.254, so the VM can get any IP from this range. Let’s change that:

$ [ -f $HOME/.boot2docker/profile ] || boot2docker config &> $HOME/.boot2docker/profile
$ open $HOME/.boot2docker/profile

Make LowerIp and UpperIp match by using 192.168.59.103; this way boot2docker will still use DHCP but the IP address will always be set to 192.168.59.103.

$ boot2docker restart

File sharing

File sharing between the host and the Docker host is an out-of-the-box feature of boot2docker, but it is damn slow. As Mitchell Hashimoto from Hashicorp benched a year ago, this is not surprising as the VirtualBox shared folders (vboxsf) are a lot slower than NFS. Let’s switch to NFS!

First, shutdown boot2docker, remove the shared folder config, and start it again:

$ boot2docker stop
$ VBoxManage sharedfolder remove boot2docker-vm --name Users
$ boot2docker start --vbox-share=disable

NOTE: Always use the --vbox-share=disable flag when you start or restart boot2docker, otherwise the shared folder will be automatically added again.

Next step is to authorize boot2docker to mount your home by using NFS:

$ sudo touch /etc/exports
$ echo "# Boot2docker
\"$HOME\" -alldirs -mapall=$(whoami) -network 192.168.59.0 -mask 255.255.255.0" | sudo tee -a /etc/exports
$ sudo nfsd checkexports && sudo nfsd restart

NOTE: You may encounter errors when running the last command; edit /etc/exports (sudo vim /etc/exports) and remove the failing lines if that’s the case.

Next step is to mount the NFS share in boot2docker itself:

$ boot2docker ssh "sudo mkdir -p $HOME && sudo mount -t nfs -o noatime,soft,nolock,vers=3,udp,proto=udp,rsize=8192,wsize=8192,namlen=255,timeo=10,retrans=3,nfsvers=3 -v 192.168.59.3:$HOME $HOME"
$ boot2docker ssh mount

And you should see your user home directory mounted when running the last command.

NOTE: The NFS options result from weeks of tweaks, it should almost always work. If you tweak them, be prepared for some weird issues (composer install or npm install hanging forever, etc…)

So it works great, but if you restart boot2docker now, you will notice that you lost your home inside boot2docker. That is because the boot2docker filesystem is not persisted (it starts from an iso file), so if you change something, you have to do it over and over again. This is by design to ease maintenance and upgrades.

The Boot2docker team implemented a feature you can find at the very end of the FAQ and which is really useful here. On boot, boot2docker runs a little script located on a persisted disk/var/lib/boot2docker/bootlocal.sh if present, so you can now make the mount automatic:

NOTE: Please adapt the paths!

$ boot2docker ssh
docker@boot2docker:~$ echo '#!/bin/sh' | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod 755 /var/lib/boot2docker/bootlocal.sh
docker@boot2docker:~$ echo "sudo mkdir -p /Users/tucksaun && sudo mount -t nfs -o noatime,soft,nolock,vers=3,udp,proto=udp,rsize=8192,wsize=8192,namlen=255,timeo=10,retrans=3,nfsvers=3 -v 192.168.59.3:/Users/tucksaun /Users/tucksaun" | sudo tee -a /var/lib/boot2docker/bootlocal.sh
docker@boot2docker:~$ exit
$ boot2docker restart --vbox-share=disable
$ boot2docker ssh mount

And you should still see your user home directory mounted somewhere in the output.

Domain names/Multiple containers using the same port

The nice thing about containers is that they can be stopped and started in seconds. The problem? Their IP change at each reboot. Using a mixture of two projects, skydock and skydns, you can easily have your containers registered when they start and resolved toblackfireio_dev.dev.docker for instance.

First, configure boot2docker to automatically boot those two services on start and specify a specific IP for skydns:

$ boot2docker ssh
docker@boot2docker:~$ echo EXTRA_ARGS=\"-H unix:///var/run/docker.sock --bip=172.17.42.1/16 --dns=172.17.42.1\" | sudo tee /var/lib/boot2docker/profile
docker@boot2docker:~$ echo "sleep 5
docker start skydns || docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker
docker start skydock || docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker --name skydns" | sudo tee -a /var/lib/boot2docker/bootlocal.sh
docker@boot2docker:~$ exit
$ boot2docker restart --vbox-share=disable

Docker default subnet is 172.17.0.0/16, so to make it available on your MacOS host, add a custom route:

$ sudo route -n add 172.17.0.0/16 192.168.59.103

Then, setup resolverconf so that it can resolve the .docker domain (it should use 172.17.42.1, which is the skydns IP):

$ sudo mkdir /etc/resolver
$ sudo chmod 755 /etc/resolver
$ echo "nameserver 172.17.42.1" | sudo tee -a /etc/resolver/docker

Let’s try it by running in one terminal docker run --rm -it busybox sh. In another one, the following commands should succeed:

$ dig @172.17.42.1 busybox.dev.docker
$ ping busybox.dev.docker

Troubleshooting DNS: The command to flush DNS cache on MacOS changes often (for almost every release): see Apple Support to find the appropriate one for your version.

I can’t access my containers anymore when I reboot

This is because the custom route is not persisted across reboots. Create the/Library/LaunchDaemons/com.docker.route.plist file with this content (you need to adapt the interface name):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.docker.route</string>
  <key>ProgramArguments</key>
  <array>
    <string>bash</string>
    <string>-c</string>
    <!-- You need to adapt the vboxnet0 to the interface that suits your setup, use ifconfig to find it -->
    <string>/usr/sbin/scutil -w State:/Network/Interface/vboxnet0/IPv4 -t 0;sudo /sbin/route -n add -net 172.17.0.0 -netmask 255.255.0.0 -gateway 192.168.59.103</string>
  </array>
  <key>KeepAlive</key>
  <false/>
  <key>RunAtLoad</key>
  <true/>
  <key>LaunchOnlyOnce</key>
  <true/>
</dict>
</plist>

Then use sudo launchctl load /Library/LaunchDaemons/com.docker.route.plist to register it.

My containers can’t resolve any names anymore

On some networks, you can’t use another DNS than what the DHCP gives you.

The problem is that we fixed the upstream server IP in the docker configuration and this configuration can change often. One solution is to setup a DNSMasq on the MacOS host and make the Docker host use it as the upstream DNS server. This way, DNSMasq will forward the requests to the appropriate name server using the current network configuration.

Change 8.8.8.8 by 192.168.59.3 in /var/lib/boot2docker/bootlocal.sh in your boot2docker VM and delete the skydns container:

$ docker stop skydns && docker rm skydns

Then take care of dnsmasq (you will need homebrew installed) :

$ brew install dnsmasq
$ [ -f "/usr/local/etc/dnsmasq.conf" ] && echo "You should be able to take care of yourself ;-)" && read
$ echo "listen-address=127.0.0.1
listen-address=192.168.59.3
bind-interfaces
bind-dynamic
except-interface=en*
resolv-file=/etc/resolv.conf" | sudo tee /usr/local/etc/dnsmasq.conf
$ # used for offline mode, see https://github.com/37signals/pow/issues/104#issuecomment-7057102
$ echo "nameserver 127.0.0.1
domain ." | sudo tee /etc/resolver/root
$ if [ ! -f "/Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist" ]
then
    sudo cp /usr/local/Cellar/dnsmasq/*/homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons
    sudo launchctl load -w /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
    sudo launchctl stop homebrew.mxcl.dnsmasq
    sudo launchctl start homebrew.mxcl.dnsmasq
fi
$ boot2docker restart --vbox-share=disable

And your containers should now be able to resolve any domains (including .docker ones)

Bonus: Docker, docker-compose, and DNS names

Skydock generated names are not the most sexiest names. For us, it generates names likeblackfireio_dev.dev.docker but would it be better to get dev.blackfireio.docker ordb.blackfireio.docker?

To do so, we came up with a custom skydock plugins that you can easily customize to meet your needs. Create a skydns_plugins folder somewhere under your home directory, and create acustom_names.js file inside with the following content:

function createService(container) {
    if (isDockerComposeContainer(container)) {
        var name = removeSlash(container.Name).split('_');

        return {
            Environment: name.shift(),
            Instance: name.pop(),
            Service: name.join('_'),
            Host: container.NetworkSettings.IpAddress,
            Port: getDefaultPort(container),
            TTL: defaultTTL,
        };
    } else {
        return {
            Service: removeSlash(container.Name),
            Instance: removeSlash(container.Name),
            Host: container.NetworkSettings.IpAddress,
            Port: getDefaultPort(container),
            Environment: defaultEnvironment,
            TTL: defaultTTL,
        };
    }
}

function isDockerComposeContainer(container) {
    return removeSlash(container.Name).match(/^\w+_\w+_\d+$/);
}

// this is ported from skydock core:
// https://github.com/crosbymichael/skydock/blob/3a5125fc2a1fcffa42d577817d8b6e2d019dd55c/plugins/default.js
function getDefaultPort(container) {
    // if we have any exposed ports use those
    var port = 0;
    var ports = container.NetworkSettings.Ports;
    if (Object.keys(ports).length > 0) {
        for (var key in ports) {
            var value = ports[key];
            if (value !== null && value.length > 0) {
                for (var i = 0; i < value.length; i++) {
                    var hp = parseInt(value[i].HostPort);
                    if (port === 0 || hp < port) {
                        port = hp;
                    }
                }
            } else if (port === 0) {
                // just grab the key value
                var expose = parseInt(key.split("/")[0]);
                port = expose;
            }
        }
    }

    if (port === 0) {
        port = 80;
    }
    return port;
}

Edit /var/lib/boot2docker/bootlocal.sh to launch skydock with this file (adapt the path to the skydns_plugins folder):

docker start skydock || docker run -d -v /var/run/docker.sock:/docker.sock -v /Users/tucksaun/Work/skydns_plugins:/plugins --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -plugins /plugins/custom_names.js --name skydns

Then delete the skydock container and restart boot2docker:

$ docker rm skydock
$ boot2docker restart --vbox-share=disable

Bonus: Symfony cache (or any stack using the filesystem as a cache)

Even if NFS is faster than vboxsf, it is still slower than a local filesystem. So, for your cache or log directories, using a volume (persisted or not), improves the speed of execution quite a bit.

Conclusion

Thanks to this setup, you can now spin-up containers in milliseconds, or your whole project in seconds using docker-compose. At Blackfire.io, Docker already allowed us to help dozens of users, to improve our public Chef cookbook test suite and to get new developers (on new machines) up to speed in minutes.

Now that we have been able to test Docker setups thoroughly, we plan to spread the use of Docker to our CI and prod environments to make our deployments even safer.

The tweaks mentioned in this post are also used successfully by other developers at SensioLabs and it makes the use of Docker easier for their teams.

We hope than those recipes will allow you to use Docker in your day to day workflow!

Happy containerizing!