How Blackfire leverages Docker

By Tugdual Saunier, on Apr 28, 2015

As you may know, Blackfire was represented at the SymfonyLive conference in Paris. During this event, several people came to us and asked how we use Docker at Blackfire.io.

One of our goals is to make profiling straightforward for anyone, and it means that we need to be able to easily test our product on a lot of different platforms. And Docker gives us the ability to spin up new containers in milliseconds.

Moreover, our website relies a lot on different tools, so containers can also help us reach an iso-production development environment.

But Docker is only available on Linux and a big part of the Blackfire’s team is using MacOS X. So how one using MacOS X can use the best of both worlds?

Boot2docker

As you know, Docker is developed in Go and it is therefore quite easy to port to the MacOS platform. Docker is composed of two pieces: the client and the daemon, the latter being Linux specific. But as many client-server communications, changing the transport is often simple and one can easily imagine a Docker client running on MacOS X speaking to a Docker server running on Linux (in a Virtual Machine for example).

Enter boot2docker! boot2docker works quite well, install the package and you have a running Docker setup on MacOS X in minutes.

Hurray, we are done! Well not too fast.

Boot2docker and developers

As a DevOps, you probably want to be able to create containers to run a few commands to try something, or pull a pre-built image to try a piece of software; and those use cases work great.

But as a developer, you need to be able to tweak the stack, share the DB, have nice DNS names, and much more. The rest of this post is the result of several weeks of trying to improve our Docker usage to ease the development of Blackfire.

Networking

The first annoying thing you may notice when using boot2docker is that the IP of the virtual machine changes very often. The first tweak is to tell boot2docker to always use the same IP for the Docker virtual machine; that will also allow for more advanced tweaks.

boot2docker uses a default DHCP range going from 192.168.59.103 to 192.168.59.254, so the VM can get any IP from this range. Let’s change that:

Make LowerIp and UpperIp match by using 192.168.59.103; this way boot2docker will still use DHCP but the IP address will always be set to 192.168.59.103.

File sharing

File sharing between the host and the Docker host is an out-of-the-box feature of boot2docker, but it is damn slow. As Mitchell Hashimoto from Hashicorp benched a year ago, this is not surprising as the VirtualBox shared folders (vboxsf) are a lot slower than NFS. Let’s switch to NFS!

First, shutdown boot2docker, remove the shared folder config, and start it again:

NOTE: Always use the --vbox-share=disable flag when you start or restart boot2docker, otherwise the shared folder will be automatically added again.

Next step is to authorize boot2docker to mount your home by using NFS:

NOTE: You may encounter errors when running the last command; edit /etc/exports (sudo vim /etc/exports) and remove the failing lines if that’s the case.

Next step is to mount the NFS share in boot2docker itself:

And you should see your user home directory mounted when running the last command.

NOTE: The NFS options result from weeks of tweaks, it should almost always work. If you tweak them, be prepared for some weird issues (composer install or npm install hanging forever, etc…)

So it works great, but if you restart boot2docker now, you will notice that you lost your home inside boot2docker. That is because the boot2docker filesystem is not persisted (it starts from an iso file), so if you change something, you have to do it over and over again. This is by design to ease maintenance and upgrades.

The Boot2docker team implemented a feature you can find at the very end of the FAQ and which is really useful here. On boot, boot2docker runs a little script located on a persisted disk/var/lib/boot2docker/bootlocal.sh if present, so you can now make the mount automatic:

NOTE: Please adapt the paths!

And you should still see your user home directory mounted somewhere in the output.

Domain names/Multiple containers using the same port

The nice thing about containers is that they can be stopped and started in seconds. The problem? Their IP change at each reboot. Using a mixture of two projects, skydock and skydns, you can easily have your containers registered when they start and resolved toblackfireio_dev.dev.docker for instance.

First, configure boot2docker to automatically boot those two services on start and specify a specific IP for skydns:

Docker default subnet is 172.17.0.0/16, so to make it available on your MacOS host, add a custom route:

Then, setup resolverconf so that it can resolve the .docker domain (it should use 172.17.42.1, which is the skydns IP):

Let’s try it by running in one terminal docker run --rm -it busybox sh. In another one, the following commands should succeed:

Troubleshooting DNS: The command to flush DNS cache on MacOS changes often (for almost every release): see Apple Support to find the appropriate one for your version.

I can’t access my containers anymore when I reboot

This is because the custom route is not persisted across reboots. Create the/Library/LaunchDaemons/com.docker.route.plist file with this content (you need to adapt the interface name):

Then use sudo launchctl load /Library/LaunchDaemons/com.docker.route.plist to register it.

My containers can’t resolve any names anymore

On some networks, you can’t use another DNS than what the DHCP gives you.

The problem is that we fixed the upstream server IP in the docker configuration and this configuration can change often. One solution is to setup a DNSMasq on the MacOS host and make the Docker host use it as the upstream DNS server. This way, DNSMasq will forward the requests to the appropriate name server using the current network configuration.

Change 8.8.8.8 by 192.168.59.3 in /var/lib/boot2docker/bootlocal.sh in your boot2docker VM and delete the skydns container:

Then take care of dnsmasq (you will need homebrew installed) :

And your containers should now be able to resolve any domains (including .docker ones)

Bonus: Docker, docker-compose, and DNS names

Skydock generated names are not the most sexiest names. For us, it generates names likeblackfireio_dev.dev.docker but would it be better to get dev.blackfireio.docker ordb.blackfireio.docker?

To do so, we came up with a custom skydock plugins that you can easily customize to meet your needs. Create a skydns_plugins folder somewhere under your home directory, and create acustom_names.js file inside with the following content:

Edit /var/lib/boot2docker/bootlocal.sh to launch skydock with this file (adapt the path to the skydns_plugins folder):

Then delete the skydock container and restart boot2docker:

Bonus: Symfony cache (or any stack using the filesystem as a cache)

Even if NFS is faster than vboxsf, it is still slower than a local filesystem. So, for your cache or log directories, using a volume (persisted or not), improves the speed of execution quite a bit.

Conclusion

Thanks to this setup, you can now spin-up containers in milliseconds, or your whole project in seconds using docker-compose. At Blackfire.io, Docker already allowed us to help dozens of users, to improve our public Chef cookbook test suite and to get new developers (on new machines) up to speed in minutes.

Now that we have been able to test Docker setups thoroughly, we plan to spread the use of Docker to our CI and prod environments to make our deployments even safer.

The tweaks mentioned in this post are also used successfully by other developers at SensioLabs and it makes the use of Docker easier for their teams.

We hope than those recipes will allow you to use Docker in your day to day workflow!

Happy containerizing!

Tugdual Saunier

Tugdual is a Product Developer at Blackfire.io. He started PHP programming when he was a teenager, and hasn’t stopped since. He discovered Symfony right after his studies and soon joined SensioLabs. A couple of years ago, with Fabien Potencier, he was exploring some options to optimize Symfony and Twig. And they got so frustrated by how hard it was to evaluate performance impact of some changes in code that they decided to explore the options available to improve the situation.