HOW DOCKER V1.13 MESSED UP MY CUSTOM ROOT FOLDER ON UBUNTU 16.04 AND HOW {I} FIXED IT

A couple of weeks ago I’ve changed the docker root folder for images and filesystem from /var/lib/docker to a different folder on a different partition with lots of SSD space on it. Now everything looked good until two days ago or so when I wanted to play with a couple of new containers. My terminal started acting weird when I tried to use autocompletion on a command. Well, after searching I found out that docker was running again in its default root folder and eating up all the space that was left there.

How could this happen, I asked myself. In the same second I’ve remembered receiving an e-mail from the docker procject, promoting their brand new version 1.13. How would that correlate to my problem? Well, my linux machine does autoupdates and that’s basically how I’ve got the new version. Reading up on the net I’ve found that the /etc/default/docker file is not consumed by systems other than the ones using systemd. Hmm, strange… before the docker update everything was running smooth on my Ubuntu 16.04. Not really an argument to get into the bathtub and cry like a baby for hours. Been there, done that.

Here’s the standard config file for docker, which “normally” gets consumed by the docker daemon:

# Docker Upstart and SysVinit configuration file

#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
#   Please see the documentation for "systemd drop-ins":
#   https://docs.docker.com/engine/articles/systemd/
#

# Customize location of Docker binary (especially for development testing).
#DOCKERD="/usr/local/bin/dockerd"

# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export DOCKER_TMPDIR="/mnt/bigdrive/docker-tmp"

The general approach would now be to uncomment the DOCKER_OPTS attribute and fill in something like “-g /mnt/new_root_folder” and restart the docker service. At least that’s the way I did it the first time. Nah, this won’t work anymore. Instead I show you how they want you to do it from now on. As the net states one should use so called drop ins, we’re going to do that now. I’m not going into this topic (dropins) because if you’re reading this you want your docker containers back online and then go back to your daily porn routine.

We’re going to create a new drop in by creating a new file in /etc/systemd/system/docker.service.d.

└── my.conf

For demonstration purposes I’ve called mine my.conf. And the contents looks like this:

[Service]
ExecStart=/usr/bin/dockerd --graph=/<new_root_folder> --storage-driver=aufs

There’s the binary dockerd called with the -g option, followed by the new root folder. Also I’ve added the storage driver which happends to be aufs in my case. However, there are more. If you would like to read more about them, here you go:

https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

You might have noticed that we didn’t stop the docker service. If you did notice that then you’re a smartass – bravo! First, we’re going to stop the daemon, reload the new configuration for systemctl (my.conf) and then start the daemon again. BUT before we do that, we make sure we copy all the data from the old root path to the new one with the rsync command. You know, you might have realized that your docker daemon was running in the old root folder for about a month and your containers and volumes might have changed since then. Thank me later for saving your ass.

$ rsync -a /var/lib/docker/* /<new_root_folder>/

Now that we’ve got this out of our way we can restart the deamon.

$ systemctl stop docker
$ systemctl daemon-reload
$ systemctl start docker

Use docker info to check if the new root folder has been assigned.

$ docker info | grep "Docker Root Dir"

That’s it. Your previously running containers should have started by themself, otherwise start them. You should be able to delete the old root folder and not get any busy warnings there.

So, go back to your porn and enjoy that you’re alive.

Comments