Docker Volumes

https://docs.docker.com/storage/

When we talk about volumes, we are talking about storage, which is data storage.

• Data created in a container is stored in the read-write layer, which is considered a writable layer, that is, non-persistent. When the container dies, the data is lost.
• The writable layer is coupled to the host machine and cannot be easily moved to another location.
• Using volumes writes directly to the file system of the host where the container is running.
• Writing data to the top layer requires a file system abstraction that reduces performance compared to using volumes.

Supported Storage Drivers

https://docs.docker.com/storage/storagedriver/select-storage-driver/

Driver	Description
overlay2	Preferred and requires no extra configuration. It's the evolution of aufs.
fuse-overlayfs	Only used when overlay2 is not supported
btrfs and zfs	Uses the Copy on Write (CoW) system that uses the same file when reading, but when it needs to write, it creates a new file. Allows more advanced options, but also requires more configuration and maintenance. Enables combining multiple physical devices to form a single file system
vfs	The performance of this driver is poor and is generally not recommended for production use. Writes directly to disk and does not support CoW
aufs	Only used in very old Debian-based kernel versions that don't support overlay2.
devicemapper	Very poor performance and only used in older RedHat-based kernel versions
overlay	Obsolete and legacy, don't waste time using it

To find out which file system is currently being used

docker system info
# or
vagrant@worker1:~$ docker system info | grep Storage
WARNING: No swap limit support
 Storage Driver: overlay2
vagrant@worker1:~$

Modifying the Storage Driver requires a change in the daemon at /etc/docker/daemon.json

Let's use the JSON below to make a test change

{
    "storage-driver": "devicemapper"
}

vagrant@worker1:~$ sudo tee  /etc/docker/daemon.json << EOF
{
    "storage-driver": "devicemapper"
}
EOF
vagrant@worker1:~$ sudo systemctl restart docker
vagrant@worker1:~$ sudo docker system info | grep Storage
WARNING: No swap limit support
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
# notice how our storage driver has changed
 Storage Driver: devicemapper
vagrant@worker1:~$

Go back to overlay2 to continue the rest of our lab.

vagrant@worker1:~$ sudo tee  /etc/docker/daemon.json << EOF
{
    "storage-driver": "overlay2"
}
EOF
{
    "storage-driver": "overlay2"
}
vagrant@worker1:~$ sudo systemctl restart docker
vagrant@worker1:~$ sudo docker system info | grep Storage
WARNING: No swap limit support
 Storage Driver: overlay2
vagrant@worker1:~$

Volumes

A volume is basically a directory that is designated to work with one or more containers. It's a way to share data so you can have multiple processes reading and writing from the same place. If the container dies, the data remains persistent in the volume, so when the container runs again, the data has not been lost. When a container dies, it does not automatically delete the volume. It's a separate and managed resource.

3 basic types:

• bind mount (volume that resides within the same host running Docker) is when the container is using files directly from the host filesystem. A directory within the container reflects a directory on the host.

• volume (named or anonymous type) is a file system that Docker manages to make available inside the container. Just like a virtual machine has a file that represents its hard drive? It's the same thing.

  • It's better than using bind mount
  • Better for making backups
  • Can be managed by the Docker CLI
  • Shared by multiple containers in a safer way
  • External storage can be enabled in the cloud, for example
  • Volumes are located inside /var/lib/docker

• tmpfs mount is a temporary volume type, it resides in the machine's RAM memory. If the host crashes, for example, you lose the content. It's useful for use in high-performance environments. Very few use cases.

Commands

The docker volume command

vagrant@worker1:~$ docker volume --help
Commands:
  create      Create a volume
  inspect     Display detailed information on one or more volumes
  ls          List volumes
  prune       Remove all unused local volumes
  rm          Remove one or more volumes

In the container manipulation command docker container, if it has the -v or --volume parameter, it's passing the volume to the container. The --mount option is a more explicit and verbose option.

• -v : works for bind mount

• --mount : works for bind mount and for volumes, including cloud services, etc.

Creating a container with bind mount and checking

                                           #host:container
docker container run -dit --name servidor -v /srv:/srv debian

# Running in the container
vagrant@worker1:~$ docker container exec servidor df -Th
Filesystem     Type     Size  Used Avail Use% Mounted on
# see the overlay here inside the container
overlay        overlay   39G  2.3G   37G   6% /
tmpfs          tmpfs     64M     0   64M   0% /dev
tmpfs          tmpfs    489M     0  489M   0% /sys/fs/cgroup
shm            tmpfs     64M     0   64M   0% /dev/shm
/dev/sda1      ext4      39G  2.3G   37G   6% /srv
tmpfs          tmpfs    489M     0  489M   0% /proc/acpi
tmpfs          tmpfs    489M     0  489M   0% /proc/scsi
tmpfs          tmpfs    489M     0  489M   0% /sys/firmware
# Now running on the host
vagrant@worker1:~$ df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  472M     0  472M   0% /dev
tmpfs          tmpfs      98M  1.1M   97M   2% /run
# Note that sda1 has the same size as the container's overlay, meaning they're the same
/dev/sda1      ext4       39G  2.3G   37G   6% /
tmpfs          tmpfs     489M     0  489M   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     489M     0  489M   0% /sys/fs/cgroup
/dev/loop0     squashfs   62M   62M     0 100% /snap/core20/1518
/dev/loop1     squashfs   47M   47M     0 100% /snap/snapd/16010
/dev/loop2     squashfs   68M   68M     0 100% /snap/lxd/22753
vagrant        vboxsf    440G   48G  392G  11% /vagrant
tmpfs          tmpfs      98M     0   98M   0% /run/user/1000

It simply mounted the /srv path on the host inside /srv in the container. If we create any file inside on either side, it will appear.

# See that srv appeared in /
vagrant@worker1:~$ ls /
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv  sys  tmp  usr  vagrant  var
vagrant@worker1:~$ cd /srv/
vagrant@worker1:~$ cd /srv/
vagrant@worker1:/srv$ sudo touch arquivo1.txt
vagrant@worker1:/srv$ docker container exec servidor ls -lR /srv
/srv:
total 0
-rw-r--r-- 1 root root 0 Jun 23 17:11 arquivo1.txt
vagrant@worker1:/srv$ docker container exec servidor touch /srv/arquivo2.txt
# after being created inside the container, it shows on the host
vagrant@worker1:/srv$ ls
arquivo1.txt  arquivo2.txt

##
vagrant@worker1:/srv$ docker container rm -f servidor
servidor
vagrant@worker1:/srv$ docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
vagrant@worker1:/srv$

Creating a container with anonymous volume

When I pass only the volume without specifying source and destination, it creates a volume inside /var/lib/docker/volumes with a 64-character hash for the volume.

vagrant@worker1:~$ docker container run -dit --name servidor -v /srv debian
vagrant@worker1:~$ docker volume ls
DRIVER    VOLUME NAME
local     7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
vagrant@worker1:~$ docker container inspect servidor | grep volume
                "Type": "volume",
                "Source": "/var/lib/docker/volumes/7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec/_data",
vagrant@worker1:~$

Looking at the inspect, notice that the volume hash is the same as the volume we have.

Note: Docker volumes are located in /var/lib/docker/volumes

A volume always has _data which is where the data actually is, but at the same level there can be other volume metadata files, such as permissions.

It's also possible to inspect the volume

vagrant@worker1:~$ docker volume inspect 7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
[
    {
        "CreatedAt": "2022-06-23T17:48:15Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec/_data",
        "Name": "7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec",
        "Options": null,
        "Scope": "local"
    }
]

vagrant@worker1:~$ docker container exec servidor touch /srv/arquivoVolume.txt
vagrant@worker1:~$ docker container inspect servidor | grep volume
                "Type": "volume",
                "Source": "/var/lib/docker/volumes/7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec/_data",
vagrant@worker1:~$ sudo ls /var/lib/docker/volumes/7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec/_data
arquivoVolume.txt

If a volume is created by passing the name without the / in front, it will understand that it's just a name, not the source

vagrant@worker1:~$ docker container run -dit --name servidor -v volumedebian:/srv debian
76540b187448053054b227f20ba5a1bdab43185857c35dabf2c248f2aa42968e
vagrant@worker1:~$ docker volume list
DRIVER    VOLUME NAME
local     7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
local     volumedebian

Creating a container using --mount

As mentioned earlier, --mount requires more parameters.

vagrant@worker1:~$ docker container run -dit --name servidor --mount source=volume2,target=/meuvolume debian
be42b33fb8b6f928ceac632ec791c3da632b1a24ab36b239bab5e1e2c6eed597
vagrant@worker1:~$ docker volume list
DRIVER    VOLUME NAME
local     7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
local     volume2
local     volumedebian
vagrant@worker1:~$

vagrant@worker1:~$ docker container inspect servidor --format '{{json .Mounts}}' | jq
[
  {
    "Type": "volume",
    "Name": "volume2",
    "Source": "/var/lib/docker/volumes/volume2/_data",
    "Destination": "/meuvolume",
    "Driver": "local",
    # this mode z (lowercase) means it shares between multiple containers; if it were Z (uppercase) it means it doesn't share
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  }
]

Let's remove the volumes

vagrant@worker1:~$ docker volume ls
DRIVER    VOLUME NAME
local     7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
local     volume2
local     volumedebian
vagrant@worker1:~$ docker volume rm volume2 volumedebian 7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
volumedebian
7f17a7c16d36d501b49da3cf2d7a7fda9c384ac145387377fa4cf099556ae5ec
Error response from daemon: remove volume2: volume is in use - [be42b33fb8b6f928ceac632ec791c3da632b1a24ab36b239bab5e1e2c6eed597]
# note that it's not possible to remove a volume in use by a container
vagrant@worker1:~$ docker volume ls
DRIVER    VOLUME NAME
local     volume2
vagrant@worker1:~$ docker container rm -f servidor
servidor
# The prune command removes all volumes not in use by any container
vagrant@worker1:~$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
volume2

Total reclaimed space: 0B
vagrant@worker1:~$ docker volume ls
DRIVER    VOLUME NAME
vagrant@worker1:~$

tmpfs mounts

--tmpfs

As mentioned earlier, it's used in RAM memory. Only useful for increasing input and output for necessary applications. It's also useful for sensitive data that you don't want to write to disk.

• It's not possible to pass parameters like size limit, etc.
• It has no source, since the source is memory, only the destination
• This parameter doesn't work in clusters; in that case, you would need to use --mount type=tmpfs
• tmpfs in swap is not possible

vagrant@worker1:~$ docker container run -dit --name servidor --tmpfs /volumetpmfs debian
30ce1b88b693104ec78c0360ed87dad23da7da74e410d8784ae88db36244cf91
vagrant@worker1:~$ docker container run -dit --name servidor2 --mount type=tmpfs,destination=/volumetmpfs debian
6d85918d60734564df7a3ed8e5937c3f446a5a9d0158fa398a43a53643aa5998
vagrant@worker1:~$
# note that there's no mount when we start with --tmpfs
vagrant@worker1:~$ docker container inspect servidor -f '{{json .Mounts}}' | jq
[]
vagrant@worker1:~$ docker container inspect servidor -f '{{json .HostConfig.Tmpfs}}' | jq
{
  "/volumetpmfs": ""
}
vagrant@worker1:~$ docker container inspect servidor2 -f '{{json .Mounts}}' | jq
[
  {
    "Type": "tmpfs",
    "Source": "",
    "Destination": "/volumetmpfs",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]
# Not in HostConfig.Tmpfs when we start with --mount
vagrant@worker1:~$ docker container inspect servidor2 -f '{{json .HostConfig.Tmpfs}}' | jq
null

vagrant@worker1:~$ docker container exec servidor df -Th
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay   39G  2.4G   37G   6% /
tmpfs          tmpfs     64M     0   64M   0% /dev
# created a tmpfs here with the size of 500M which is its default
tmpfs          tmpfs    489M     0  489M   0% /sys/fs/cgroup
shm            tmpfs     64M     0   64M   0% /dev/shm
tmpfs          tmpfs    489M     0  489M   0% /volumetpmfs
/dev/sda1      ext4      39G  2.4G   37G   6% /etc/hosts
tmpfs          tmpfs    489M     0  489M   0% /proc/acpi
tmpfs          tmpfs    489M     0  489M   0% /proc/scsi
tmpfs          tmpfs    489M     0  489M   0% /sys/firmware

To change the default size or permission, only through --mount

# removing the containers and creating again with a fixed size
vagrant@worker1:~$ docker container rm -f $(docker container ls -aq)
6d85918d6073
30ce1b88b693

vagrant@worker1:~$ docker container run -dit --name servidor --mount type=tmpfs,destination=/volumetmpfs,tmpfs-size=100M debian
vagrant@worker1:~$ docker container inspect servidor -f '{{json .HostConfig.Mounts}}' | jq
[
  {
    "Type": "tmpfs",
    "Target": "/volumetmpfs",
    "TmpfsOptions": {
      "SizeBytes": 104857600
    }
  }
]
vagrant@worker1:~$ docker container exec servidor df -Th
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay   39G  2.4G   37G   6% /
tmpfs          tmpfs     64M     0   64M   0% /dev
tmpfs          tmpfs    489M     0  489M   0% /sys/fs/cgroup
shm            tmpfs     64M     0   64M   0% /dev/shm
# Showing the 100M here
tmpfs          tmpfs    100M     0  100M   0% /volumetmpfs
/dev/sda1      ext4      39G  2.4G   37G   6% /etc/hosts
tmpfs          tmpfs    489M     0  489M   0% /proc/acpi
tmpfs          tmpfs    489M     0  489M   0% /proc/scsi
tmpfs          tmpfs    489M     0  489M   0% /sys/firmware

I tried downloading an Ubuntu image inside the 100M volume to see what would happen

vagrant@worker1:~$ docker attach servidor
root@467701c637fc:/ apt-get update && apt-get install wget
....
root@467701c637fc:/ cd /volumetmpfs
root@467701c637fc:/volumetmpfs# wget https://releases.ubuntu.com/22.04/ubuntu-22.04-desktop-amd64.iso
.........
......... # Removed section
ubuntu-22.04-desktop-amd64.iso          2%[=>                                                                      ] 100.00M  21.7MB/s    in 5.4s

# see that the space ran out
Cannot write to 'ubuntu-22.04-desktop-amd64.iso' (No space left on device).
root@467701c637fc:/volumetmpfs#

Another scenario I tested was creating a container with tmpfs larger than the host's memory and it allowed it, but when I downloaded the Ubuntu image using wget, following the same procedure above... when it reached close to 620MB of download, the machine became extremely slow and unresponsive. So use it wisely. It didn't crash the machine, but improper use of tmpfs can affect other containers.

vagrant@worker1:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          976Mi       207Mi       634Mi       0.0Ki       134Mi       627Mi
Swap:            0B          0B          0B
vagrant@worker1:~$ docker container run -dit --name servidor --mount type=tmpfs,destination=/volumetmpfs,tmpfs-size=2000M debian
7355006196d27418913d904dd7e5ea8c9bf3187c44e171b22bcb6ea13f36d463
vagrant@worker1:~$

Backup and restore

Let's start a container and put a file inside it

vagrant@worker1:~$ docker container run -dit --name servidor -v /webdata  debian
e8d3713e08bb7c35346f5b230fbd322e04e82497f58cc1513861563e067d4c3a
vagrant@worker1:~$ mkdir -p ~/dockerfiles/images/
vagrant@worker1:~$ echo "{FROM debian} "~/dockerfiles/images/Dockerfile
{FROM debian} ~/dockerfiles/images/Dockerfile
                                        #source       #destination machine:path
vagrant@worker1:~$ docker container cp ./dockerfiles/ servidor:/webdata
vagrant@worker1:~$ docker container exec servidor ls -lha /webdata
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 20:50 .
drwxr-xr-x 1 root root 4.0K Jun 23 20:50 ..
drwxrwxr-x 3 1000 1000 4.0K Jun 23 20:47 dockerfiles
vagrant@worker1:~$ docker volume ls
DRIVER    VOLUME NAME
local     4806a0c128952f1fc2cc3c1101c8c465a5233fc0203cd87fa294fbad19fe13da
vagrant@worker1:~$ docker container inspect servidor | grep volume
                "Type": "volume",
                "Source": "/var/lib/docker/volumes/4806a0c128952f1fc2cc3c1101c8c465a5233fc0203cd87fa294fbad19fe13da/_data",

Now let's start another container pointing to the same volume

vagrant@worker1:~$ docker container run -dit --name servidor2 -v 4806a0c128952f1fc2cc3c1101c8c465a5233fc0203cd87fa294fbad19fe13da:/webdata  debian
e8ac0f51f6eaa42401da39ae6e2fec182970c5b04b8fa5456cdc216c9657e384
vagrant@worker1:~$ docker container exec servidor2 ls -lha /webdata
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 20:50 .
drwxr-xr-x 1 root root 4.0K Jun 23 20:56 ..
drwxrwxr-x 3 1000 1000 4.0K Jun 23 20:47 dockerfiles

However, this is somewhat impractical and there's a flag called --volumes-from that greatly improves how to do this.

vagrant@worker1:~$ docker container run -dit --name servidor3 --volumes-from servidor  debian
c60e0d3f1f1ddc935414d98b424ff5964cfbaf540e257b3797b78ed7f44013f1
# Checking if the volume is there the way it was on server 1
vagrant@worker1:~$ docker container exec servidor3 ls -lha /webdata
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 20:50 .
drwxr-xr-x 1 root root 4.0K Jun 23 20:57 ..
drwxrwxr-x 3 1000 1000 4.0K Jun 23 20:47 dockerfiles

Taking advantage of what we learned here, let's now start an Alpine container to make a backup of the volume

vagrant@worker1:~$ docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED          STATUS          PORTS     NAMES
c60e0d3f1f1d   debian    "bash"    31 minutes ago   Up 31 minutes             servidor3
e8ac0f51f6ea   debian    "bash"    33 minutes ago   Up 33 minutes             servidor2
e8d3713e08bb   debian    "bash"    43 minutes ago   Up 43 minutes             servidor
vagrant@worker1:~$ mkdir backup
vagrant@worker1:~$ docker container run -dit --rm --volumes-from servidor -v ./backup:/backup alpine tar cvf /backup/backup.tar /webdata
c6ffeaa65384af74738c81cea51504795c4361fad0fca52055856b21186ae08b
vagrant@worker1:~$ ls backup/
backup.tar
vagrant@worker1:~$ tar tvf backup/backup.tar
drwxr-xr-x root/root         0 2022-06-23 20:50 webdata/
drwxrwxr-x 1000/1000         0 2022-06-23 20:47 webdata/dockerfiles/
drwxrwxr-x 1000/1000         0 2022-06-23 20:47 webdata/dockerfiles/images/

Now let's create another container with the backup.tar file we created, but first let's delete all containers and volumes.

# as we saw in the last command above, there is the webdata folder that contains the rest, strip 1 removes that level
docker container run -dit --name servidor -v ~/backup:/webdata debian bash -c "tar -xf /webdata/backup.tar --strip 1 --directory /webdata && rm /webdata/backup.tar"

In this case I deleted the backup, but if you didn't want to, just start a machine with an empty volume and another one with volumes from and backup from one directory to the other.