Docker Compose

Well, now we've reached one of the most important parts of the book, the sensational and complete Docker Compose!

Docker Compose is simply a way for you to write all the details of your application environment in a single file. Previously we used the dockerfile only to create images, whether for my application, my database, or my webserver, but always in a unit way, since I have a dockerfile for each "type" of container: one for my app, another for my database, and so on.

With Docker Compose, we talk about the entire environment. For example, in Docker Compose we define which services we want to create and what characteristics each service has (number of containers under that service, volumes, network, secrets, etc.).

The standard that compose files follow is YML, super simple and easy to understand, but it's always good to pay attention to the syntax that the YML standard imposes on you. ;)

Well, let's stop talking and start the fun!

Previously, we needed to install Docker Compose to use it. However, today we have the "docker stack" subcommand, already available with the Docker installation. It is responsible for deploying our services through Docker Compose in a simple, fast, and very effective way.

Let's get started! The first thing we should do is create the compose file itself. We'll start with a simpler one and increase the complexity as we progress.

Remember: to continue with the next examples, your swarm cluster must be working perfectly. Therefore, if you don't have swarm active yet, run:

# docker swarm init

Let's create a directory called "Composes", just so we can better organize our files.

# mkdir /root/Composes
# mkdir /root/Composes/1
# cd /root/Composes/1
# vim docker-compose.yml

version: "3"

services:
    web:
        image: nginx
        deploy:
        replicas: 5
        resources:
        limits:
            cpus: "0.1"
            memory: 50M
        restart_policy:
            condition: on-failure
        ports:
        - "8080:80"
        networks:
        - webserver

networks:
    webserver:

Done! Now we have our first docker-compose. What we need now is to perform the deploy, but first let's get to know some options we used earlier:

version: "3" -- Version of compose we are using.
services: -- Beginning of my service definition.
web: -- Service name.
image: nginx -- Image we will use.
deploy: -- Beginning of the deploy strategy.
replicas: 5 -- Number of replicas.
resources: -- Beginning of the resource usage strategy.
limits: -- Limits.
cpus: "0.1" -- CPU limit.
memory: 50M -- Memory limit.
restart_policy: -- Restart policies.
condition: on-failure -- Will only "restart" the container in case of failure.
ports: -- Which ports we want to expose.
- "8080:80" -- Exposed and "bound" ports.
networks: -- Definition of the networks I will use in this service.
- webserver -- Name of this service's network.
networks: -- Declaring the networks we will use in this docker-compose.
webserver: -- Name of the network to be created, if it doesn't exist.

As simple as flying, right? :D

15.1. The docker stack command

Now we need to perform the deploy of this service through the compose file we created. For this, we will use the sensational "docker stack":

root@linuxtips-01:~/Composes/1# docker stack deploy -c docker-compose.yml primeiro
Creating network primeiro_webserver
Creating service primeiro_web

root@linuxtips-01:~/Composes/1#

Simple as that, and our service is already available for use. Now let's check if the service really came up and is responding as expected:

root@linuxtips-01:~/Composes/1# curl 0:8080
<!DOCTYPE html>
<html>
    <head>
        <title>Welcome to nginx!</title>
        <style>
        body {
            width: 35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
        </style>
    </head>
    <body>
        <h1>Welcome to nginx!</h1>
        <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>
        <p>For online documentation and support please refer to
        <a href="http://nginx.org/">nginx.org</a>.<br/>
        Commercial support is available at
        <a href="http://nginx.com/">nginx.com</a>.</p>
        <p><em>Thank you for using nginx.</em></p>
    </body>
</html>

root@linuxtips-01:~/Composes/1#

Sensational, our service is up because we received the Nginx welcome page!

Let's check if everything is ok with the service:

root@linuxtips-01:~/Composes/1# docker service ls
ID     NAME          MODE        REPLICAS    IMAGE           PORTS
mw95t  primeiro_web  replicated  5/5         nginx:latest    *:8080->80/tcp

root@linuxtips-01:~/Composes/1# docker service ps primeiro_web
ID            NAME             IMAGE         NODE           DESIRED STATE   CURRENT STATE           ERROR   PORTS
lrcqo8ifultq  primeiro_web.1   nginx:latest  LINUXtips-02   Running         Running 2 minutes ago
ty16mkcqdwyl  primeiro_web.2   nginx:latest  LINUXtips-03   Running         Running 2 minutes ago
dv670shw22o2  primeiro_web.3   nginx:latest  LINUXtips-01   Running         Running 2 minutes ago
sp0k1tnjftnr  primeiro_web.4   nginx:latest  LINUXtips-01   Running         Running 2 minutes ago
4fpl35llq1ih  primeiro_web.5   nginx:latest  LINUXtips-03   Running         Running 2 minutes ago

root@linuxtips-01:~/Composes/1#

To list all created stacks, just run:

root@linuxtips-01:~/Composes/1# docker stack ls
NAME        SERVICES
primeiro    1

root@linuxtips-01:~/Composes/1#

Notice: the output says we have only one stack created and this stack has one service, which is exactly our Nginx one.

To view the services that exist in a given stack, run:

root@linuxtips-01:~/Composes/1# docker stack services primeiro
ID            NAME          MODE         REPLICAS   IMAGE          PORTS
mx0p4vbrzfuj  primeiro_web  replicated   5/5        nginx:latest   *:8080->80/tcp

root@linuxtips-01:~/Composes/1#

We can check the details of our created stack using the following command:

root@linuxtips-01:~/Composes/1# docker stack ps primeiro
ID             NAME             IMAGE          NODE           DESIRED STATE    CURRENT STATE            ERROR   PORTS
x3u03509w9u3   primeiro_web.1   nginx:latest   LINUXtips-03   Running          Running 5 seconds ago
3hpu5lo6yvld   primeiro_web.2   nginx:latest   LINUXtips-02   Running          Running 5 seconds ago
m82wbwuwoza0   primeiro_web.3   nginx:latest   LINUXtips-03   Running          Running 5 seconds ago
y7vizedqvust   primeiro_web.4   nginx:latest   LINUXtips-02   Running          Running 5 seconds ago
wk0acjnyl6jm   primeiro_web.5   nginx:latest   LINUXtips-01   Running          Running 5 seconds ago

root@linuxtips-01:~/Composes/1#

Wonderful! Our service is UP and everything is at peace!

In a few minutes we brought up our Nginx service in our cluster using docker-compose and "docker stack", as simple as flying!

Now let's imagine I want to remove this service of mine. How do I do it? Simple:

root@linuxtips-01:~/Composes/1# docker stack rm primeiro
Removing service primeiro_web
Removing network primeiro_webserver

root@linuxtips-01:~/Composes/1#

To check if it really removed the service:

root@linuxtips-01:~/Composes/1# docker service ls
ID    NAME    MODE     REPLICAS     IMAGE     PORTS

root@linuxtips-01:~/Composes/1#

Done! Our service is removed!

Let's increase the complexity a bit in creating our docker-compose in this new example.

Let's create another directory, where we will create our new compose file:

root@linuxtips-01:~/Composes# mkdir 2
root@linuxtips-01:~/Composes# cd 2
root@linuxtips-01:~/Composes# vim docker-compose.yml

version: '3'
    services:
        db:
            image: mysql:5.7
            volumes:
                - db_data:/var/lib/mysql
            environment:
                MYSQL_ROOT_PASSWORD: somewordpress
                MYSQL_DATABASE: wordpress
                MYSQL_USER: wordpress
                MYSQL_PASSWORD: wordpress

        wordpress:
            depends_on:
            - db
            image: wordpress:latest
            ports:
            - "8000:80"
            environment:
                WORDPRESS_DB_HOST: db:3306
                WORDPRESS_DB_USER: wordpress
                WORDPRESS_DB_PASSWORD: wordpress

volumes:
    db_data:

Perfect!

In this example we are learning some more options we can use in docker-compose. They are:

volumes: -- Definition of volumes used by the service.
- db_data:/var/lib/mysql -- Volume and destination.
environment: -- Definition of environment variables used by the service.
MYSQL_ROOT_PASSWORD: somewordpress -- Variable and value.
MYSQL_DATABASE: wordpress -- Variable and value.
MYSQL_USER: wordpress -- Variable and value.
MYSQL_PASSWORD: wordpress -- Variable and value.
depends_on: -- Indicates that this service depends on another to start.
- db -- Name of the service needed for its execution.

Very simple, right?!?

Now let's perform the deploy of this example. As you can see, our stack is composed of two services, Wordpress and MySQL.

root@linuxtips-01:~/Composes/2# docker stack deploy -c docker-compose.yml segundo
Creating network segundo_default
Creating service segundo_db
Creating service segundo_wordpress

root@linuxtips-01:~/Composes/2#

As expected, it created the two services and the stack network.

To access your Wordpress, just access it in a browser:

http://YOUR_IP:8000

Your Wordpress is ready to use!

To check if everything went well with the services, remember the commands:

root@linuxtips-01:~/Composes/1# docker stack ls
root@linuxtips-01:~/Composes/1# docker stack services segundo
root@linuxtips-01:~/Composes/1# docker service ls
root@linuxtips-01:~/Composes/1# docker service ps segundo_db
root@linuxtips-01:~/Composes/1# docker service ps segundo_wordpress

To view the logs of a given service:

root@linuxtips-01:~/Composes/2# docker service logs segundo_wordpress
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | WordPress not found in /var/www/html - copying now...
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | Complete! WordPress has been successfully copied to /var/www/html
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.0.4.5. Set the 'ServerName' directive globally to suppress this message
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.0.4.5. Set the 'ServerName' directive globally to suppress this message
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | [Sun Jun 11 10:32:47.392836 2017] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.10 (Debian) PHP/5.6.30 configured -- resuming normal operations
segundo_wordpress.1.r6reuq8fsil0@LINUXtips-01 | [Sun Jun 11 10:32:47.392937 2017] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

root@linuxtips-01:~/Composes/2#

And if a modification to my stack is necessary and then a re-deploy, how do I do it? Is it possible?

Of course! After all, Docker is life!

root@linuxtips-01:~/Composes# mkdir 3
root@linuxtips-01:~/Composes# cd 3
root@linuxtips-01:~/Composes/3# vim docker-compose.yml

version: "3"
    services:
        web:
            image: nginx
            deploy:
                replicas: 5
            resources:
                limits:
                cpus: "0.1"
                memory: 50M
            restart_policy:
                condition: on-failure
            ports:
            - "8080:80"
            networks:
            - webserver

        visualizer:
            image: dockersamples/visualizer:stable
            ports:
            - "8888:8080"
            volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
            deploy:
                placement:
                    constraints: [node.role == manager]

    networks:
    - webserver

networks:
    webserver:

Notice that we only added a new service to our stack, the visualizer. The idea is to perform the update only on the stack to add the visualizer, without making the web service unavailable.

Before we perform the update of this stack, let's learn about the new options in the compose file of this example:

deploy:

placement: -- Used to define the location of our service.
constraints: [node.role == manager] -- Rule that requires the creation of this service only on manager nodes.

Now let's update our stack:

root@linuxtips-01:~/Composes/3# docker stack deploy -c docker-compose.yml primeiro
Creating service primeiro_visualizer
Updating service primeiro_web (id: mx0p4vbrzfujk087c3xe2sjvo)

root@linuxtips-01:~/Composes/3#

Notice that, to perform the update of the stack, we used the same command we used to perform the first deploy of the stack, the "docker stack deploy".

How about further increasing the complexity and number of services of a stack? Let's go?

For this example, we will use a Docker project itself (https://github.com/dockersamples/example-voting-app), where we will have several services. Let's create another directory to receive our project:

root@linuxtips-01:~/Composes# mkdir 4
root@linuxtips-01:~/Composes# cd 4
root@linuxtips-01:~/Composes/4# vim compose-file.yml

version: "3"

services:
    redis:
        image: redis:alpine
        ports:
        - "6379"
        networks:
        - frontend
        deploy:
            replicas: 2
            update_config:
                parallelism: 2
                delay: 10s
            restart_policy:
                condition: on-failure

    db:
        image: postgres:9.4
        volumes:
        - db-data:/var/lib/postgresql/data
        networks:
        - backend
        deploy:
            placement:
                constraints: [node.role == manager]

    vote:
        image: dockersamples/examplevotingapp_vote:before
        ports:
        - 5000:80
        networks:
        - frontend
        depends_on:
        - redis
        deploy:
            replicas: 2
            update_config:
                parallelism: 2
            restart_policy:
                condition: on-failure

    result:
        image: dockersamples/examplevotingapp_result:before
        ports:
        - 5001:80
        networks:
        - backend
        depends_on:
        - db
        deploy:
            replicas: 1
            update_config:
                parallelism: 2
                delay: 10s
            restart_policy:
                condition: on-failure

    worker:
        image: dockersamples/examplevotingapp_worker
        networks:
        - frontend
        - backend
        deploy:
            mode: replicated
            replicas: 1
            labels: [APP=VOTING]
            restart_policy:
                condition: on-failure
                delay: 10s
                max_attempts: 3
                window: 120s
            placement:
                constraints: [node.role == manager]

    visualizer:
        image: dockersamples/visualizer:stable
        ports:
        - "8080:8080"
        stop_grace_period: 1m30s
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
        deploy:
            placement:
                constraints: [node.role == manager]
networks:
    frontend:
    backend:

volumes:
    db-data:

Did it become more complex or not? I don't think so, because in Docker everything is quite simple!

We have some new options in this example, let's learn about them:

deploy:

mode: replicated -- What is the deployment type? We have two, global and replicated. In replicated you choose the number of replicas of your service, while in global you don't choose the number of replicas, it will bring up one replica per node in your cluster (one replica on each node of your cluster).

update_config:

parallelism: 2 -- How updates will occur (in this case, 2 at a time).
delay: 10s -- With a 10-second interval.

restart_policy:

condition: on-failure -- In case of failure, restart.
delay: 10s -- With a 10-second interval.
max_attempts: 3 -- With a maximum of three attempts.
window: 120s -- Time to define if the container restart was successful.

Now let's perform the deploy of our stack:

root@linuxtips-01:~/Composes/4# docker stack deploy -c docker-compose.yml quarto
Creating network quarto_default
Creating network quarto_frontend
Creating network quarto_backend
Creating service quarto_worker
Creating service quarto_visualizer
Creating service quarto_redis
Creating service quarto_db
Creating service quarto_vote
Creating service quarto_result

root@linuxtips-01:~/Composes/4#

Checking the services:

root@linuxtips-01:~/Composes/4# docker service ls
ID            NAME               MODE       REPLICAS   IMAGE                                             PORTS
3hi3sx2on3t5  quarto_worker      replicated 1/1        dockersamples/examplevotingapp_worker:latest
hbsp4fdcvgnz  quarto_visualizer  replicated 1/1        dockersamples/visualizer:stable                   :8080->8080/tcp
k6xuqbq7g55a  quarto_db          replicated 1/1        postgres:9.4
p2reijydxnsw  quarto_result      replicated 1/1        dockersamples/examplevotingapp_result:before      :5001->80/tcp
rtwnnkwftg9u  quarto_redis       replicated 2/2        redis:alpine                                      :0->6379/tcp
w2ritqiklpok  quarto_vote        replicated 2/2        dockersamples/examplevotingapp_vote:before        :5000->80/tcp

root@linuxtips-01:~/Composes/4#

Remember to always use the commands we already know to view stack, services, volumes, container, etc.

To access the running services, open a browser and go to the following addresses:

View the voting page: http://CLUSTER_IP:5000/
View the results page: http://CLUSTER_IP:5001/
View the page with the containers and their nodes: http://CLUSTER_IP:8080/

Let's move to another example. Now we will perform the deploy of a complete monitoring stack for our cluster and all other machines in our infrastructure. In this example we will use a YML file that will perform the deploy of several containers so we can have the following integrated tools:

Prometheus -- To store all the metrics of our environment.
cAdvisor -- To collect information from the containers.
Node Exporter -- To collect information from the cluster nodes and other environment machines.
Netdata -- To collect more than 5,000 metrics from our machines, in addition to providing a sensational dashboard.
Rocket.Chat -- So we can communicate with other teams and people and also to integrate it with the monitoring system, notifying when alerts happen. Rocket.Chat is an excellent alternative to Slack.
AlertManager -- Integrated with Prometheus and Rocket.Chat, it is responsible for managing our alerts.
Grafana -- Integrated with our monitoring solution, it is responsible for the dashboards that are produced through the metrics that are stored in Prometheus.

With this stack it is possible to monitor containers, VMs and physical machines. However, our focus now is only on what refers to the book and this chapter, that is, the information contained in the compose file that will define our stack.

For more details regarding Giropops-Monitoring, access the repository at: https://github.com/badtuxx/giropops-monitoring.

Before learning about our compose file, we need to clone the project:

# git clone https://github.com/badtuxx/giropops-monitoring.git

Access the "giropops-monitoring" directory:

# cd giropops-monitoring

Our focus here will be on three things: the "grafana.config" file, the "conf" directory and our beloved and idolized "docker-compose.yml".

The "grafana.config" file contains variables we want to pass to our Grafana. At this moment the only important information is the admin password, the user we will use to log in to the Grafana web interface.

The "conf" directory has the necessary files for the integration between the applications of our stack to work correctly.

Our compose file brings all the necessary information so we can perform the deploy of our stack.

Since our focus is the compose file, let's get to know it!

# cat docker-compose.yml
version: '3.3'
services:
    prometheus:
        image: linuxtips/prometheus_alpine
        volumes:
        - ./conf/prometheus/:/etc/prometheus/
        - prometheus_data:/var/lib/prometheus
        networks:
        - backend
        ports:
        - 9090:9090

    node-exporter:
        image: linuxtips/node-exporter_alpine
        hostname: {% raw %}'{{.Node.ID}}'{% endraw %}
        volumes:
        - /proc:/usr/proc
        - /sys:/usr/sys
        - /:/rootfs
        deploy:
            mode: global
        networks:
        - backend
        ports:
        - 9100:9100

    alertmanager:
        image: linuxtips/alertmanager_alpine
        volumes:
        - ./conf/alertmanager/:/etc/alertmanager/
        networks:
        - backend
        ports:
        - 9093:9093

    cadvisor:
        image: google/cadvisor
        hostname: {% raw %}'{{.Node.ID}}'{% endraw %}
        volumes:
        - /:/rootfs:ro
        - /var/run:/var/run:rw
        - /sys:/sys:ro
        - /var/lib/docker/:/var/lib/docker:ro
        - /var/run/docker.sock:/var/run/docker.sock:ro
        networks:
        - backend
        deploy:
            mode: global
        ports:
        - 8080:8080

    grafana:
        image: nopp/grafana_alpine
        depends_on:
        - prometheus
        volumes:
        - ./conf/grafana/grafana.db:/grafana/data/grafana.db
        env_file:
        - grafana.config
        networks:
        - backend
        - frontend
        ports:
        - 3000:3000

    # If you already has a RocketChat instance running, just comment the code of rocketchat, mongo and mongo-init-replica services bellow
    rocketchat:
        image: rocketchat/rocket.chat:latest
        volumes:
        - rocket_uploads:/app/uploads
        environment:
        - PORT=3080
        - ROOT_URL=http://YOUR_IP:3080
        - MONGO_URL=mongodb://giropops_mongo:27017/rocketchat
        - MONGO_OPLOG_URL=mongodb://giropops_mongo:27017/local
        depends_on:
        - giropops_mongo
        ports:
        - 3080:3080

    mongo:
        image: mongo:3.2
        volumes:
        - mongodb_data:/data/db
        #- ./data/dump:/dump
        command: mongod --smallfiles --oplogSize 128 --replSet rs0
        mongo-init-replica:
        image: mongo:3.2
        command: 'mongo giropops_mongo/rocketchat --eval "rs.initiate({_id: ''rs0'', members: [ { _id: 0, host: ''localhost:27017''} ]})"'
        depends_on:
        - giropops_mongo

networks:
    frontend:
    backend:

volumes:
    prometheus_data:
    grafana_data:
    rocket_uploads:
    mongodb_data:

Notice that we already know all the options that are in this example, nothing new. :D

What we need now is to perform the deploy of our stack:

# docker stack deploy -c docker-compose.yml giropops
Creating network giropops_backend
Creating network giropops_frontend
Creating network giropops_default
Creating service giropops_grafana
Creating service giropops_rocketchat
Creating service giropops_mongo
Creating service giropops_mongo-init-replica
Creating service giropops_prometheus
Creating service giropops_node-exporter
Creating service giropops_alertmanager
Creating service giropops_cadvisor

If you want to check if the services are running:

# docker service ls

To list the stacks:

# docker stack ls

To access the services we just performed the deploy, just access the following addresses:

Prometheus: http://YOUR_IP:9090
AlertManager: http://YOUR_IP:9093
Grafana: http://YOUR_IP:3000
Node_Exporter: http://YOUR_IP:9100
Rocket.Chat: http://YOUR_IP:3080
cAdivisor: http://YOUR_IP:8080

To remove the stack:

# docker stack rm giropops

Reminder: to learn more about giropops-monitoring access the repository on GitHub and watch the series of videos where Jeferson talks in detail about how he built this solution:

Repo:

https://github.com/badtuxx/giropops-monitoring
Videos:

https://www.youtube.com/playlist?list=PLf-O3X2-mxDls9uH8gyCQTnyXNMe10iml

And so ends our journey in the world of Docker. We hope you have learned and, more than that, have enjoyed sharing this time with us to talk about what we love most, technology!

15.2. Is it over already? :(

We hope you enjoyed traveling with us during your learning about containers and especially about the Docker ecosystem, which is sensational!

Don't stop learning more about Docker! Keep following the LinuxTips Channel at https://www.youtube.com/linuxtips and stay tuned to the Docker website, as there are always news and great documentation!

Join us on Discord so you can follow along and ask questions that may have arisen during your studies!

#VAIIII

15.1. The docker stack command​

15.2. Is it over already? :(​

15.1. The docker stack command

15.2. Is it over already? :(