Docker Logging Methods that Fit your Deployment Strategy

With the huge success of micro-services and container technologies, most of the enterprises are migrating their architecture to the container-based solution which is more reliable and flexible as compared to monolithic architecture.

Moving to containerized solution like Docker, logging each container and its insights is a new challenge. Analyzing logs provides insight into stability, performance, and reliability of each container and its application which is deployed over them.

Due to dynamic and flexible nature of Docker, there are multiple approaches for storing and analyzing logs. There are a variety of solutions, each has its own drawbacks and benefits. So I am going to discuss some of the solutions which I have faced while logging containerized architecture.

Key considerations before moving to Docker logging

1. Containers are transient
   As running Docker containers are just a process. Containers start, stop, destroy and rebuilt on regular basis. So, storing the application data inside the container is not at all a good practice, as data will be lost when the process is completed or you might end up losing the data when it becomes dead. So it is highly recommended to export your data to persistent storage.
2. Containers are generally known by its container ID
   As we learned in last point that containers are transient, so every time container start, it comes with unique container ID so maintaining logs of the same type of containers with the different container ID can be slightly difficult, so it’s better to provide some label or tag to a running container.

Methods of Logging in Docker

1) Log directly from your application:

In this method, the application handles its own log using some logging framework and send it to the remote destinations like Elasticsearch. For example, a Python application might use logging module for formatting and send logs to some remote destinations. This is the easiest way for developers to control logs but it creates extra load on the application process.

However, this type of logging method is limited to application, means under the head of the container, so if we are storing any logs in the container it will be lost if a container was terminated or was shut down in any manner. To overcome this data loss, we will have to configure a persistent storage or add to add some log forwarder in the container itself which send logs to the remote destination. Also, it will be very difficult to identify logs when we deploy same application in multiple containers to know which log belongs to which container.

2) Logging using data volumes

In this method, we have to map a log directory of a container to a directory on the host machine. We can also share single volume across multiple containers for centralized logging. However, it is difficult to move containers to different hosts without any loss of log data.

Follow this link, if you want to refer more about implementation on data volumes

3) Logging using Docker logging driver

In this method, first, we have to setup any log forwarder in the host or in any dedicated docker container. After setting up this log forwarder, we can tell Docker container to send the log to the logging driver which we are using as the forwarder such as fluentd, Logstash). We can also add tags in this method, which segregate containers from each other and maintain consistency if the container is terminated.

However, in this method, Docker container only sends its stdout and stderr logs to logging driver. So this method will be best if your application is running as the main process inside the docker container which is sending its log to Docker logs.

[js]

docker run -dit –log-driver=fluentd –log-opt fluentd-address=localhost:24224 –log-opt tag="docker.app1" app1 [/js]

As discussed, there is no single approach to logging with Docker. It depends upon the use-cases we have, like in orchestration tool Kubernetes, we can not implement logging driver as there is no support for that, so either we move to data volumes or installing forwarder in each container. So every use cases have its own solution. Still, there’s is no clearly defined best solution but that doesn’t mean that you should stop logging and managing the Docker logs.