Running Curator in Docker container to remove old Elasticsearch indexes

We have been using ELK as a centralized logs management system. ELK stands for Elasticsearch, Logstash, and Kibana. Each of the three services is running inside its own docker container in one docker-network (say elk-net) on a single server. A separate block storage device is mapped from the host into Elasticsearch container (named es) as a data directory.

Since ELK creates one index each day into Elasticsearch, we wanted to keep only previous 15 days’ indexes on the system and remove the older ones after taking their backup on AWS S3 service. Elasticsearch allows us to take backup on s3 so we configured a cronjob which daily takes incremental backups on the indexes.

Curator is a good tool to remove the older indexes based on the dates. The problem was that the curator needs IP address of the Elasticsearch container and it is not a good practice to use container’s IP as this may change later if we play around with the container. So, we came up with an idea of running a curator inside a container in the same docker-network in which Elasticsearch container is running.

Scenario: Remove indexes of Elasticsearch older than fifteen days using a curator where Elasticsearch is running inside a docker container without using IP address of Elasticsearch container.
The following steps can be followed to implement this scenario:

1. Create a docker image which contains the curator binary. We have created a public docker images navjotsingh/curator using ubuntu:14.04 image.
2. Create a container using this image as below:

   [js]docker run –rm –name curator –net “elk-net” –entrypoint curator navjotsingh/curator –host es delete indices –older-than 15 –time-unit days –timestring ‘%Y.%m.%d'[/js]

   We have run a container name “curator” using docker image “navjotsingh/curator” in docker network “elk-net” with –rm option. This option removes the container only the container exits.
   We have used curator command as entry point and asked it to delete indices of format “%Y.%m.%d” which are older than 15 days .

3. Schedule this container to run every night. So every night this container will get created and will destroy itself after removing older indexes.

This is one typical use-case of docker where we are using it as a curator binary and is based on docker philosophy of “one process per container”.