Elasticsearch, Logstash and Kibana (ELK) v7.5 … running on Docker Swarm

Docker is a great concept. The idea is to create containers that we can simply download and run on servers without going through the process of installation again and again… But (you were expecting it!) it adds its own complexity. We need to learn how Docker works – which seems to be endless madness. There is quite a bit of commands to learn and configurations to do. And lastly, we have to hope that the software has somme sort of support for Docker.

Creating Docker images

I created customized images of ELK 7.5 to integrate the configuration and also by necessity for Logstash to update the RabbitMQ output plugin. For each component, we need a Dockerfile with its configuration.

Elasticsearch

FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.0

# Change ownership of the data files
USER root
RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
USER elasticsearch

# copy configuration files from local to container
COPY config/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml

Kibana

FROM docker.elastic.co/kibana/kibana:7.5.0

# Add your kibana plugins setup here
# Example: RUN kibana-plugin install 
COPY config /usr/share/kibana/config

Logstash

FROM docker.elastic.co/logstash/logstash:7.5.0

# Copy configuration files
COPY config /usr/share/logstash/config
COPY pipelines /usr/share/logstash/pipelines

# Change ownership of the data files
USER root
RUN chown -R logstash:logstash /usr/share/logstash/data
USER logstash

# needed to upgrade rabbitmq packaged in the orginal image as 7.0.0 but does not work
RUN ["logstash-plugin", "update", "logstash-integration-rabbitmq"]

Elasticsearch Configuration

Configuration is where the difficulties starts. I won’t detail Kibana’s since it is pretty straightforward, but Elasticsearch and Logstash need more explanation. In my case, I am deploying the containers as a Docker Swarm which implies clustering. The problem with Elasticsearch is that it has its own clustering system. It is not a simple replication of one container.

For Elasticsearch/Kibana (since Kibana stores its data as Elasticsearch indices) to work with multi-nodes, you need to configure the following parameters. ES revamped the way clustering works in 7.X.X. I have tried to configure ES with one service replicated by Docker Swarm, but it just does not work at this point. The problem resides in the way ES manages the list of masters and the host names.

My configuration looked like:

discovery.seed_hosts=elasticsearch
# list of the master nodes
cluster.initial_master_nodes=my-master-node-1,my-master-node-2,etc. 
node.name={{.Node.Hostname}}

I even tried to update the node name with my-es-node.{{.Task.Slot}} based on https://github.com/deviantony/docker-elk/issues/410, but the problem is that ES automatically generates a string added after the node name. The only solution was to create multiple ES configurations for each node.

Warning: this configuration does not guarantee that an instance of the Elasticsearch container will run on your master node(s). That is why we decided to duplicate the elasticsearch configuration for each node though it diminishes the benefits of using Docker Swarm. Ex:

elasticsearch-01:
  image: localhost:5000/elasticsearch-logging:1.0
  deploy:
    ...
  volumes:
    ...
  environment:
    ...
    - discovery.seed_hosts=elasticsearch-02,elasticsearch-03
    - cluster.initial_master_nodes=elasticsearch-01,elasticsearch-02,elasticsearch-03
  elasticsearch-02:
    image: localhost:5000/elasticsearch-logging:1.0
    deploy:
      ...
    volumes:
      ...
    environment:
      ...
      - discovery.seed_hosts=elasticsearch-01,elasticsearch-03
      - cluster.initial_master_nodes=elasticsearch-01,elasticsearch-02,elasticsearch-03
    elasticsearch-XX
      ...

This solution is not finished yet. The configuration is duplicated, but worse, we would lose the “load balancing” benefit from using Docker Swarm. For example, you would need to configure Kibana and Logstash to point to one of the nodes. That is why I needed to create an alias for all the ES masters so that they can be addressed by Kibana and Logstash as “elasticsearch” instead of elasticsearch-01 or 02:

networks:
   my-network:
     aliases:
       - elasticsearch

Elasticsearch 7 also requires distinct data folders for each node. In your elasticsearch.yml, add the following configuration:

volumes:
  # data path must be unique to each node: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes
  # folder must be created on each host before starting the swarm: https://docs.docker.com/engine/swarm/services/#data-volumes
  - /my/node/data/folder:/usr/share/elasticsearch/data:rw

Logstash Configuration

Logstash requires volume configuration for persistence and networks if you need to reach a container on an external network. In my case, logstash is used to send data to a RabbitMQ exchange.

  volumes:
    - /my/logstash/data:/usr/share/logstash/data:rw
    - /my/logstash/pipelines:/usr/share/logstash/pipelines:ro
  networks:
    - my-logstash-net
    - rabbitmq-net

networks:
  my-logstash-net:
    external: true # depends on your configuration
  rabbitmq-net:
    external: true

Now you need to configure the pipelines which allows you to connect an input with an output. One useful setting is the auto-reload of the pipeline configuration:

config.reload.automatic: true # to add in logstash.yml

In this case, I am using Elasticsearch as an input to send data to a RabbitMQ exchange. First, we need to define the pipeline in pipelines.yml:

- pipeline.id: my-pipe-id
  path.config: /usr/share/logstash/pipelines/my-pipe.conf # define where the configuration file is

I created my-pipe.conf with the following configuration. It is worth noting the need for scheduling the elasticsearch input as well as the index to use.

# Configuration to send the error logs to notification modules
input {
  elasticsearch {
    hosts => ['elasticsearch:9200']
    schedule => "*/15 * * * *" # every 15 minutes. If not scheduled, the pipeline does not work. The logs said connection to RabbitMQ "opened" and then "closed" a couple of second after
    query => '{
	    "query" :  {
		    "bool": {
			    "must": {
			      "regexp": {"log": ".*exception.*|.*error.*"} # search everything with exception or error
			    },
			    "filter": {
				    "range": {"@timestamp": {"gte": "now-15m"} } # only search the data logged in the past 15 minutes since the pipeline is triggered every 15 min.
			    }
		    }
	    }
    }'
    index => "my-custom-index-*" # !! by default, the index is not *, but logstash-* which is an issue if you configured Elasticsearch indices with a different prefix!
  }
}

filter {
  mutate {
    add_field => {
      "log" => "Added by the filter: %{message}" # update as needed
    }
  }
}

output {
  rabbitmq {
    id => "my_id"
    host => "rabbitmq"
    port => "5672"
    user => "my_rabbitmq_user"
    password => "my_rabbitmq_pwd" # Unfortunately, using a file or docker secret is not possible for now...
    connect_retry_interval => 10
    exchange => "my-exchange-name"
    exchange_type => topic  # if topic is your queue type
    codec => json
    key => "my-routing-key"
  }
}

In the case of a file as input, there is no schedule, but this configuration is necessary to feed the pipeline:

start_position => "beginning"

Then build the Docker images and push them to the registry with the following commands:

docker build -t my-tag
docker push my-repo-url

Deployment on Docker Swarm

When deploying on the Swarm, there are a few differences to account for. First, you need to create the volume folders for elasticsearch and logstash. Docker stack does not automatically create the missing folder contrary to docker-compose. Seond, change permission on the folders on the mapped folders to allow elasticsearch and logstash users to be able to write data, i.e chmod 777 /mnt/my/data/folder (not /usr/share/… in the containers)

Then deploy with:

docker stack deploy -c my-docker-stack.yml my-cluster-name

If you see Kibana messing up in the logs or if you are not able to create and retrieve an index pattern, then you messed up the Elasticsearch clustering configuration. Here is an example of errors I saw in the logs

Error: [resource_already_exists_exception] index [.kibana_1/ZeFEi9gkSESfQYSOkZwoFA] already exists, with { index_uuid=\"ZeFEi9gkSESfQYSOkZwoFA\" & index=\".kibana_1\" }"}

Good Luck!! Hours and hours can be spent on Docker…

Author: Toujon Lyfoung

This paragraph is supposed to be the place where I put my credentials and achievements. In my opinion, degrees and jobs do not tell much about a person. If you want to know me, read my posts! Blogging has been fun. I do not pretend to do much. I am simply processing, tracking and sharing my reflection. Comments are definitely welcomed to help me continue in my learning.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s