Elasticsearch, Logstash and Kibana (ELK) v7.5 … running on Docker Swarm

Docker is a great concept. The idea is to create containers that we can simply download and run on servers without going through the process of installation again and again… But (you were expecting it!) it adds its own complexity. We need to learn how Docker works – which seems to be endless madness. There is quite a bit of commands to learn and configurations to do. And lastly, we have to hope that the software has somme sort of support for Docker.

Creating Docker images

I created customized images of ELK 7.5 to integrate the configuration and also by necessity for Logstash to update the RabbitMQ output plugin. For each component, we need a Dockerfile with its configuration.

Elasticsearch

FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.0

# Change ownership of the data files
USER root
RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
USER elasticsearch

# copy configuration files from local to container
COPY config/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml

Kibana

FROM docker.elastic.co/kibana/kibana:7.5.0

# Add your kibana plugins setup here
# Example: RUN kibana-plugin install 
COPY config /usr/share/kibana/config

Logstash

FROM docker.elastic.co/logstash/logstash:7.5.0

# Copy configuration files
COPY config /usr/share/logstash/config
COPY pipelines /usr/share/logstash/pipelines

# Change ownership of the data files
USER root
RUN chown -R logstash:logstash /usr/share/logstash/data
USER logstash

# needed to upgrade rabbitmq packaged in the orginal image as 7.0.0 but does not work
RUN ["logstash-plugin", "update", "logstash-integration-rabbitmq"]

Elasticsearch Configuration

Configuration is where the difficulties starts. I won’t detail Kibana’s since it is pretty straightforward, but Elasticsearch and Logstash need more explanation. In my case, I am deploying the containers as a Docker Swarm which implies clustering. The problem with Elasticsearch is that it has its own clustering system. It is not a simple replication of one container.

For Elasticsearch/Kibana (since Kibana stores its data as Elasticsearch indices) to work with multi-nodes, you need to configure the following parameters. ES revamped the way clustering works in 7.X.X. I have tried to configure ES with one service replicated by Docker Swarm, but it just does not work at this point. The problem resides in the way ES manages the list of masters and the host names.

My configuration looked like:

discovery.seed_hosts=elasticsearch
# list of the master nodes
cluster.initial_master_nodes=my-master-node-1,my-master-node-2,etc. 
node.name={{.Node.Hostname}}

I even tried to update the node name with my-es-node.{{.Task.Slot}} based on https://github.com/deviantony/docker-elk/issues/410, but the problem is that ES automatically generates a string added after the node name. The only solution was to create multiple ES configurations for each node.

Warning: this configuration does not guarantee that an instance of the Elasticsearch container will run on your master node(s). That is why we decided to duplicate the elasticsearch configuration for each node though it diminishes the benefits of using Docker Swarm. Ex:

elasticsearch-01:
  image: localhost:5000/elasticsearch-logging:1.0
  deploy:
    ...
  volumes:
    ...
  environment:
    ...
    - discovery.seed_hosts=elasticsearch-02,elasticsearch-03
    - cluster.initial_master_nodes=elasticsearch-01,elasticsearch-02,elasticsearch-03
  elasticsearch-02:
    image: localhost:5000/elasticsearch-logging:1.0
    deploy:
      ...
    volumes:
      ...
    environment:
      ...
      - discovery.seed_hosts=elasticsearch-01,elasticsearch-03
      - cluster.initial_master_nodes=elasticsearch-01,elasticsearch-02,elasticsearch-03
    elasticsearch-XX
      ...

This solution is not finished yet. The configuration is duplicated, but worse, we would lose the “load balancing” benefit from using Docker Swarm. For example, you would need to configure Kibana and Logstash to point to one of the nodes. That is why I needed to create an alias for all the ES masters so that they can be addressed by Kibana and Logstash as “elasticsearch” instead of elasticsearch-01 or 02:

networks:
   my-network:
     aliases:
       - elasticsearch

Elasticsearch 7 also requires distinct data folders for each node. In your elasticsearch.yml, add the following configuration:

volumes:
  # data path must be unique to each node: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes
  # folder must be created on each host before starting the swarm: https://docs.docker.com/engine/swarm/services/#data-volumes
  - /my/node/data/folder:/usr/share/elasticsearch/data:rw

Logstash Configuration

Logstash requires volume configuration for persistence and networks if you need to reach a container on an external network. In my case, logstash is used to send data to a RabbitMQ exchange.

  volumes:
    - /my/logstash/data:/usr/share/logstash/data:rw
    - /my/logstash/pipelines:/usr/share/logstash/pipelines:ro
  networks:
    - my-logstash-net
    - rabbitmq-net

networks:
  my-logstash-net:
    external: true # depends on your configuration
  rabbitmq-net:
    external: true

Now you need to configure the pipelines which allows you to connect an input with an output. One useful setting is the auto-reload of the pipeline configuration:

config.reload.automatic: true # to add in logstash.yml

In this case, I am using Elasticsearch as an input to send data to a RabbitMQ exchange. First, we need to define the pipeline in pipelines.yml:

- pipeline.id: my-pipe-id
  path.config: /usr/share/logstash/pipelines/my-pipe.conf # define where the configuration file is

I created my-pipe.conf with the following configuration. It is worth noting the need for scheduling the elasticsearch input as well as the index to use.

# Configuration to send the error logs to notification modules
input {
  elasticsearch {
    hosts => ['elasticsearch:9200']
    schedule => "*/15 * * * *" # every 15 minutes. If not scheduled, the pipeline does not work. The logs said connection to RabbitMQ "opened" and then "closed" a couple of second after
    query => '{
	    "query" :  {
		    "bool": {
			    "must": {
			      "regexp": {"log": ".*exception.*|.*error.*"} # search everything with exception or error
			    },
			    "filter": {
				    "range": {"@timestamp": {"gte": "now-15m"} } # only search the data logged in the past 15 minutes since the pipeline is triggered every 15 min.
			    }
		    }
	    }
    }'
    index => "my-custom-index-*" # !! by default, the index is not *, but logstash-* which is an issue if you configured Elasticsearch indices with a different prefix!
  }
}

filter {
  mutate {
    add_field => {
      "log" => "Added by the filter: %{message}" # update as needed
    }
  }
}

output {
  rabbitmq {
    id => "my_id"
    host => "rabbitmq"
    port => "5672"
    user => "my_rabbitmq_user"
    password => "my_rabbitmq_pwd" # Unfortunately, using a file or docker secret is not possible for now...
    connect_retry_interval => 10
    exchange => "my-exchange-name"
    exchange_type => topic  # if topic is your queue type
    codec => json
    key => "my-routing-key"
  }
}

In the case of a file as input, there is no schedule, but this configuration is necessary to feed the pipeline:

start_position => "beginning"

Then build the Docker images and push them to the registry with the following commands:

docker build -t my-tag
docker push my-repo-url

Deployment on Docker Swarm

When deploying on the Swarm, there are a few differences to account for. First, you need to create the volume folders for elasticsearch and logstash. Docker stack does not automatically create the missing folder contrary to docker-compose. Seond, change permission on the folders on the mapped folders to allow elasticsearch and logstash users to be able to write data, i.e chmod 777 /mnt/my/data/folder (not /usr/share/… in the containers)

Then deploy with:

docker stack deploy -c my-docker-stack.yml my-cluster-name

If you see Kibana messing up in the logs or if you are not able to create and retrieve an index pattern, then you messed up the Elasticsearch clustering configuration. Here is an example of errors I saw in the logs

Error: [resource_already_exists_exception] index [.kibana_1/ZeFEi9gkSESfQYSOkZwoFA] already exists, with { index_uuid=\"ZeFEi9gkSESfQYSOkZwoFA\" & index=\".kibana_1\" }"}

Good Luck!! Hours and hours can be spent on Docker…

Mockk framework: Could not initialize class io.mockk.impl.JvmMockKGateway

If you are looking for a mock framework for Kotlin tests, Mockk is definitely on first on the list. I wrote my first test following a simple example on the official website (https://mockk.io/#dsl-examples). In my case, I am testing an export class using a parser to deserialize a file into an object:

@Test
fun `test export excluding sub-folders`() {
    // given
    val path = "my/new/path"
    val parser = mockk()
    every { parser.buildObject("$path/data.xml") } returns MyObject("first name", "last name", "title", "manager")

    // when
    var myObject = exporter.parse(
            parser,
            getResourceFile(path),
            mutableListOf()
        )

    // then
    verify { parser.buildObject("$path/data.xml") }
    confirmVerified(parser) // verifies all the verify calls were made
    assertEquals("first name", myObject.firstName)
    assertEquals("last name", myObject.lastName)
    assertEquals("title", myObject.title)
    assertEquals("manager", myObject.manager)

}

Wunderbar! I ran my test… and it failed:

java.lang.NoClassDefFoundError: Could not initialize class io.mockk.impl.JvmMockKGateway

What in the world is that?! I checked my pom file and verified I had the latest version:

<dependency>
  <groupId>io.mockk</groupId>
  <artifactId>mockk</artifactId>
  <version>1.9.3</version>
  <scope>test</scope>
</dependency>

After a while, I figured out the issue resides in the IDE (https://github.com/mockk/mockk/issues/254). Mockk has issue when running on JDK 11. I went to Intellij IDEA -> File -> Project Structure, and changed the Project SDK to JDK 8 (which I use for my current project). You can probably find another version that would work for your own project.

Defying Windows’ 260 character paths!

Windows is known for limiting paths to 260 characters which can be an issues when you deal with java packages. You cannot write, rename or delete files once the limit is reached. However there are a few tricks that can be helpful to go around this issue if you cannot shorten your paths.

Since Windows 10 Build 14352 and later, it is now possible to enable long paths… but it only works with Powershell (!!) Windows Explorer and third party applications do not work. (Please do not tell me this is weird…)

  1. Enable Long path on windows
    1.1 Go to start menu and open Group Policy in Control Panel by searching for “gpedit”

1.2 Go to Local Computer Policy → Administrative Templates → System → Filesystem and edit “Enable Win32 long paths”

1.3 Enable this option

Apply policy change by running this command in a terminal:

gpupdate /target:computer /force

If you are using Git to clone a repository that contains long paths, Git allows you to add a setting to handle them. For that, open a new terminal as admin (I repeat “as admin”), and run:

git config --system core.longpaths true

Hopefully, this helps… until Windows find something more coherent. Why would you disable long paths by default? Unless it is because only Powershell can use it 🙂

Important tricks to create Draw.io diagrams programmatically with CSV format

Throughout the years, I have seen short programs exporting dependencies from POM files into visual diagrams. A few of them are in 3D and it looks pretty cool… but it is not very helpful. Most of the projects I have worked on have too many dependencies to be viewable in a diagram format. Lately, I have written a short program to export those dependencies and modules in a csv format importable by draw.io. I added filters to remove dependencies that are not useful. This post is about how I had to deal with the draw.io format to accomplish that task.

The official documentation is here: https://about.draw.io/automatically-create-draw-io-diagrams-from-csv-files/

Draw.io provides a template with instructions that you can find under Arrange -> Insert -> CSV:

You can change the info you want to display as well as the styles of the boxes and arrows. You can find examples here: https://about.draw.io/import-from-csv-to-drawio/

First trick: To figure out what style what I wanted, I used draw.io’s UI to set it up usng “edit style” and copied/pasted the style in the csv content.

## Node style (placeholders are replaced once).
## Default is the current style for nodes.
#
# style: label;whiteSpace=wrap;html=1;rounded=1;fontStyle=0;spacingLeft=0;strokeWidth=1;align=center;%ExtraStyle%
#

Second trick: Simple one, but very useful: one csv line = one block! To create connections, you need at least two columns: your data ID and the referent data ID. However, having a referent ID does not create a block. If it does not exist, Draw.io will ignore it.

Third trick: Connections! the csv format is not well designed to handle nuumerous one to many/many to many relations. Remember the second trick: one line = one block. if you have:

id 1, connection 1
id 1, connection 2
id 1, connection 3

Draw IO will display three id 1 blocks having each their connection. Instead, what is needed is:

id 1, connection 1, connection 2, connection 3...

This implies that you create as many connection configurations as connections you need.

# connect: {"from": "id 1", "to": "connection 1", "invert": true, "label": "depends on", \
#          "style": "curved=1;endArrow=blockThin;endFill=1;fontSize=11;"}
# connect: {"from": "id 1", "to": "connection 2", "invert": true, "label": "depends on", \
#          "style": "curved=1;endArrow=blockThin;endFill=1;fontSize=11;"}
# connect: {"from": "id 1", "to": "connection 3", "invert": true, "label": "depends on", \
#          "style": "curved=1;endArrow=blockThin;endFill=1;fontSize=11;"}

etc.

And this can look tedious, but it is very easy to do programmatically.

for (i in 1..maxNumDependencies) {
    destinationFile.appendText(
        "# connect: {\"from\": \"DependencyId$i\", \"to\": \"Id\", \"invert\": false, \"label\": \"depends on\", \\\n" +
            "#          \"style\": \"curved=1;endArrow=blockThin;endFill=1;fontSize=11;\"}\n"
    )
}

All the rows need to have the same number of columns as the csv header row. Otherwise, the import will be fail:

I was able to import more than 100 dependencies without problem. The only issue is not so much the import, but how to present so many connections in a diagram. It is unreadable!

Last trick: another simple and trivial: make sure there is no empty line. You can waste much time trying to figure this out since the error message does not say much (same as previous one).

My code inserted a empty line before listing the connections:

Connections between rows ("from": source colum, "to": target column).
 Label, style and invert are optional. Defaults are '', current style and false.
 In addition to label, an optional fromlabel and tolabel can be used to name the column
 that contains the text for the label in the edges source or target (invert ignored).
 The label is concatenated in the form fromlabel + label + tolabel if all are defined.
 The target column may contain a comma-separated list of values.
 Multiple connect entries are allowed.
 #
 connect: {"from": "ModuleId1", "to": "Id", "invert": false, "label": "includes", \
 "style": "curved=1;endArrow=blockThin;endFill=1;fontSize=11;dashed=1"}

And I wasted a few minutes to figure out why the import did not work… But now, you know!