Security configuration in ELK Stack

Not too long ago I have installed the ELK stack. It is used for production servers so I decided to enable security connection between Filebeat and Logstash instances and also enable passwords for built-in users. Below described is how I did it.

Securing the Panel and Services Using User Accounts with Appropriate Privileges

1. Turn off Kibana and Elasticsearch:

• service kibana stop
• service elasticsearch stop

2. Add the following in the /etc/elasticsearch/elasticsearch.yml file:

• xpack.security.enabled: true
• discovery.type: single-node

3. Run Elasticsearch using commandservice elasticsearch start.

4. Configure passwords for all built-in users [e.g. elastic, kibana_system, logstash_system, beats_system] by running /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive.

5. Create kibana keystore /usr/share/kibana/bin/kibana-keystore create --allow-root.

6. Add Elasticsearch login and password to keystore:

• /usr/share/kibana/bin/kibana-keystore add elasticsearch.username --allow-root
• /usr/share/kibana/bin/kibana-keystore add elasticsearch.password --allow-root

7. Add the following to the /etc/kibana/kibana.yml file (best at the end for readibility):

• elasticsearch.username: ${elasticsearch.username}
• elasticsearch.password: ${elasticsearch.password}

8. Run Kibana service kibana start.

9. Log into Kibana with user elastic, then navigate to Stack Management → Security → Roles in the menu on the left.

10. Click the Create role button to create a role with a given name and privileges: write, delete, create_index for used indexes: filebeat-*, metrics-*, logs-*. Additionally, configure the monitor and manage_index_templates options for Cluster privileges.

11. Go to Stack Management → Security → Users and create a user assigning it the previously created role.

12. Configure a password to Logstash:

• Create keystore /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create --allow-root
• Add login and password:
• /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add user --allow-root
• /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add password --allow-root
• Configure accesses in the file /etc/logstash/conf.d/general.conf
• user => "${user}"
• password => "${password}"

13. Restart Logstash with the commandservice logstash restart. If the service does not restart it means it is processing something. It’s then worth looking into logs dynamically displayed onto the standard output (the console) with the commandtail -f /var/log/logstash/logstash-plain.log. It’s best to save the logstash process to a variable executing x=$(ps aux | grep logstash | awk 'NR==1 {print $2}') and thenkill -9 $x.

Certificate Configuration

Logstash

1. Create a catalogue certs in the path /etc/logstash. It will store all of the generated files:

• cd /etc/logstash
• mkdir certs

2. Generating a key and key-based CA certificate:

• openssl genrsa -out ca.key 2048
• openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt

3. Generating certificate for logstash:

• Create a configuration file logstash.conf containing data provided at the bottom of the page.
• Generate key — openssl genrsa -out logstash.key 2048.
• Generate Certificate Signing Request (CSR) — openssl req -sha512 -new -key logstash.key -out logstash.csr -config logstash.conf.
• Get CA certificate’s serial number and save it to file serial — openssl x509 -in ca.crt -text -noout -serial | grep serial | awk -F= '{ print $2 }' > serial.
• Create and sign the certificate — openssl x509 -days 3650 -req -sha512 -in logstash.csr -CAserial serial -CA ca.crt -CAkey ca.key -out logstash.crt -extensions v3_req -extfile logstash.conf
• Convert certificate key to a logstash-readable format (pk8) — mv logstash.key logstash.key.pem && openssl pkcs8 -in logstash.key.pem -topk8 -nocrypt -out logstash.key

4. It is critical to change the owner from root to logstash for the following files: ca.crt, logstash.crt, logstash.key.

5. Configure the certificates in an appropriate file described in the next section.

Filebeat

1. In a similar way, create catalogue certs, this time in the path /etc/filebeat.
2. Move the ca.crt, ca.key files and serial (each time after using the option -CAserial in one of the below commands serial number is incremented, that’s why it’s good to move the serial file from the server on which filebeat was installed and configured as last, unless you use only one server with filebeat) from the logstash server to the filebeat server. The easiest way to do it is by displaying the contents of these files in a server console, highlighting the returned text (ensuring there are no whitespaces) and pasting it into an opened file with the same name in another console.
3. Generating a certificate for filebeat:

• Create a configuration file filebeat.conf containing data provided at the bottom of the page.
• Generate a key — openssl genrsa -out filebeat.key 2048.
• Generate Certificate Signing Request (CSR) — openssl req -sha512 -new -key filebeat.key -out filebeat.csr -config filebeat.conf.
• Create and sign the certificate — openssl x509 -days 3650 -req -sha512 -in filebeat.csr -CAserial serial -CA ca.crt -CAkey ca.key -out filebeat.crt -extensions v3_req -extensions usr_cert -extfile filebeat.conf

4. Configure the certificates in an appropriate file described in the next section.

Configuration Files Used in the Previous Section

In the fields commonName and DNS.1 I provided the value returned by hostname -f (if commandshostname and hostname -f return the same value, you have to edit the file /etc/hosts changing the first value in the first line after the IP address to a domain name used by the server):

• logstash.conf

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName                     = X
stateOrProvinceName             = X
localityName                    = X
organizationName                = X
organizationalUnitName          = X
commonName                      = logstash.domain.com
emailAddress                    = admin@domain.com
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = logstash.domain.com

• filebeat.conf

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName                     = X
stateOrProvinceName             = X
localityName                    = X
organizationName                = X
organizationalUnitName          = X
commonName                      = filebeat.domain.com
emailAddress                    = admin@domain.com
[ usr_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL FileBeat Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth

• /etc/logstash/conf.d/general.conf (a custom file for downloading data from filebeat and sending it to elasticsearch — below you will find the part responsible for encrypted communication with filebeat):

input {
  beats {
    client_inactivity_timeout => 600 <---- If filebeat returns an error "write: connection reset by peer" then it's probably caused by rare reporting to logstash. It's useful to add this attribute and set its value to seconds appropriately, in this case 600 was enough.
    port => 5045 <-------------!!!!!! We need to change the port here, because the default 5044 is probably responsible for communication using http only. There's no such information in the official documentation, but in the logs for 5044 the following notification appears: "Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record" which suggests https traffic on the http channel
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.key"
    ssl_verify_mode => "force_peer"
  }
}

• /etc/filebeat/filebeat.yml (default file where the download and sending of data to logstash/elasticsearch is configured — you’ll find the part responsible for encrypted communication with logstash below):

output.logstash:
  # The Logstash hosts - domain name below
  hosts: ["logstash.domain.com:5045"] <-------------!!!!!! We need to change the port here, because the default 5044 in logstash is probably responsible for http communication only.
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
  # Certificate for SSL client authentication
  ssl.certificate: "/etc/filebeat/certs/filebeat.crt"
  # Client Certificate Key
  ssl.key: "/etc/filebeat/certs/filebeat.key"
  # I added supported protocols, the default is from TLS1.0 - not secure
  ssl.supported_protocols: [TLSv1.2, TLSv1.3]

Disclaimer: This tutorial was done on fresh installation Ubuntu Server 20.04 LTS and ELK stack 7.10.