ELK version: 7.17.10
Charts
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
elasticsearch logging 1 2023-10-09 22:36:22.805826199 +0000 UTC deployed elasticsearch-7.17.10-spacex.6.3 7.17.10
kibana logging 1 2023-10-09 22:43:26.513922363 +0000 UTC deployed kibana-7.17.10-spacex.6.1 7.17.10
Pods
NAME READY STATUS RESTARTS AGE
elk-master-0 3/3 Running 0 45h
elk-master-1 3/3 Running 0 45h
elk-master-2 3/3 Running 0 45h
kibana-kibana-c479b8d7f-ddhzs 3/3 Running 0 46h
ldapsearch or ldapwhoami CLI-- i.e. running elasticsearch-keystore show shows the password for bind_dn user (elk-bind)
-- Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html
$ curl http://127.0.0.1:9200 -u elastic:${p} -v
* About to connect() to 127.0.0.1 port 9200 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic Kmxkc3RpYzpGoodLuckWithThat9OEotQ3e6
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 544
<
{
"name" : "elk-master-0",
"cluster_name" : "elk_cluster",
"cluster_uuid" : "31tgxJzTSdGRbn9PbWDt7k",
"version" : {
"number" : "7.17.10",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "fecd68e3150eda0c307ab9a9d7557f5d5fd71349",
"build_date" : "2023-04-23T05:33:18.138275597Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Following is my elasticsearch yml file for xpack settings:
clusterName: "elk_cluster"
masterService: "elk-master"
minimumMasterNodes: 1
podAnnotations:
consul.hashicorp.com/connect-inject: true
consul.hashicorp.com/connect-service: elasticsearch
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-run-as-user: 22020
vault.hashicorp.com/agent-run-as-group: 22020
vault.hashicorp.com/role: elasticsearch
vault.hashicorp.com/auth-path: auth/kubernetes-secure
vault.hashicorp.com/tls-skip-verify: true
vault.hashicorp.com/agent-inject-secret-elk-bind: "secret/data/passwords/ldap/elk-bind"
vault.hashicorp.com/agent-inject-perms-elk-bind: "0644"
vault.hashicorp.com/agent-inject-template-elk-bind: |
{{- with secret "secret/data/passwords/ldap/elk-bind" -}}
{{ .Data.data.value }}
{{- end }}
vault.hashicorp.com/agent-inject-secret-elastic: secret/data/passwords/elk/elastic
vault.hashicorp.com/agent-inject-command-elastic: chmod 400 /vault/secrets/elastic
vault.hashicorp.com/agent-inject-template-elastic: |
{{- with secret "secret/data/passwords/elk/elastic" -}}
{{ .Data.data.value }}
{{- end }}
vault.hashicorp.com/agent-inject-secret-kibana: secret/data/passwords/elk/kibana
vault.hashicorp.com/agent-inject-command-kibana: chmod 400 /vault/secrets/kibana
vault.hashicorp.com/agent-inject-template-kibana: |
{{- with secret "secret/data/passwords/elk/kibana" -}}
{{ .Data.data.value }}
{{- end }}
extraEnvs:
- name: ELASTIC_USERNAME
value: elastic
- name: ELASTIC_PASSWORD_FILE
value: /vault/secrets/elastic
extraInitContainers:
- name: ulimit-1
image: artifactory:8443/docker/elastic/elasticsearch:7.17.10
command: ["/bin/sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
- name: ulimit-2
image: artifactory:8443/docker/elastic/elasticsearch:7.17.10
command: ["/bin/sh", "-c", "ulimit -u 4096"]
securityContext:
privileged: true
- name: wait-for-consul
image: artifactory:8443/docker/elastic/elasticsearch:7.17.10
command:
- "/bin/bash"
- "-c"
- |
consul_url=https://$CONSUL_NAME:8501/v1/health/node/$NODE_NAME
while [[ "$(curl -sk -o /dev/null -w '%{http_code}\n' $consul_url)" != "200" ]] && [[ "$(curl -sk $consul_url)" = "[]" ]]; \
do echo waiting for consul; sleep 5; done
env:
- name: CONSUL_NAME
value: consul-consul-server-0.consul-consul-server.consul.svc.cluster.local
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
extraVolumes:
- name: elasticsearch-tls-cert
secret:
secretName: elasticsearch-cert
- name: cacerts
persistentVolumeClaim:
claimName: cacerts
- name: elasticsearch-data
persistentVolumeClaim:
claimName: elasticsearch-data
- name: elasticsearch-backup
persistentVolumeClaim:
claimName: elasticsearch-backup
extraVolumeMounts:
- name: elasticsearch-tls-cert
readOnly: true
mountPath: /usr/share/elasticsearch/config/elasticsearch.crt
subPath: tls.crt
- name: elasticsearch-tls-cert
readOnly: true
mountPath: /usr/share/elasticsearch/config/elasticsearch.key
subPath: tls.key
- name: cacerts
mountPath: /var/local/cacerts.d
- name: cacerts
mountPath: /usr/share/elasticsearch/config/CA_cert.crt
subPath: CA_cert.crt
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
- name: elasticsearch-backup
mountPath: /usr/backup
esJavaOpts: "-Xmx25g -Xms25g"
breakerLimit: "80%"
resources:
requests:
cpu: "1000m"
memory: "4Gi"
limits:
cpu: "8000m"
memory: "32Gi"
initResources:
limits:
cpu: "200m"
memory: "128Mi"
requests:
cpu: "200m"
memory: "128Mi"
rbac:
create: true
serviceAccountName: "elasticsearch"
podSecurityPolicy:
create: true
podSecurityContext:
fsGroup: 22020
runAsUser: 22020
securityContext:
runAsUser: 22020
persistence:
enabled: false
esConfig:
elasticsearch.yml: |
path.repo: [ "/usr/backup" ]
node.max_local_storage_nodes: 3
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
ingest.geoip.downloader.enabled: false
#index.merge.scheduler.max_thread_count: 1
cluster.name: elk_cluster
network.host: "elk-master.logging"
xpack.license.self_generated.type: trial
xpack:
security:
authc:
realms:
ldap:
ldap1:
order: 0
metadata: cn
# secure port 636 of ldap connection, no need to define below, even if you define, no behavior changed.
url: "ldaps://ldap"
#bind_dn: "cn=elk-bind,ou=serviceaccounts,dc=infra.spacex"
# !!NOTE!!: In my LDAP, elk-bind is setup using uid instead of cn. It's a service account and it'll be used to bind to LDAP first(using uid=elk-bind and secure_bind_password) i.e. es keystore's xpack.security.authc.realms.ldap.ldap1.secure_bind_password
# ----- THEN >> it'll search for the DN of the Kibana UI's "login user (nklbobbyb)", and using Kibana login user's password (entered through Kibana UI) to authenticate with LDAP.
# ----- This process will search the user_search.filter option to find the DN of the Kibana's UI login user "nklbobbyb" (i.e. ldap user that I'm using in Kibana UI)
bind_dn: "uid=elk-bind,ou=ServiceAccounts,dc=infra.spacex"
# The following setting 'secure_bind_password' is already shoved to elasticsearch-keystore. elasticsearch-keystore add xpack.security.authc.realms.ldap.ldap1.secure_bind_password and elasticsearch-keystore show <same path> spits the value for secure_bind_password used for the bind_dn user i.e. uid=elk-bind.
# -- I don't think ELK official docs for 7.x/8.x requires the following line or anything like that. Saw a blog with this, will try to see if it helps.
#secure_bind_password: xpack.security.authc.realms.ldap.ldap1.secure_bind_password
ssl.verification_mode: none
#ssl.verification_mode: certificate
user_search.base_dn: "ou=People,dc=infra.spacex"
user_search.filter: "(uid={0})"
#user_search.filter: "(cn={0})"
group_search.base_dn: "ou=Groups,dc=infra.spacex"
# As role mapping file is a file on the filesystem, we don't need to explicitly call REST API (mapping api) to map Kibana roles with LDAP groups. This file IIRC is read every 5 seconds(default) by ELK for mapping purposes.
files:
role_mapping: "/usr/share/elasticsearch/config/role_mapping.yml"
unmapped_groups_as_roles: false
native:
native1:
order: 1
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/CA_cert.crt" ]
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.ignore_filters.test.users: [ "any_logstash_user_writer", "kibana" ]
# For testing purpose, I'm not much worried about the following settings (secure / encryption settings for now). later.
# ----------------------------------------------------------------------
#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.verification_mode: none
##xpack.security.http.ssl.verification_mode: certificate
#xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
#xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
#xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/CA_cert.crt" ]
#xpack.security.authc.token.enabled: true
logger.org.elasticsearch: DEBUG
logger.org.elasticsearch.http: DEBUG
logger.org.elasticsearch.transport: DEBUG
roles.yml: |
ams:
cluster: [ "manage_index_templates", "monitor" ]
indices:
- names: [ "metricbeat-*", "syslog-*" ]
privileges: [ 'read' ]
beat_reader:
indices:
- names: [ "metricbeat-*" ]
privileges: ["read", "view_index_metadata"]
beat_writer:
cluster: [ "manage_index_templates","monitor","manage_ilm" ]
indices:
- names: [ "metricbeat-*" ]
privileges: ["read","write","delete","create_index","manage","manage_ilm"]
# Kibana 7.16.3 missing manage permissions on indexes
kibana_supp:
cluster: [ "all" ]
indices:
- names: [ "*" ]
privileges: ["manage"]
sec_admin:
cluster: [ "manage_index_templates", "monitor" ]
indices:
- names: [ 'metricbeat-*', 'syslog-*' ]
privileges: [ 'read' ]
role_mapping.yml: |
ams:
- "cn=sys_integration,ou=Groups,dc=infra.spacex"
kibana_admin:
- "cn=system_admin,ou=Groups,dc=infra.spacex"
kibana_system:
- "cn=system_admin,ou=Groups,dc=infra.spacex"
kibana_user:
- "cn=ldap_admins,ou=Groups,dc=infra.spacex"
- "cn=system_admin,ou=Groups,dc=infra.spacex"
reporting_user:
- "cn=sys_integration,ou=Groups,dc=infra.spacex"
superuser:
- "cn=system_admin,ou=Groups,dc=infra.spacex"
- "uid=nklbobbyb,ou=people,dc=infra.spacex"
image: artifactory:8443/docker/elastic/elasticsearch
service:
type: LoadBalancer
annotations:
metallb.universe.tf/address-pool: default
metallb.universe.tf/allow-shared-ip: elasticsearch
loadBalancerIP: 10.20.30.40
nodeSelector:
com.company.host.hostType: deployhosts
ingress:
enabled: true
path: /
hosts:
- elasticsearch.domain.secure
- elasticsearch
tls:
- secretName: elasticsearch-cert
hosts:
- elasticsearch.domain.secure
- elasticsearch
# Do not need a high successThreshold because lifecycle won't work unless service is listening
readinessProbe:
successThreshold: 1
# Must be yellow or green to be ready (yellow meaning not all shared are assigned/active)
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
## POST START HOOK is where I'm using shell code to add secure_bind_password to elasticsearch-keystore.
lifecycle:
postStart:
exec:
command:
- bash
- -c
- |
#!/bin/bash
# Update passwords based on what is in vault
source /usr/share/elasticsearch/bin/elasticsearch-env-from-file
CREDS="-u $ELASTIC_USERNAME:$ELASTIC_PASSWORD"
ES_URL=http://localhost:9200
while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $CREDS $ES_URL)" != "200" ]]; do sleep 2; done
curl -k $CREDS -XPOST "${ES_URL}/_security/user/healthcheck" -H 'Content-Type: application/json' -d '{ "password":"'healthcheck'", "roles" : ["remote_monitoring_collector"] }'
# Set bind_dn(ldap) password (for uid=elk-bind) as secure_bind_password in ElasticSearch Keystore using elk-bind secret
# --------------------------
# https://www.elastic.co/guide/en/elasticsearch/reference/current/ldap-realm.html#mapping-roles-ldap
#
if test -f "/vault/secrets/elk-bind"; then
echo "Injecting bind_dn password as secure_bind_password (in ElasticSearch Keystore) using elk-bind secret (from Vault)"
ELK_BIND_PASSWORD="$(cat /vault/secrets/elk-bind)"
echo "${ELK_BIND_PASSWORD}" | elasticsearch-keystore add xpack.security.authc.realms.ldap.ldap1.secure_bind_password
fi
# Create a different kibana user because the default one in 7.16.3 is broken (permissions issues)
# May be able to be removed at a newer version (replace all instances of kibana_mgmr)
if test -f "/vault/secrets/kibana"; then
echo "Setting kibana credentials"
KIB_PASSWORD=$(cat /vault/secrets/kibana)
curl -k $CREDS -XPOST "${ES_URL}/_security/user/kibana_mgmr" -H 'Content-Type: application/json' -d '{ "password":"'$KIB_PASSWORD'", "roles" : ["kibana_system", "kibana_supp"] }'
fi
# Sleeping for a period of time before gathering indices to let elasticsearch load index names from filesystem
# Currently do not know of an endpoint that we can use to check for index loading.
sleep 60
# IGNORE ----- All code listed below.
# some curl etc command below.
Main / relevant code in Kibana .yml looks like:
elasticsearchHosts: "http://elk-master:9200"
#... more stuff ...generic stuff
kibanaConfig:
kibana.yml: |-
server.name: kibana
server.host: 0.0.0.0
server.publicBaseUrl: https://kibana.domain.secure
xpack.monitoring.ui.container.elasticsearch.enabled: true
#elasticsearch.hosts: ["http://elk-master.logging:9200"]
elasticsearch.hosts: ["http://elk-master:9200"]
logging.root.level: debug
elasticsearch.ssl.verificationMode: none
#elasticsearch.ssl.verificationMode: certificate
server.ssl.clientAuthentication: none
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "1000m"
memory: "2Gi"
podSecurityContext:
fsGroup: 22060
securityContext:
runAsUser: 22060
serviceAccount: "kibana"
extraVolumes:
- name: keystore
emptyDir: {}
extraVolumeMounts:
- name: keystore
mountPath: /usr/share/kibana/config/kibana.keystore
subPath: kibana.keystore
service:
type: LoadBalancer
loadBalancerIP: 10.20.30.40
ingress:
enabled: true
path: /
hosts:
- kibana.domain.secure
- kibana
tls:
- secretName: kibana-cert
hosts:
- kibana.domain.secure
- kibana
nodeSelector:
com.company.host.hostType: deployhosts
When I login using nklbobbyb (in Kibana UI), I'm seeing the following in elk-master-0 pod's log. i.e.
$ kubectl logs -n logging elk-master-0 -c elasticsearch -f | egrep -i "ldap|nklbobbyb|auth"
ERROR mesg is listed below:
{"type": "server", "timestamp": "2023-10-02T19:52:36,018Z", "level": "WARN", "component": "o.e.x.s.a.l.s.LdapUtils", "cluster.name": "elk_cluster", "node.name": "elk-master-0", "message": "Failed to obtain LDAP connection from pool - LDAPException(resultCode=89 (parameter error), diagnosticMessage='Simple bind operations are not allowed to contain a bind DN without a password.', ldapSDKVersion=4.0.8, revision=28812)", "cluster.uuid": "31tgxJzTSdGRbn9PbWDt7K", "node.id": "lOYvRY7zSNW7ZMwewL9Vzg" }
{"type": "server", "timestamp": "2023-10-02T19:52:36,018Z", "level": "WARN", "component": "o.e.x.s.a.RealmsAuthenticator", "cluster.name": "elk_cluster", "node.name": "elk-master-0", "message": "Authentication to realm ldap1 failed - authenticate failed (Caused by LDAPException(resultCode=89 (parameter error), diagnosticMessage='Simple bind operations are not allowed to contain a bind DN without a password.', ldapSDKVersion=4.0.8, revision=28812))", "cluster.uuid": "31tgxJzTSdGRbn9PbWDt7K", "node.id": "lOYvRY7zSNW7ZMwewL9Vzg" }
Kibana logs shows:
$ kubectl logs -n logging kibana-kibana-5645ccf7dd-kx7jh -c kibana -f | egrep -i "ldap|asangal|auth"
---
{"type":"log","@timestamp":"2023-10-02T21:22:32+00:00","tags":["info","plugins","security","authentication"],"pid":8,"message":"Authentication attempt failed: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [nklbobbyb] for REST request [/_security/_authenticate]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [nklbobbyb] for REST request [/_security/_authenticate]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"}
===
What I could be missing to get this simple LDAP working?
It's very odd that in Kibana UI, when I log in as "elastic" (which has all the superuser powers), there's NO LDAP section under Security section, i.e. so a user can see the settings within LDAP in a web browser..
I was missing few things.
We can use both either lifeCycle > postStart or go with extraInitContainers way, but I went with the latter approach.
Solution:
Under podAnnotations: add agent-init-first: true, this will make SECRETS (from vault) available to extra Init level containers as well. Otherwise, those secrets won't be visible to any extra Init level containers.
Note: Keep those agent-inject-secret-elk-bind and lines
Define an entry in extraInitContainer (in the above yml, after wait-for-consul section). See below
Under extraVolumeMounts: add keystore path, see below.
Under extraVolumes: add keystore volume path
bind_dn line under xpack section, will remain as it is. In my case, I also removed/commented out any user|group.filter lines
vault.hashicorp.com/agent-init-first: true
- name: elasticsearch-keystore
image: artifactory:8443/docker/elastic/elasticsearch:7.17.10
command:
- bash
- -c
- |
set -euo pipefail
elasticsearch-keystore create
echo "Adding bootstrap.password to keystore"
cat $ELASTIC_PASSWORD_FILE | bin/elasticsearch-keystore add --stdin bootstrap.password
echo "Adding bind_dn password as secure_bind_password to keystore"
cat $ELASTIC_BIND_PASSWORD_FILE | bin/elasticsearch-keystore add --stdin xpack.security.authc.realms.ldap.ldap1.secure_bind_password
cp -a /usr/share/elasticsearch/config/elasticsearch.keystore /tmp/keystore/
env:
- name: ELASTIC_BIND_PASSWORD_FILE
value: /vault/somepath/secrets/elk-bind
- name: ELASTIC_PASSWORD_FILE
value: /vault/somepath/secrets/elastic
securityContext:
runAsUser: 23059
volumeMounts:
- name: keystore
mountPath: /tmp/keystore
- name: keystore
mountPath: /usr/share/elasticsearch/config/elasticsearch.keystore
subPath: elasticsearch.keystore
- name: keystore
emptyDir: {}