We have elasticsearch cluster at ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}
and filebeat pod at k8s cluster that exports other pods' logs
There is filebeat.yml:
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: develop
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
hints.enabled: true
hints.default_config:
type: container
multiline.type: pattern
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
http:
enabled: true
host: localhost
port: 5066
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
indices:
- index: "develop"
when:
equals:
kubernetes.namespace: "develop"
- index: "kubernetes-dev"
when:
not:
and:
- equals:
kubernetes.namespace: "develop"
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- decode_json_fields:
fields: ["message"]
add_error_key: true
process_array: true
overwrite_keys: false
max_depth: 10
target: json_message
I've checked: filebeat has access to /var/log/containers/ on kuber but elastic cluster still doesn't get any develop or kubernetes-dev indices. (Cluster has relative index templates for this indices)
http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/health?pretty:
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 14,
"active_shards" : 28,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Filebeat log:
{
"log.level": "info",
"@timestamp": "2022-11-25T08:35:18.084Z",
"log.logger": "monitoring",
"log.origin": {
"file.name": "log/log.go",
"file.line": 184
},
"message": "Non-zero metrics in the last 30s",
"service.name": "filebeat",
"monitoring": {
"metrics": {
"beat": {
"cgroup": {
"cpu": {
"stats": {
"periods": 38
}
},
"cpuacct": {
"total": {
"ns": 1576170001
}
},
"memory": {
"mem": {
"usage": {
"bytes": 4096
}
}
}
},
"cpu": {
"system": {
"ticks": 13570,
"time": {
"ms": 56
}
},
"total": {
"ticks": 23320,
"time": {
"ms": 90
},
"value": 23320
},
"user": {
"ticks": 9750,
"time": {
"ms": 34
}
}
},
"handles": {
"limit": {
"hard": 1048576,
"soft": 1048576
},
"open": 11
},
"info": {
"ephemeral_id": "a88f461e-1fd3-48d3-a1c7-3bfea1124ab9",
"uptime": {
"ms": 7530198
},
"version": "8.1.0"
},
"memstats": {
"gc_next": 17767824,
"memory_alloc": 16385608,
"memory_total": 709250600,
"rss": 104206336
},
"runtime": {
"goroutines": 23
}
},
"filebeat": {
"harvester": {
"open_files": 0,
"running": 0
}
},
"libbeat": {
"config": {
"module": {
"running": 0
}
},
"output": {
"events": {
"active": 0
}
},
"pipeline": {
"clients": 0,
"events": {
"active": 0
}
}
},
"registrar": {
"states": {
"current": 0
}
},
"system": {
"load": {
"1": 3.99,
"15": 3.91,
"5": 4.08,
"norm": {
"1": 0.4988,
"15": 0.4888,
"5": 0.51
}
}
}
},
"ecs.version": "1.6.0"
}
}
Where can be problem? Im absolutely new to elk so working with it not from the start is a bit hard
In the end I've found out that elastic cluster's version used by company is much older than that of filebeat (8.1 against 7.9). So temporary allowing usage of older versions has solved the issue for time being.
output.elasticsearch:
allow_older_versions: true
BUT @Paulo also had a point and construction not + and was excessive in the end.