I have a kubernetes cluster on which I have deployed a opensearch cluster and opensearch dashboard using Helm, I am also able to deploy logstash using helm successfully but I am confused on how to integrate those, I want to feed data to my Opensearch using logstash as my OBJECTIVE as I am not able to find much documentation on it as well. Any help is appreciated....Thanks in advance!
Deployed opensearch using Helm and logstash as well but unable to integrate them
Update here!!!
Have made a few changes to simplify the deployment and more control over the function,
I am testing deployment and service files this time, I will add the files below
Opensearch deployment file
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: logging
name: opensearch
labels:
component: opensearch
spec:
selector:
matchLabels:
component: opensearch
replicas: 1
serviceName: opensearch
template:
metadata:
labels:
component: opensearch
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: opensearch
securityContext:
capabilities:
add:
- IPC_LOCK
image: opensearchproject/opensearch
env:
- name: KUBERNETES_CA_CERTIFICATE_FILE
value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "cluster.name"
value: "opensearch-cluster"
- name: "network.host"
value: "0.0.0.0"
- name: "discovery.seed_hosts"
value: "[]"
- name: discovery.type
value: single-node
- name: OPENSEARCH_JAVA_OPTS
value: -Xmx512M -Xms512M
- name: "plugins.security.disabled"
value: "false"
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: os-mount
mountPath: /data
volumes:
- name: os-mount
persistentVolumeClaim:
claimName: nfs-pvc-os-logging
Opensearch svc file
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
Opensearch dashboard deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: open-dash
namespace: logging
spec:
replicas: 1
selector:
matchLabels:
app: open-dash
template:
metadata:
labels:
app: open-dash
spec:
# securityContext:
# runAsUser: 0
containers:
- name: opensearch-dashboard
image: opensearchproject/opensearch-dashboards:latest
ports:
- containerPort: 80
env:
# - name: ELASTICSEARCH_URL
# value: https://opensearch.logging:9200
# - name: "SERVER_HOST"
# value: "localhost"
# - name: "opensearch.hosts"
# value: https://opensearch.logging:9200
- name: OPENSEARCH_HOSTS
value: '["https://opensearch.logging:9200"]'
Opensearch Dashboard svc
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
with the above configuration I am able to get the Dashboard UI open but in Dashboard pod logs I can see a 400 code logs can anyone please try to reproduce this issue, Also I need to integrate the logstash with this stack.
{"type":"response","@timestamp":"2023-02-20T05:05:34Z","tags":[],"pid":1,"method":"head","statusCode":400,"req":{"url":"/app/home","method":"head","headers":{"connection":"Keep-Alive","content-type":"application/json","host":"3.108.199.0:30406","user-agent":"Manticore 0.9.1","accept-encoding":"gzip,deflate","securitytenant":"user"},"remoteAddress":"10.244.1.1","userAgent":"Manticore 0.9.1"},"res":{"statusCode":400,"responseTime":2,"contentLength":9},"message":"HEAD /app/home 400 2ms - 9.0B
When deploying a logstash pod I get an error that
[WARN ] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:[email protected]:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:[email protected]:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}
Can somebody please try to help me with this puzzle
@Benla have made changes as per your recommendation to image now I am getting the following logs in logstash
[2023-02-20T05:18:43,028][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2023-02-20T05:18:43,147][INFO ][org.reflections.Reflections] Reflections took 70 ms to scan 1 urls, producing 127 keys and 444 values [2023-02-20T05:18:43,351][INFO ][logstash.javapipeline ] Pipeline
mainis configured withpipeline.ecs_compatibility: v8setting. All plugins in this pipeline will default toecs_compatibility => v8unless explicitly configured otherwise. [2023-02-20T05:18:43,370][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3bf49916@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run>"} [2023-02-20T05:18:43,811][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.44} [2023-02-20T05:18:43,816][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"} [2023-02-20T05:18:43,821][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2023-02-20T05:18:43,835][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2023-02-20T05:18:43,869][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044
I started getting these endless loop of logs
[WARN ] 2023-02-20 05:13:37.191 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:[email protected]:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:[email protected]:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}
I have not tested this with SSO please let us know suggestions on this
Opensearch, Opensearch Dashboard(Official Helm charts), Filebeat(Official Helm Charts) & Logstash Deployment(Official Helm Charts)
My ELK stack is divided into 3 Parts Each part deploys a separate components using Helm charts
Recomended to create a seperate namespace for logging stack on k8s cluster
Opensearch Deployment(A)
Navigate to "/opensearchhelm/opensearch/helm-charts/charts/opensearch" directory
Inside opensearch directory you will find the values.yaml that contains the configuration for helm charts
Create persistent Volume for Opensearch you can then run the command to deploy the helm chart in the namespace of your choice
helm install opensearch opensearch/ -n namespace
Verify the deployment using below commands
kubectl get pods -n development
Exec into a existing pod that has curl installed to check connectivity between the cluster
kubectl exec -it -n namespace podname /bin/sh
Once inside pod curl the serviceName.namespace as below and that should return a json output in return
curl -X GET https://opensearch.logging:9200 -u 'admin:admin' --insecure
{
"name" : "opensearch-cluster-master-0",
"cluster_name" : "opensearch-cluster",
"cluster_uuid" : "nEFZv3l0RRyx3A0nGtQgug",
"version" : {
"distribution" : "opensearch",
"number" : "2.4.0",
"build_type" : "tar",
"build_hash" : "744ca260b892d119be8164f48d92b8810bd7801c",
"build_date" : "2022-11-15T04:42:29.671309257Z",
"build_snapshot" : false,
"lucene_version" : "9.4.1",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
Once you get a similar output we can confirm that your opensearch is installed successfully
Opensearch Dashboard Deployment(B)
Navigate to "/opensearchhelm/opensearch/helm-charts/charts/opensearch-dashboard" in elk directory.
Inside opensearch directory you will find the values.yaml that contains the configuration for helm charts.
Replace the service name with your opensearch service name as below example to the "opensearchHosts" attribute in values.yaml
opensearchHosts: "https://opensearch-cluster-master:9200"
From opensearchhelm/opensearch/helm-charts/charts/ you can then run the command to deploy the helm chart in the namespace of your choice
helm install opensearchdash opensearch-dashboard/ -n namespace
This will deploy the Opensearch Dashboard that can be used to create dashboard from the Opensearch data
Filebeat Deployment
To get the data from opensearch to logstash we use filebeat, it scrapes the data from Opensearch and gives it to Logstash Filebeat does not use the Persistent Volume claims so we can go ahead with the deployment. Filebeat will automatically pick up the logs from the existing Opensearch in the namespace
*Filebeat configuration and other files for helm chart can be found in filebeat Navigate to directory containing filebeat
helm install filebeat filebeat -n namespace
This will deploy filebeat pods inside your namespace along with other resources.
The number of filebeat pods should be same as the number of your kubernetes Nodes(excluding namespace)
You can validate the deployment by checking logs of the pods
kubectl logs -f filebeat-filebeat-4tkk6 -n namespace
Querying logs should get you an output something like below
{"log.level":"info","@timestamp":"2023-03-22T07:55:12.614Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":104,"throttled":{"ns":506844664,"periods":5}}},"cpuacct":{"total":{"ns":1293495129}},"memory":{"mem":{"usage":{"bytes":150925312}}}},"cpu":{"system":{"ticks":3000,"time":{"ms":60}},"total":{"ticks":30650,"time":{"ms":630},"value":30650},"user":{"ticks":27650,"time":{"ms":570}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":18},"info":{"ephemeral_id":"c4c9fe7b-a469-4a98-a59a-9031a681c0a7","uptime":{"ms":1350147},"version":"8.5.1"},"memstats":{"gc_next":41205704,"memory_alloc":24577840,"memory_total":3591430176,"rss":123326464},"runtime":{"goroutines":90}},"filebeat":{"events":{"active":293,"added":415,"done":122},"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":420,"active":0,"batches":22,"total":420},"read":{"bytes":132},"write":{"bytes":88426}},"pipeline":{"clients":1,"events":{"active":0,"published":415,"total":415},"queue":{"acked":420}}},"registrar":{"states":{"current":454,"update":420},"writes":{"success":18,"total":18}},"system":{"load":{"1":0.06,"15":0.46,"5":0.22,"norm":{"1":0.0038,"15":0.0288,"5":0.0138}}}},"ecs.version":"1.6.0"}}
Logstash deployment
Logstash configuration and other files for helm chart can be found in logstash/
Navigate to logstash
Edit the values.yaml and update the configuration for your opensearch configuration and Index as shown below in example
vim values.yaml
Add the example configuration below
# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
### ***warn*** there is a hardcoded logstash.conf in the image, override it first
logstashPipeline:
logstash.conf: |
input {
beats {
port => 5044
}
}
output {
opensearch{
hosts => "https://opensearch-cluster-master.logging"
user => "admin"
password => "admin"
index => "myindex"
ssl_certificate_verification => false } }
This will create an index that we will be able to query in the Opensearch Dashboard UI
Navigate to logstah directory and install the helm chart Run the helm command to deploy the Logstash tool
helm install logstash logstash/ -n namespace
To validate the working of the stack login to your Opensearch Dashboard
Navigate to dev tools from side menu
Run the below query if you are able to find the ondex you created
GET myindex/_search?pretty
You should get a JSON output of the index if everything is working properly