Search code examples
pythonkubernetespython-requestslogstashlogstash-configuration

What host and port to use for http plugin for a Logstash container running in K8s pod?


I am receiving this error Error: Cannot assign requested address in logstash when attempting to set up a pipeline that uses http input plugin.

I am trying to send data from a Python process to logstash using the Python requests http library. I am not sure which host and port to use in my logstash http input configs. Should I be using the defaults, the logstash pod ClusterIP service IP, the logstash pod IP, or something else?

input {
    http {
        host => "0.0.0.0"
        port => 80
    }
  }

The defaults url is 0.0.0.0:80 but I get a connection error on the Python side. I've also tried the url of the Logstash K8s pod that the logstash container is running in and get Error: Cannot assign requested address in the logstash container.

Edit: included logstash service details

Name:              central-logstash
Namespace:         default
Labels:            app=logstash
                   chart=logstash-1.10.0
                   heritage=Tiller
                   release=central-logstash
Annotations:       <none>
Selector:          app=logstash,release=central-logstash
Type:              ClusterIP
IP:                10.110.133.189
Port:              beats  5044/TCP
TargetPort:        beats/TCP
Endpoints:         192.168.0.79:5044
Session Affinity:  None
Events:            <none>

Solution

  • I ended up switching to TCP to avoid HTTP headers in my messages.

    In my Logstash Helm configuration (https://github.com/helm/charts/tree/master/stable/logstash), I set up the service as such:

    service:
      type: ClusterIP
      annotations: {}
      ports:
         tcp-data:
           port: 1514
           targetPort: tcp-data
           protocol: TCP
         tcp-event:
           port: 1515
           targetPort: tcp-event
           protocol: TCP
    
    ports:
      - name: tcp-data
        containerPort: 1514
        protocol: TCP
      - name: tcp-event
        containerPort: 1515
        protocol: TCP
    

    And the tcp plugin:

    inputs:
     data: |-
        input {
            tcp {
                port => 1514
                type => json
            }
          }
     event: |-
        input {
            tcp {
                port => 1515
                type => json
            }
          }
    

    And then on the Python side, I was able to use the socket library to send messages to Logstash using the Logstash clusterIP service's IP and port 1514 or 1515.