Search code examples
xmlpipelinebenthos

Benthos pipleline to read XML from postman and publish to kafka topic


i want to create a pipeline to read XML data from postman http url and consume it through benthos input configuration and publish this message to a kafka topic using benthos processor . Following is the configuration , i was trying but doesn't seem working

input:
  http_server:      
      address: ""      
      path: /
      ws_path: /ws
      allowed_verbs:        
        - POST
      timeout: 5s
      rate_limit: ""
pipeline:
  processors:
    - xml: {}
output:
  kafka:
    addresses:
      - abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
    topic: abc_abc-central-1.abc-tyur-service-in.v1
    client_id: abc-tyur-service-dev-1
    tls:
      enabled: true
      root_cas_file: /Users/ca.crt
      client_certs:
        - cert_file: /Users/cert.pem
          key_file: /Users/key.pem

logger:
  level: ALL

Solution

  • When we don't want to modify our input messages and want to publish unmodified messages to the output , processor can be removed and only input and output configuration are sufficient. Removing processor solved my problem . Following the working configuration

    input:
      http_server:      
          address: ""      
          path: /
          ws_path: /ws
          allowed_verbs:        
            - POST
          timeout: 5s
          rate_limit: ""
    output:
      kafka:
        addresses:
          - abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
          - abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
          - abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
        topic: abc_abc-central-1.abc-tyur-service-in.v1
        client_id: abc-tyur-service-dev-1
        tls:
          enabled: true
          root_cas_file: /Users/ca.crt
          client_certs:
            - cert_file: /Users/cert.pem
              key_file: /Users/key.pem
    
    logger:
      level: ALL
    

    Thanks to benthos community on Discord