Search code examples
elasticsearchkibana

Elasticsearch cut strings with forward slash

I have configured my squid proxy to send the logs to Elasticsearch 7.17.6, everything is working fine, except the USER mapping.

The string expect is DOMAIN/USER, the ELK recieves it correctly on original message but then on mapping I just see DOMIN, /USER is missed.

kibana_dashboard

The field type is configured as keyword in mapping and index patterns.

filebeat pipeline.json

{
 "description": "Pipeline for parsing squid elasticsearch.log",
          "processors": [{
              "grok": {
                  "field": "message",
                  "patterns":[
                      "%{POSINT:squid.proxy.request_time}.%{POSINT:squid.proxy.request_time_ms}\\|.*?%{NUMBER:squid.proxy.response_time}\\|%{IP:squid.proxy.src_ip}\\|(%{MAC:squid.proxy.mac}|-)\\|%{DATA:squid.proxy.request_status}\\|%{DATA:squid.proxy.status_code}\\|%{NUMBER:squid.proxy.http_size}\\|%{WORD:squid.proxy.http_method}\\|(%{IP:squid.proxy.server_ip}|%{HOSTNAME:squid.proxy.website})\\|(%{USER:squid.proxy.user})|(%{USER:{WORD}/{WORD}}|-)\\|%{WORD:squid.proxy.hierarchy}\\|(%{IP:squid.proxy.server_ip}|-)\\|(%{DATA:squid.proxy.ct}|-)\\|(%{NUMBER:squid.proxy.total_time}|-)\\|%{UUID:squid.proxy.uuid}\\|(%{HOSTNAME:squid.proxy.website}|-)\\|(%{DATA:squid.proxy.useragent}|-)\\|\\|\\|.*?(category:.*?cinfo:%{NUMBER:squid.proxy.category_id}-%{WORD:squid.proxy.category_name};|-).*"
                  ],
                  "ignore_missing": false,
                  "ignore_failure": false
              }
          },{
              "rename":{
                  "field": "message",
                  "target_field": "squid.proxy.original_message"
              }
          },

              {
                  "date": {
                      "field": "squid.proxy.request_time",
                      "formats": ["UNIX", "dd MMM H:m:s"],
                      "ignore_failure": true
                  }
              },
               {
                  "geoip": {
                      "field": "squid.proxy.server_ip",
                      "target_field": "squid.proxy.geo",
                      "ignore_missing": true
                  }
              },

              {
              "remove": {
                  "field": "message",
                  "ignore_missing": true,
                  "ignore_failure": true
              }
          }],
          "on_failure" : [{
              "drop" : {
                  "ignore_failure" : true
              }
          }]
      }

Anyone can help me to solve this issue.

Thanks in advance

Best regards

Solution

  • Problem fixed

    In grok i must use

    (%{DATA:squid.proxy.user}|-)

    instead of

    (%{USER:{DATA}|-)