Search code examples
elasticsearchlogstashelastic-stacklogstash-configuration

Logstash MalformedCSVError: Illegal quoting in line | how to configure logstash conf to read multi column mutiline CSV?

Already went through CSV::MalformedCSVError: Illegal quoting in line 1 with SmarterCSV and Rescue CSV::MalformedCsvError: Illegal quoting in line n None of them addresses my problem.

I'm facing problem while basic integration of CSV conf with Log Stash:

    input {
      file {
        path => "/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
      }
      date {
        match => ["creation_date", "dd-MM-yy"]
        target => "creation_date"
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

Error Stack prints following line on loop:


    [2020-03-02T13:36:51,968][WARN ][logstash.filters.csv     ][another_test] Error parsing csv {:field=>"message", :source=>"{\"level\":\"DEBUG\",\"timeUTC\":\"Mon Feb 24 08:05:20 UTC 2020\",\"timeUnix\":1582531520800,\"appID\":\"TEST\",\"hub\":\"CI\",\"environment\":\"NONE\",\"host\":\"Yashpal3150153\",\"logMessage\":\"http-outgoing-1 << Content-Type: application/x-amz-json-1.0\",\"threadID\":\"SimpleAsyncTaskExecutor-1\",\"cdll_version\":\"6.5.0\"}\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

What am I Missing, what's wrong here?

Update:

CSV:

    125,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid3,100,TRUE
    124,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid2,100,TRUE
    123,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid1,252,TRUE

Update: Pipeline:

     - pipeline.id: another_test
       queue.type: persisted
       path.config: "/softwares/logstash-7.6.0/config/*.conf"

config contains: enter image description here My conf I mentioned is: cust.conf

Also, I updated the cust.conf to:

    input {
      file {
        path => "D:/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
        codec => plain
        convert => {
          "imiles_wallet_id" => "integer"
          "is_active" => "boolean"
          "creation_date" => "date"
          "gcid_cod" => "string"
          "total_imiles" => "integer"
        } 
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

Now the Error is:

    LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`
              create at org/logstash/execution/ConvergeResultExt.java:109
                 add at org/logstash/execution/ConvergeResultExt.java:37
      converge_state at D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339
    [2020-03-04T18:20:05,345][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    [2020-03-04T18:20:05,397][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    ```
Solution

  • By starting Logstash with a pipeline definition /softwares/logstash-7.6.0/config/*.conf you probably have many configuration files that interfere with each other (i.e. several different inputs, filters and outputs). All those configuration files are merged into a single configuration.

    Read this article in order to learn how to best organize your Logstash pipelines. In the meantime, you can simply start your Logstash from the command-line like this:

    bin/logstash -f /softwares/logstash-7.6.0/config/cust.conf