Processing large json's using logstash - Not working and not printing any error

Question

I started using logstash (on windows) when my main cause of use will be passing logstash a large json (10 mb), filtering the json somehow and write it out to elastic search.

As for now, I don't really care about the json filtering (I will care after I'll get this to work). I wan't the file to pass through logstash and get to my elastic search.

The client who feeds logstash uses a tcp connection.

My logstash simple configuration file looks like:

input
{
    tcp
    {
        port=>7788
        codec=>"json"
    }
}

output
{
    elasticsearch
    {
        hosts=>"localhost:9200"
        codec=>"json"
    }
    stdout
    {
        codec=>rubydebug
    }
}

This does work for me on small json inputs like:

{"foo":"bar", "bar": "foo"}

I see the logstash working and passing the data to elastic search and everything's ok.

Also, when using the default codec ("text") it worked, but not as expected.

My problem starts when the inputs are large jsons.

Assuming I have a 10 mb json - what do I need to do with it so logstash will be able to handle it over tcp as a json? Should the file be indented or not? What encoding should I use before I convert it into bytes? What codec\settings should my logstash have?

BTW, when I use curl and through the large json directly to elastic search - it works - So there are no problems with the json.

Is there any way I can get some better tracing or at least know why I fail?

Answer 1

I found out that the problem wasn't the length but the lack of a newline - So all I needed to do was to add a newline to my log files.

BTW, there is no 4K length limit - At least not when working with TCP.