Search code examples
javaspringspring-batchflat-file

Spring Batch: How to process multi-line log files

I am trying to import the contents of a log file into a database using Spring Batch.

I am currently using a FlatFileItemReader, but there are unfortunately many log entries that doesn't catch. The two main problems are:

  1. Lines that contain multi-line JSON Strings:

    2012-03-22 11:47:35,307  DEBUG main someMethod(SomeClass.java:56): Do Something(18,true,null,null,null): my.json = '{
    "Foo":"FooValue",
    "Bar":"BarValue",
    ... etc
}'

  2. Lines that contain stack traces

    2012-03-22 11:47:50,596  ERROR main com.meetup.memcached.SockIOPool.createSocket(SockIOPool.java:859): No route to host
java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:364)
        at sun.nio.ch.Net.connect(Net.java:356)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
        at com.meetup.memcached.SockIOPool$SockIO.getSocket(SockIOPool.java:1703)
        at com.meetup.memcached.SockIOPool$SockIO.<init>(SockIOPool.java:1674)
        at com.meetup.memcached.SockIOPool.createSocket(SockIOPool.java:850)
        at com.meetup.memcached.SockIOPool.populateBuckets(SockIOPool.java:737)
        at com.meetup.memcached.SockIOPool.initialize(SockIOPool.java:695)

Basically, I need the FlatFileItemReader to keep reading until it reaches the next timestamp, while aggregating all the lines before that. Has any such thing been done before (in Spring Batch)

Solution

  • There's now an FAQ in the Spring Batch documentation addressing this use case.