Search code examples
clojureclojure-contrib

reading large command output with clojure

I'm using the sh function from the clojure.java.shell command to read the very large output of a command. The output is around 60meg of data.

I keep getting java.lang.OutOfMemoryError. Is there a way to open a sort of pipe that would let me read the output and parse it into a vector . Like a lazy sequence for the output of the command?

Basically the data is a large array of byte that I want to convert to just numbers and put into a vector.

Solution

  • clojure.java.shell/sh will always return a non-lazy string

    A solution (doesn't handle closing, environment passing and encoding) using lazy line-seq on an BufferedReader:

    (->> (.exec (Runtime/getRuntime) "YOUR_LONG_RUNNING_COMMAND ARG ...")
    .getInputStream
    clojure.java.io/reader
    line-seq
    (map YOUR-FUNCTION))