I'm trying to read data (cca 760k rows) from a single column into one (flattened) vector. Result of clojure.java.jdbc/query
is seq of maps, e.g. ({:key "a"} {:key "b"} ...)
. With option :as-arrays? true
provided, [[:key] ["a"] ["b"] ...]
is returned. To flatten the result, I've also used option :row-fn first
and got [:key "a" "b" ...]
. Finally, I've applied rest
to get rid of the :key
.
Wrapping and unwrapping of rows with vectors seems like a lot of unnecessary work. I'm also not happy with performance. Is there a faster / more idiomatic way? I've tried...
(jdbc/with-db-connection [con -db-spec-]
(with-open [^Statement stmt (.createStatement (:connection con))
^ResultSet res (.executeQuery stmt query)]
(let [ret (ArrayList.)]
(while (.next res)
(.add ret (.getString res 1)))
(into [] ret))))
... but it's not much faster, and it's ugly.
EDIT
Nicer way to do it is via transducers (see here):
(into []
(map :key)
(jdbc/reducible-query
connection
["SELECT key FROM tbl"]
{:raw? true}))
You can just use :row-fn :key
. Not sure what performance you are expecting but on my i5 PC, retrieving 760K records took ~3 seconds (H2 file based database)
(time
(count
(jdbc/query db ["select top 760000 key from table1"] {:row-fn :key})))
;; => 760000
"Elapsed time: 3003.456295 msecs"