Search code examples
jdbcclojure

Clojure jdbc - query single column flattened result


I'm trying to read data (cca 760k rows) from a single column into one (flattened) vector. Result of clojure.java.jdbc/query is seq of maps, e.g. ({:key "a"} {:key "b"} ...). With option :as-arrays? true provided, [[:key] ["a"] ["b"] ...] is returned. To flatten the result, I've also used option :row-fn first and got [:key "a" "b" ...]. Finally, I've applied rest to get rid of the :key.

Wrapping and unwrapping of rows with vectors seems like a lot of unnecessary work. I'm also not happy with performance. Is there a faster / more idiomatic way? I've tried...

(jdbc/with-db-connection [con -db-spec-]
  (with-open [^Statement stmt (.createStatement (:connection con))
              ^ResultSet res  (.executeQuery stmt query)]
    (let [ret (ArrayList.)]
      (while (.next res)
        (.add ret (.getString res 1)))
      (into [] ret))))

... but it's not much faster, and it's ugly.


EDIT

Nicer way to do it is via transducers (see here):

(into []
      (map :key)
      (jdbc/reducible-query
       connection
       ["SELECT key FROM tbl"]
       {:raw? true}))

Solution

  • You can just use :row-fn :key. Not sure what performance you are expecting but on my i5 PC, retrieving 760K records took ~3 seconds (H2 file based database)

    (time
     (count
      (jdbc/query db ["select top 760000 key from table1"] {:row-fn :key})))
    ;; => 760000
    
    "Elapsed time: 3003.456295 msecs"