Search code examples
clojureapache-sparkspark-streamingnupic

will Spark support Clojure?

I am about to start learning functional programming and Clojure appeals to me the most, I love its community, syntax and concept of immutable data structures. I am also interested in bio inspired ML for rich data Numenta. However, my huge concern is that Spark does not support it as yet, and Spark rocks!!! There is a Flambo Flambo Clojure ,but is it a satisfactory solution?

My course and job is in Scala. Should I defeat and enter Scala world or do you think that I should focus solely on Clojure?

Solution

  • Being the author or Sparkling (thanks to Josh Rosen for pointing that out), I can tell you that we use it at our company for ETL processing.

    Here's what's good:

    • it provides a Clojuristic way of interacting with Spark, and as you can see in the presentation "Big Data with style - the Clojure/Spark way"
    • it's optimized for performance
    • it's used in production and there are others also using it

    Here's what's missing:

    • There's currently no support for Spark Streaming, Spark SQL, Spark Dataframes or Spark ML. That might come in the future, I'm happy to accept pull requests, but it's currently not main focus (at the time of writing, April 2015).

    I hope this helps you make up your mind on going with Clojure or starting to learn Scala.