Search code examples
sqlapache-sparkignite

Spark SQL with Ignite

I am trying to speed spark sql queries by introduce ignite as cache layer, by using IgniteRDD. From the example by ignite doc, it loads data from ignite cache to construct the RDD. But in our usecase the data size may too big to put into ignite memory, actually we just put the data in hbase, so is it possible to do: 1, construct igniteRDD by loading data from hbase 2, Just use ignite to cache share rdd which is generated by spark sql to speed up spark sql.

Solution

  • Apart from above three approaches, if you have flexibility to add another component, use Apache Phoenix. It supports integration with Spark SQL. You can check it on their official website. In this case you will not need Apache Ignite.