Search code examples
apache-sparkhana

SAP Spark Controller not caching data

I've got Spark Controller 2.0.0 running on a HDP 2.4.3 with Spark 1.6.2

In the configuration I have these parameters configured:

sap.hana.es.enable.cache=true
sap.hana.es.cache.max.capacity=500
sap.hana.hadoop.datastore=Hive

I've got HANA 1.00.122 connected to that Spark Controller, set enable_remote_cache parameter to true in indexserver.ini, and imported one of exposed Hive tables as a virtual table in HANA.

Then I ran select-statements against that virtual table, but every time I see that no cache is created (nothing in the Storage tab of Spark UI), nor it is hit (query runtime doesn't drop, and I see the job going through the same stages every time).

Using the hint "with hint (USE_REMOTE_CACHE)" doesn't help either.

Are there any other settings I forgot to make?

Solution

  • Accordingly to SAP, the HANA version for caching to work should be 2.0+.