Search code examples
cachingapache-sparkapache-spark-sqllru

Why SparkUI doesn't show memory usage for SparkSQL LRU cache?

When I run sql query spark-sql will use LRU cache. Why LRU cache usage doesn't reflects in spark's WebUI? According responce time I feel my queries cached, but "Memory Used" says "0.0 B / 707.0 MB" Spark version is 1.3.1

Solution

  • Spark does show the caching status.
    It's available via Spark application UI on the "Storage" tab.
    It will show the storage-level (cache type), number of cached partitions, size in memory & size on disk.

    You didn't specify how you use Spark caching mechanism.
    Spark caching has to be enabled explicitly.
    You can check here how to enable/disable caching for Spark tables

    You can also enable the cache for RDDs/DataFrams using: 

    rdd.cache() / df.cache() 
rdd.persist(StorageLevel newLevel) / df.persist(StorageLevel newLevel)