I have facing this issue with hive. When i Query a table ,which is partitioned on date column, SELECT count(*) from table_name where date='2018-06-01' the query reads the entire table data and keeps for running hours, Using EXPLAIN I found that HIVE is not applying the PartitionFilter on the query I have double checked that the table is partitioned on date column by desc table_name.
Execution engine is Spark And Data is stored in Azure Data lake in Parquet Format
However I have another table in the Database for which the PartitionFilter is applied and it executes as expected.
Can there be some issue with the hive metadata or it is something else
Found the cause for this issue, Hive wasn't applying the Partition Filters on some table ,because those tables were cached. Thus when i restared the thrift server the cache was removed and Partition Filters were applied