Search code examples
hadoopapache-sparkhiveapache-spark-sqlhivecontext

Repairing hive table using hiveContext in java


I want to repair the hive table for any newly added/deleted partitions.Instead of manually running msck repair command in hive,is there any way to achieve this in java?I am trying to get all partitions from hdfs and from hive metastore and then after comparing them will put newly added/deleted partitions in hive metastore.But i am not able to get the api from hivecontext.I have tried to get all the partitions using hivecontext,but it is throwing error table not found.

System.out.println(hiveContext.metadataHive().getTable("anshu","mytable").getAllPartitions());

Is there any way to add/remove partitions in hive using java?


Solution

  • Spark Option :

    using hivecontext you can execute this like below example. no need to do it manually

    sqlContext = HiveContext(sc)
    sqlContext.sql("MSCK REPAIR TABLE your table")
    

    Is there any way to add/remove partitions in hive using java?

    Plain java option :

    If you want to do it in plain java way with out using spark, with plain java code then You can use class HiveMetaStoreClient to query directly from HiveMetaStore.

    enter image description here enter image description here

    Please see my answer here with example usage