According to the documentation, you should be able to specify a custom pattern for a partition
Hive external tables partitions. However, I can't get it to work:
select * from rawlog_test7 limit 10; returns no records.
This is what I am doing
set hcat.dynamic.partitioning.custom.pattern="${year}/${month}/${day}/${hour}"
I create my table with ...
partitioned by (year int, month int, day int, hour int)
location '/history.eu1/ed_reports/hourly/';
and my directory structure is ../2014/06/18/13/ ...
If I use static partitions
alter table rawlog_test7 add partition (year=2014,month=6,day=18,hour=13) location '/history.eu1/ed_reports/hourly/2014/06/18/13';
it works (select * from rawlog_test7 limit 10; returns records!)
Maybe I can clear some things up about how Hive partitions work:
There are two components to a partition: its directory on the filesystem, and an entry in Hive's metastore. This entry is essentially just the pair (partition values, partition location).
When you create a Hive table, it has no partition entries in the metastore.
When you query Hive, it checks the metastore for partitions it would need to query, and then scans those.
Hive does not automatically detect partitions on the filesystem to add metastore entries.
"Dynamic partitioning" refers to Hive's ability to create partitions both in the filesystem and metastore based on a data column when inserting into a partitioned Hive table, i.e. doing an actual insert into table rawlog_test7 partition(y,m,d,h) ... command.
If you have directories in the filesystem that do not yet have metastore entries, you can either add them one by one, as you have been doing:
alter table rawlog_test7 add partition (year=2014,month=6,day=18,hour=13) location '/history.eu1/ed_reports/hourly/2014/06/18/13';
Or you can run a table repair:
msck repair table rawlog_test7;
Although I have not tested the latter with a custom partitioning pattern.