Search code examples
hadoophadoop2hue

Having trouble to import new data in an existing table using Hue Hadoop

When I'm loading new data in an existing table and then do select count(1) to get the total rows number loaded, I'm only getting the count of one HDFS file.

enter image description here

The rows number only represents the amount of one HDFS file.

To import the "new data" I'm clicking here: enter image description here

Also, here the total count in MySQL: enter image description here

And the total count in Hue Hadoop: enter image description here

By the way, here the file browser: enter image description here

Do you have any idea what I'm doing wrong?

Solution

  • Try this: 

    invalidate metadata default.movie;

    Most probably you were using Impala as engine to retrieve the data and this command is to reload the metadata.

    By default, the cached metadata for all tables is flushed. If you specify a table name, only the metadata for that one table is flushed. Even for a single table, INVALIDATE METADATA is more expensive than REFRESH, so prefer REFRESH in the common case where you add new data files for an existing table.

    If you want to go further, check this out.