Search code examples
hadoophivelimit

Any column limits in hive

I have created some tables with more than 800 columns. But I see only the values of approximately 500 columns. Is there any limit or what other causes are there?

Solution

  • According to this answer:

    I am not aware of any "hard" limitation in hive in regards to column count, there are some on column size though. This being said a restriction on column count would also probably depend on the file format, ORC having indexes and predicate pushdown does not behave as a Text file would. ORC has configurations for number of rows that are grouped together for an index.

    In Hive issue: for example the number of columns > 1K created memory pressure in ORC resulting in OOM. In test 15K columns were loaded and saw OOM only at 20K columns.

    And this answer:

    The scalability of metastore(jetty) + database chosen will also impact the performance of queries and may restrict success rate at every query execution , with increase in number of columns per table