Search code examples
sqlhivepivotexplodeunpivot

How do i convert column values in a table to row values using hive


eg: current table

employee_id employee_name   location_1  location_2  location_3
111 Reeta   Delhi
112 Pinky   Chennai Kerala  Null
113 Tinku   Noida   Ranchi  Jaipur
114 Teena   Null
115 Bing    Assam   Assam

Desired table :

employee_id employee_name   Location
111 Delhi
112 Pinky   Chennai
112 Pinky   Kerala
113 Tinku   Noida
113 Tinku   Ranchi
113 Tinku   Jaipur
115 Bing    Assam

1.The Null values on location column should be ignored while transforming column to row in target table 2. the employee_id and name with null value as location shouldnt be brought to target table. 3.When emplyee_id and employee_name as duplicate values only one should be brought to target table


Solution

  • A simple option uses union all:

    select employee_id, employee_name, location_1 location from mytable where location_1 is not null
    union all 
    select employee_id, employee_name, location_2 from mytable where location_2 is not null
    union all 
    select employee_id, employee_name, location_3 from mytable where location_3 is not null
    

    A hive-specific approach using a map and lateral view explode might be more efficient:

    select employee_id, employee_name, location
    from (
        select 
            employee_id, 
            employee_name, 
            map("location_1", location_1, "location_2", location_2, "location_3", location_3) as mp
        from mytable
    ) t
    lateral view explode(mp) m as locname, location  
    where location is not null