Search code examples
pythonpandasgoogle-bigquerypivottranspose

How to do pivot operation on multiple columns on python pandas or big query. Preferably on big query


original data:

enter image description here

how the data would look like after required transformation:

enter image description here

I have tried melt function in python pandas, but I am only able to pivot on one column. I am sure I must be missing something.


Solution

  • Below is for BigQuery Standard SQL

    execute immediate (
    with types as (
      select 
        array_to_string(types, ',') values_list,
        regexp_replace(array_to_string(types, ','), r'([^,]+)', r'"\1"') columns_list
      from (
        select regexp_extract_all(to_json_string(t), r'"([^""]+)":') types
        from (
          select * except(Country, Branch, Category)
          from `project.dataset.your_table` limit 1
        ) t
      )
    ), categories as (
      select distinct Category  
      from `project.dataset.your_table`
    )
    select '''
    select Country, Branch, Output, ''' || 
      (select string_agg(''' 
      max(if(Category = "''' || Category || '''", val, null)) as ''' || Category ) 
      from categories)
    || ''' 
    from (
      select Country, Branch, Category, 
        type[offset(offset)] Output, val 
      from `project.dataset.your_table` t,
      unnest([''' || values_list || ''']) val with offset,
      unnest([struct([''' || columns_list || '''] as type)])
    )
    group by Country, Branch, Output
    '''
    from types
    );   
    

    if applied to sample data in your question - output is

    enter image description here