Expand/Unnest Polars struct into rows, not into columns

Question

I have this code

import polars as pl

df = pl.DataFrame({
    'as_of':    ['2024-08-01', '2024-08-02', '2024-08-03', '2024-08-04'],
    'quantity': [{'A': 10, 'B': 5}, {'A': 11, 'B': 7}, {'A': 9, 'B': 4, 'C': -3},
                 {'A': 15, 'B': 3, 'C': -14, 'D': 50}]
}, schema={'as_of': pl.String, 'quantity': pl.Struct})

df.unnest('quantity')

which gives

shape: (4, 5)
┌────────────┬─────┬─────┬──────┬──────┐
│ as_of      ┆ A   ┆ B   ┆ C    ┆ D    │
│ ---        ┆ --- ┆ --- ┆ ---  ┆ ---  │
│ str        ┆ i64 ┆ i64 ┆ i64  ┆ i64  │
╞════════════╪═════╪═════╪══════╪══════╡
│ 2024-08-01 ┆ 10  ┆ 5   ┆ null ┆ null │
│ 2024-08-02 ┆ 11  ┆ 7   ┆ null ┆ null │
│ 2024-08-03 ┆ 9   ┆ 4   ┆ -3   ┆ null │
│ 2024-08-04 ┆ 15  ┆ 3   ┆ -14  ┆ 50   │
└────────────┴─────┴─────┴──────┴──────┘

Instead of each unnesting into columns, can I unnest into rows to get a dataframe like so:

shape: (11, 3)
┌────────────┬──────┬──────────┐
│ as_of      ┆ name ┆ quantity │
│ ---        ┆ ---  ┆ ---      │
│ str        ┆ str  ┆ i64      │
╞════════════╪══════╪══════════╡
│ 2024-08-01 ┆ A    ┆ 10       │
│ 2024-08-01 ┆ B    ┆ 5        │
│ 2024-08-02 ┆ A    ┆ 11       │
│ 2024-08-02 ┆ B    ┆ 7        │
│ 2024-08-03 ┆ A    ┆ 9        │
│ …          ┆ …    ┆ …        │
│ 2024-08-03 ┆ C    ┆ -3       │
│ 2024-08-04 ┆ A    ┆ 15       │
│ 2024-08-04 ┆ B    ┆ 3        │
│ 2024-08-04 ┆ C    ┆ -14      │
│ 2024-08-04 ┆ D    ┆ 50       │
└────────────┴──────┴──────────┘

?

Answer 1

You can't do it in one step but what you're after is an unpivot (used to be melt).

(
    df
    .unnest('quantity')
    .unpivot(
        index='as_of', 
        variable_name='name',
        value_name='quantity'
        )
    .filter(pl.col('quantity').is_not_null())
    .sort('as_of')
    )
shape: (11, 3)
┌────────────┬──────┬──────────┐
│ as_of      ┆ name ┆ quantity │
│ ---        ┆ ---  ┆ ---      │
│ str        ┆ str  ┆ i64      │
╞════════════╪══════╪══════════╡
│ 2024-08-01 ┆ A    ┆ 10       │
│ 2024-08-01 ┆ B    ┆ 5        │
│ 2024-08-02 ┆ A    ┆ 11       │
│ 2024-08-02 ┆ B    ┆ 7        │
│ 2024-08-03 ┆ A    ┆ 9        │
│ …          ┆ …    ┆ …        │
│ 2024-08-03 ┆ C    ┆ -3       │
│ 2024-08-04 ┆ A    ┆ 15       │
│ 2024-08-04 ┆ B    ┆ 3        │
│ 2024-08-04 ┆ C    ┆ -14      │
│ 2024-08-04 ┆ D    ┆ 50       │
└────────────┴──────┴──────────┘