pythonpandasseaborntrend
Plot multindex series with trend line
Sample data for MWE:
d = [{'Date': Timestamp('2022-08-02 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-14 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-18 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-01-19 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-20 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-21 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-01-22 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-01-23 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-01-24 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-01-25 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-01-26 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-01-27 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-29 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-30 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-01-31 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-03 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-04 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-05 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-06 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-02-07 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-20 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-21 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-22 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-23 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-24 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-02-25 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-02-26 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-27 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-02-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-02 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-03 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-04 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-05 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-06 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-07 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-08 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-09 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-10 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-11 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-12 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-13 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-14 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-15 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-16 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-18 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-19 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-14 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-01-18 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-01-19 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-20 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-01-21 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-26 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-27 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-28 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-03-29 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-03-30 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-03-31 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-03 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-04 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-05 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-06 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-07 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-08 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-09 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-10 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-11 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-12 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-13 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-14 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-15 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-16 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-18 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-19 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-20 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-21 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-22 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-23 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-24 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-25 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-04-26 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-27 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-28 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-04-29 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-04-30 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-05-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-03 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-04 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-05 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-06 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-07 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-08 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-09 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-10 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-11 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-12 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-13 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-14 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-15 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-16 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-05-17 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-18 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-19 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-20 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-05-21 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-05-22 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-05-23 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-24 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-25 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-26 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-27 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-05-29 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-30 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-05-31 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-01 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-03 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-04 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-05 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-06 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-07 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-08 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-09 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-10 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-11 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-12 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-13 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-14 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-15 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-16 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-18 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-19 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-20 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-21 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-22 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-23 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-24 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-25 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-26 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-27 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-06-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-06-29 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-06-30 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-02 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-03 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-04 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-05 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-06 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-07 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-08 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-09 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-10 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-11 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-12 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-13 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-14 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-15 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-16 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-16 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-16 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-16 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-17 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-28 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-07-29 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-30 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-07-31 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-01 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-02 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-03 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-04 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-05 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-06 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-07 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-08 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-09 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-10 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-11 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-12 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-13 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-14 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-15 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-16 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-17 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-18 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-19 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-20 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-21 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-22 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-23 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-24 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-25 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-08-26 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-27 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-29 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-08-30 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-08-31 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-01 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-02 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-03 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-04 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-05 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-06 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-07 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-08 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-09 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-10 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-11 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-12 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-13 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-14 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-15 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-16 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-17 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-18 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-19 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-20 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-21 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-22 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-23 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-24 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-25 00:00:00'), 'A': 'Yes'},
{'Date': Timestamp('2022-09-26 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-27 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-28 00:00:00'), 'A': 'Unknown'},
{'Date': Timestamp('2022-09-29 00:00:00'), 'A': 'No'},
{'Date': Timestamp('2022-09-30 00:00:00'), 'A': 'No'}]
df = pd.DataFrame(d)
I get a series representing the relative frequency where A=="Yes" in any given year/month (multi-index) like so (and per this answer):
df_counts = (
df
.assign(Year=df["Date"].dt.year, Month=df["Date"].dt.month)
.pivot_table(index=["Year", "Month"], columns="A", aggfunc="count", fill_value=0)
.droplevel(0, axis=1)
)
result = df_counts["Yes"].div(df_counts.sum(axis=1))
This looks like:
Year Month
2022 1 0.277778
2 0.125000
3 0.320000
4 0.233333
5 0.161290
6 0.466667
7 0.387097
8 0.272727
9 0.281250
Name: Yes, dtype: float64
I can also convert this back to a dataframe, if needed, like so:
df2 = result.reset_index().rename(columns={0: "Freq"})
Making a line plot of this is straightforward:
results.plot(kind="bar", legend=False, figsize=(14, 8))
However, I'd like to show a trend line to better illustrate the upward trend in the data. Because this is multi-index data, I have been struggling to get seaborn/matplotlib/pandas to do this.
How do I plot this data with a trend line?
Solution
As Trenton noted in the comments in your question, first simplify the data to have a single index:
df_counts = (
df
.assign(YM=df["Date"].dt.strftime("%Y-%m"))
.pivot_table(index=["YM"], columns="A", aggfunc="count", fill_value=0)
.droplevel(0, axis=1)
)
result = df_counts["Yes"].div(df_counts.sum(axis=1))
Then convert that into a dataframe:
df_result = result.reset_index(name="freq")
One option then for showing the trend line is using seaborn like so:
ax = sns.barplot(data=df_result, x="YM", y="freq", color="lightblue", zorder=0)
ax = sns.regplot(x=df_result.index, y=df_result["freq"], marker='+')
sns.despine(offset=10, trim=False)
ax.set_ylabel("Freq")
ax.set_xlabel("Year-Month")
plt.show()
This should create this for the sample data:
