How do I concat dataframes with a for loop in Python & Pandas

Question

I have a dictionary of dataframes (248 countries) that I want to concat into one dataframe.

the data frame is called dfs, so if I want to access the contents for Albania I use:

dfs["Albania"]

I used the below code to do this with 4 dataframes earlier while learning how to merge dataframes.

Can I adapt this to work as a loop with the 248 countries that I want to include now, and also set the key for each concatenated df as the country name?

I have made very little progress on this in the last few hours!

datasets = [df_ireland, df_italy, df_france, df_germany]

frames = []

for frame in datasets:
    frames.append(frame)

df_join = pd.concat(frames, keys=['Ireland', 'Italy', 'France', 'Germany'])

Here is the loop I used to build the dictionary in case that is of any benefit:

# Import the libraries
import requests
import requests_cache

import json

import pandas as pd
import numpy as np
from pandas import Series, DataFrame, json_normalize

from datetime import datetime

# Make an API call and store the response.
sum_url = 'https://api.covid19api.com/summary'
sum_data = requests.get(sum_url)

# Store the API response in a variable.
available_sum_data = sum_data.json()
sum_df = json_normalize(available_sum_data["Countries"])

# Make a list of countries
countries = sum_df['Country'].tolist()

# Make a empty dictionary to hold dataframes
dfs = {}


for country in countries:
    print(country)

    try:
        # check the cache and if old data call api
        requests_cache.install_cache(f'{country} cache', expire_after=21600)
        url = f'https://api.covid19api.com/total/dayone/country/{country}'
        data = requests.get(url)

        # test if cache used
        print(data.from_cache)

    except requests.exceptions.RequestException as e:  # This is the correct syntax
        print(e)
        print('cant print' + country)

    try:
        available_data = data.json()
        dfs[f'{country}'] = pd.json_normalize(available_data)


        # Create Daily new cases column & SMA
        dfs[f'{country}']["New Cases"] = dfs[f'{country}']['Confirmed'].diff()
        dfs[f'{country}']["SMA_10 New Cases"] = dfs[f'{country}']["New Cases"].rolling(window=10).mean()

        # Create Daily new deaths column & SMA
        dfs[f'{country}']["New Deaths"] = dfs[f'{country}']['Deaths'].diff()
        dfs[f'{country}']["SMA_10 New Deaths"] = dfs[f'{country}']["New Deaths"].rolling(window=10).mean()


    except:
        print('cant format to json: ' + country)

Answer 1

I think you already have the great dictionary dfs so you don't need to do the loop. Could you please try this?

df_joined = pd.concat(dfs.values(), keys=dfs.keys())