Weighted average grouped by country and year

Question

I have a code which calls several apis from the UN comtrade database iterating through years. It then appends these together into a single dataset of all reporters and years from 1991 to 2018. I am trying to then get the average of the trade value weighted by the net weight for each country each year.

import requests
import pandas as pd
import json
import numpy as np

base = "http://comtrade.un.org/api/get?"
maxrec = "50000"
item = "C"
freq = "A"
px="H0"
ps="all"
r="all"
p="0"
rg="2"
cc="AG2"
fmt="json"

comtrade = pd.DataFrame(columns=['Year', 'Reporter Code', 'Reporter', 'Commodity Code', 'Commodity', 'Trade Value (US$)'])

for year in range(1991,2018):
    print(r)
    ps="{}".format(year)
    url = base + "max=" + maxrec + "&" "type=" + item + "&" + "freq=" + freq + "&" + "px=" +px + "&" + "ps=" + str(ps) + "&" + "r="+ str(r) + "&" + "p=" + p + "&" + "rg=" +rg + "&" + "cc=" + cc + "&" + "fmt=" + fmt
#    print(url)
    t = requests.get(url)
    x = t.json()
    new = pd.DataFrame(x["dataset"])
    comtrade = comtrade.append(new)
group = comtrade.groupby([ 'rtCode', 'yr'])
finalvalue = group.apply(lambda x: np.average(x['TradeValue'], 
weights=x['NetWeight']))

This may be obvious but here is where the problem seems to be coming here:

group = comtrade.groupby([ 'rtCode', 'yr'])
finalvalue = group.apply(lambda x: np.average(x['TradeValue'], weights=x['NetWeight']))

where I get an error of:

TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'

Answer 1

As you can see in the below description at this example URL from your scraping, null is a main feature of the data which is being entered in your data frame as "None".

You can change your None type using "fillna" from Pandas. I used 0 in this example which could work with your groupby, or you could do "nan" since "NA groups in GroupBy are automatically excluded," per their docs.

from numpy import nan
comtrade =  comtrade.fillna(value=0, inplace=True)