So, I have a DataFrame that contains a mix of both Categorical and Numerical values that is currently 12345 rows by 171 columns.
I have missing values in both the Categorical variable and the Numerical where I would like to impute the values. For the Numerical columns I am doing the following;
import pandas as pd
import numpy as np
data = pd.read_csv('filepath')
from sklearn.preprocessing import Imputer
imp = Imputer(missing_values=np.nan, strategy='mean', axis=0)
data = imp.fit_transform(data)
I then get the following error
ValueError: could not convert string to float: 'USD'
This I understand is because I am using sci-kit learns imputer with strategy='mean' which is not compatible with Categorical variables. I would rather not have to go through each column and manually pull out the Numerical values so I am looking for a way that I can perform this imputation only on the Numerical columns.
If you use panda's categorical encoding functionality, this can all be handled pretty simply. I however rarely find myself with properly encoded data, and would rather have a robust solution than rely on pure pandas.
Here's what I would do.
categorical_columns = []
numeric_columns = []
for c in data.columns:
if data[c].map(type).eq(str).any(): #check if there are any strings in column
categorical_columns.append(c)
else:
numeric_columns.append(c)
#create two DataFrames, one for each data type
data_numeric = data[numeric_columns]
data_categorical = pd.DataFrame(data[categorical_columns])
from sklearn.preprocessing import Imputer
imp = Imputer(missing_values=np.nan, strategy='mean', axis=0)
data_numeric = pd.DataFrame(imp.fit_transform(data_numeric), columns = data_numeric.columns, index=data_numeric.index) #only apply imputer to numeric columns
#you could do something like one-hot-encoding of data_categorical here
#join the two masked dataframes back together
data_joined = pd.concat([data_numeric, data_categorical], axis = 1)