AttributeError when using ColumnTransformer into a pipeline

Question

This is my first machine learning project and the first time that I use ColumnTransformer. My aim is to perform two steps of data preprocessing, and use ColumnTransformer for each of them.

In the first step, I want to replace the missing values in my dataframe with the string 'missing_value' for some features, and the most frequent value for the remaining features. Therefore, I combine these two operations using ColumnTransformer and passing to it the corresponding columns of my dataframe.

In the second step, I want to use the just preprocessed data and apply OrdinalEncoder or OneHotEncoder depending on the features. For that I use again ColumnTransformer.

I then combine the two steps into a single pipeline.

I am using the Kaggle Houses Price dataset, I have scikit-learn version 0.20 and this is a simplified version of my code:

cat_columns_fill_miss = ['PoolQC', 'Alley']
cat_columns_fill_freq = ['Street', 'MSZoning', 'LandContour']
cat_columns_ord = ['Street', 'Alley', 'PoolQC']
ord_mapping = [['Pave', 'Grvl'],                          # Street
               ['missing_value', 'Pave', 'Grvl'],         # Alley
               ['missing_value', 'Fa', 'TA', 'Gd', 'Ex']  # PoolQC
]
cat_columns_onehot = ['MSZoning', 'LandContour']


imputer_cat_pipeline = ColumnTransformer([
        ('imp_miss', SimpleImputer(strategy='constant'), cat_columns_fill_miss),  # fill_value='missing_value' by default
        ('imp_freq', SimpleImputer(strategy='most_frequent'), cat_columns_fill_freq),
])

encoder_cat_pipeline = ColumnTransformer([
        ('ordinal', OrdinalEncoder(categories=ord_mapping), cat_columns_ord),
        ('pass_ord', OneHotEncoder(), cat_columns_onehot),
])

cat_pipeline = Pipeline([
        ('imp_cat', imputer_cat_pipeline),
        ('cat_encoder', encoder_cat_pipeline),
])

Unfortunately, when I apply it to housing_cat, the subset of my dataframe including only categorical features,

cat_pipeline.fit_transform(housing_cat)

I get the error:

AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

...

ValueError: Specifying the columns using strings is only supported for pandas DataFrames

I have tried this simplified pipeline and it works properly:

new_cat_pipeline = Pipeline([
        ('imp_cat', imputer_cat_pipeline),
        ('onehot', OneHotEncoder()),
])

However, if I try:

enc_one = ColumnTransformer([
        ('onehot', OneHotEncoder(), cat_columns_onehot),
        ('pass_ord', 'passthrough', cat_columns_ord)
])

new_cat_pipeline = Pipeline([
        ('imp_cat', imputer_cat_pipeline),
        ('onehot_encoder', enc_one),
])

I start to get the same error.

I suspect then that this error is related to the use of ColumnTransformer in the second step, but I do not actually understand where it comes from. The way I identify the columns in the second step is the same as in the first step, so it remains unclear to me why only in the second step I get the Attribute Error...

Answer 1

ColumnTransformer returns numpy.array, so it can't have column attribute (as indicated by your error).

If I may suggest a different solution, use pandas for both of your tasks, it will be easier.

Step 1 - replacing missing values

To replace missing value in a subset of columns with missing_value string use this:

dataframe[["PoolQC", "Alley"]].fillna("missing_value", inplace=True)

For the rest (imputing with mean of each column), this will work perfectly:

dataframe[["Street", "MSZoning", "LandContour"]].fillna(
    dataframe[["Street", "MSZoning", "LandContour"]].mean(), inplace=True
)

Step 2 - one hot encoding and categorical variables

pandas provides get_dummies, which returns pandas Dataframe, unlike ColumnTransfomer, code for this would be:

encoded = pd.get_dummies(dataframe[['MSZoning', 'LandContour']], drop_first=True)
pd.dropna(['MSZoning', 'LandContour'], axis=columns, inplace=True)
dataframe = dataframe.join(encoded)

For ordinal variables and their encoding I would suggest you to look at this SO answer (unluckily some manual mapping would be needed in this case).

If you want to use transformer anyway

Get np.array from the dataframe using values attribute, pass it through the pipeline and recreate columns and indices from the array like this:

pd.DataFrame(data=your_array, index=np.arange(len(your_array)), columns=["A", "B"])

There is one caveat of this aprroach though; you will not know the names of custom created one-hot-encoded columns (the pipeline will not do this for you).

Additionally, you could get the names of columns from sklearn's transforming objects (e.g. using categories_ attribute), but I think it would break the pipeline (someone correct me if I'm wrong).