How would you do RandomizedSearchCV with VotingClassifier for Sklearn?

Question

I'm trying to tune my voting classifier. I wanted to use randomized search in Sklearn. However how could you set parameter lists for my voting classifier since I currently use two algorithms (different tree algorithms)? Do I have to separately run randomized search and combine them together in voting classifier later?

Could someone help? Code examples would be highly appreciated :)

Thanks!

Answer 1

You can perfectly combine both, the VotingClassifier with RandomizedSearchCV. No need to run them separately. See the documentation: http://scikit-learn.org/stable/modules/ensemble.html#using-the-votingclassifier-with-gridsearch

The trick is to prefix your params list with your estimator name. For example, if you have created a RandomForest estimator and you created it as ('rf',clf2) then you can set up its parameters in the form <name__param>. Specific example: rf__n_estimators: [20,200], so you refer to a specific estimator and set values to test for a specific param.

Ready to test executable code example ;)

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.grid_search import RandomizedSearchCV

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
y = np.array([1, 1, 1, 2, 2, 2])

clf1 = DecisionTreeClassifier()
clf2 = RandomForestClassifier(random_state=1)

params = {'dt__max_depth': [5, 10], 'rf__n_estimators': [20, 200],}


eclf = VotingClassifier(estimators=[('dt', clf1), ('rf', clf2)], voting='hard')

random_search = RandomizedSearchCV(eclf, param_distributions=params,n_iter=4)
random_search.fit(X, y)
print(random_search.grid_scores_)