Neural Networks - hyper parameters tuning

Question

I'm doing hyperparameter tuning for my MLPClassifier model. I'm trying to use gridsearch, but something is wrong and I cantt get what.

This is the model:

data = pd.read_csv("Xy_train.csv")
data = data.loc[(data.age <= 90) & (data.thal != 0)]
x = data.drop(columns=['y', 'id'])
y = data.y


x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=986)
scaler = StandardScaler()
x_train_s = scaler.fit_transform(x_train)
x_test_s = scaler.fit_transform(x_test)

model = MLPClassifier(random_state=986,
                      hidden_layer_sizes=(15),
                      max_iter=200,
                      activation='relu',
                      verbose=True,
                      learning_rate_init=0.01)
model.fit(x_train_s, y_train)

x_train = np.linspace(-5, 5, 50)
y_train = np.linspace(-5, 5, 50)
predictions = pd.DataFrame()

print(f"Accuracy: {accuracy_score(y_true=y_test, y_pred=model.predict(x_test_s)):.3f}")
print(confusion_matrix(y_true=y_test, y_pred=model.predict(x_test_s)))

And here is my grid search:

epochs = [5, 10]
batches = [5, 10, 100]
optimizers = ['lbfgs', 'sgd', 'adam']
hyperparameters = dict(optimizer=optimizers, epochs=epochs, batch_size=batches)
grid = GridSearchCV(estimator=model, cv=3, param_grid=hyperparameters)
grid.fit(x_train, y_train)

I get bunch of errors which doesn't say much, I think maybe the most relevant is:Check the list of available parameters with estimator.get_params().keys(). The errors are:

line 49, in <module>
    grid.fit(x_train, y_train)
line 754, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
line 209, in apply_async
    result = ImmediateResult(func)
 line 233, in set_params
    raise ValueError('Invalid parameter %s for estimator %s. '
ValueError: Invalid parameter epochs for estimator MLPClassifier(activation='relu', alpha=0.0001, batch_size=5, beta_1=0.9,
              beta_2=0.999, early_stopping=False, epsilon=1e-08,
              hidden_layer_sizes=15, learning_rate='constant',
              learning_rate_init=0.01, max_fun=15000, max_iter=200,
              momentum=0.9, n_iter_no_change=10, nesterovs_momentum=True,
              power_t=0.5, random_state=986, shuffle=True, solver='adam',
              tol=0.0001, validation_fraction=0.1, verbose=True,
              warm_start=False). Check the list of available parameters with `estimator.get_params().keys()'

But I did check the parameters keys and it seems fine.

Answer 1

In your hyperparameters grid:

hyperparameters = dict(optimizer=optimizers, epochs=epochs, batch_size=batches)

you ask for hyperparameters that don't exist, like optimizer and epochs, hence the error.

According to the documentation, you should ask respectively for solver and max_iter instead, i.e.:

hyperparameters = dict(solver=optimizers, max_iter=epochs, batch_size=batches)