How can I fine tune an ANN with variable number of layers, each one with a variable number of neurons, using Keras Tuner?

Question

I have the following Hypermodel:

def model(hp):
    units_choice = hp.Int('units', min_value=80, max_value=420, step=10)
    activation_choice = hp.Choice('activation', ['relu', 'tanh'])
    number_of_layers = hp.Int('num_layers', 2, 10)
    learning_rate = hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])

    ANN = Sequential()
    ANN.add(Dense(units=40, input_shape=(40,)))
    for i in range(number_of_layers):
        ANN.add(Dense(units=units_choice, activation=activation_choice))
    ANN.add(Dense(1))
    ANN.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
                metrics=['mean_squared_error'])
    return ANN

What I want is to fine-tune this ANN, using Keras tuner. In this case, I would like the tuner to find the optimum number of layers, every layer with its optimum number of neurons. The above code gives me the same amount of neurons for each of the layer the tuner found.

How can I modify my code to fulfill the requirement above?

Answer 1

You just need to use number_of_layers to define a list of Int hyperparameters to later pass to the layers:

def model(hp):
    activation_choice = hp.Choice('activation', ['relu', 'tanh'])
    number_of_layers = hp.Int('num_layers', 2, 10)
    learning_rate = hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])
    
    # One hyper parameter for each layer.
    units_choices = [hp.Int(f'units_layer_{i}', min_value=80, max_value=420, step=10) for i in range(number_of_layers)]

    ANN = tf.keras.models.Sequential()
    ANN.add(layers.Dense(units=40, input_shape=(40,)))
    for i in range(number_of_layers):
        # Use the list here.
        ANN.add(layers.Dense(units=units_choices[i], activation=activation_choice))
    ANN.add(layers.Dense(1))
    ANN.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
                metrics=['mean_squared_error'])
    return ANN