Search code examples
pythonkeraskeras-layerensemble-learning

How to merge keras sequential models with same input?


I am trying to create my first ensemble models in keras. I have 3 input values and a single output value in my dataset.

from keras.optimizers import SGD,Adam
from keras.layers import Dense,Merge
from keras.models import Sequential

model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation='relu'))
model1.add(Dense(2, activation='relu'))
model1.add(Dense(2, activation='tanh'))
model1.compile(loss='mse', optimizer='Adam', metrics=['accuracy'])

model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation='linear'))
model2.add(Dense(4, activation='tanh'))
model2.add(Dense(3, activation='tanh'))
model2.compile(loss='mse', optimizer='SGD', metrics=['accuracy'])

model3 = Sequential()
model3.add(Merge([model1, model2], mode = 'concat'))
model3.add(Dense(1, activation='sigmoid'))
model3.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])

model3.input_shape

The ensemble model(model3) compiles without any error but while fitting the model I have to pass the same input two times model3.fit([X,X],y). Which I think is an unnecessary step and instead of passing input twice I want to have a common input nodes for my ensemble model. How can I do it?


Solution

  • Keras functional API seems to be a better fit for your use case, as it allows more flexibility in the computation graph. e.g.:

    from keras.layers import concatenate
    from keras.models import Model
    from keras.layers import Input, Merge
    from keras.layers.core import Dense
    from keras.layers.merge import concatenate
    
    # a single input layer
    inputs = Input(shape=(3,))
    
    # model 1
    x1 = Dense(3, activation='relu')(inputs)
    x1 = Dense(2, activation='relu')(x1)
    x1 = Dense(2, activation='tanh')(x1)
    
    # model 2 
    x2 = Dense(3, activation='linear')(inputs)
    x2 = Dense(4, activation='tanh')(x2)
    x2 = Dense(3, activation='tanh')(x2)
    
    # merging models
    x3 = concatenate([x1, x2])
    
    # output layer
    predictions = Dense(1, activation='sigmoid')(x3)
    
    # generate a model from the layers above
    model = Model(inputs=inputs, outputs=predictions)
    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    
    # Always a good idea to verify it looks as you expect it to 
    # model.summary()
    
    data = [[1,2,3], [1,1,3], [7,8,9], [5,8,10]]
    labels = [0,0,1,1]
    
    # The resulting model can be fit with a single input:
    model.fit(data, labels, epochs=50)
    

    Notes:

    • There might be slight differences in the API between Keras versions (pre- and post- version 2)
    • The example above specifies different optimizer and loss function for each of the models. However, since fit() is being called only once (on model3), the same settings - those of model3 - will apply to the entire model. In order to have different settings when training the sub-models, they will have to be fit() separately - see comment by @Daniel.

    EDIT: updated notes based on comments