How to perform cross-validation in keras functional api in python

Question

I want to perform cross validation on a Keras model with multiple inputs. So, I tried KerasClassifier. This works fine with a normal sequential model with only one input. However, when using the functional api and extending to two inputs sklearn's cross_val_predict does not seem to work as expected.

def create_model():
    input_text = Input(shape=(1,), dtype=tf.string)
    embedding = Lambda(UniversalEmbedding, output_shape=(512, ))(input_text)
    dense = Dense(256, activation='relu')(embedding)    
    input_title = Input(shape=(1,), dtype=tf.string)
    embedding_title = Lambda(UniversalEmbedding, output_shape=(512, ))(input_title)
    dense_title = Dense(256, activation='relu')(embedding_title)    
    out = Concatenate()([dense, dense_title])

    pred = Dense(2, activation='softmax')(out)
    model = Model(inputs=[input_text, input_title], outputs=pred)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

Cross validation code that fails

keras_classifier = KerasClassifier(build_fn=create_model, epochs=10, batch_size=10, verbose=1)
cv = StratifiedKFold(n_splits=10, random_state=0)
results = cross_val_predict(keras_classifier, [X1, X2], y, cv=cv, method = "predict_proba")

Later, I discovered that KerasClassifier only supports sequential models: https://keras.io/scikit-learn-api/. In other words, it does not support functional api with multiple inputs.

Therefore, I am wondering if there is any other way to perform cross-validation for models that uses functional api in keras. More specifically, I want to get the prediction probability of each data instance (when it is in test slice in cross-validation) - this is what happens with cross_val_predict.

I am happy to provide more details if needed.

EDIT: My current question is how to input multiple inputs to StratifiedKFold.split(). I have put ???????????? in the code. Just thinking whether it is possible to give it as [input1, input2, input3, input4, input5]

Suppose, I have 5 inputs as input1, input2, input3, input4, input5, how can I use these inputs in StratifiedKFold.split()

k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)

for train_index, test_index in k_fold.split(????????????, labels):

    print("iteration", i, ":")
    print("train indices:", train_index)

    #input1
    print("train data:", input1[train_index])

    #input2
    print("train data:", input2[train_index])

    #input3
    print("train data:", input3[train_index])

    #input4
    print("train data:", input1[train_index])

    #input5
    print("train data:", input1[train_index])

    print("test indices:", test_index)
    print("test data:", X[test_index])

Answer 1

Interesting point that sklearn only supports Sequential but looking your model I think you can have a single input since they share the embedding etc:

def create_model():
    model = Sequential()
    model.add(Lambda(UniversalEmbedding, output_shape=(2, 512), input_shape=(2,)))
    # (2, 512)
    model.add(Flatten()) # (2*512)
    model.add(Dense(2*256, activation='relu')) # (2*256)
    model.add(Dense(2, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

In words, you have 2 inputs of the same domain embedded the same way, so you can use a single input of size 2. Then to mimic two dense layers, you flatten and have a single Dense layer with twice the size :) This brings you to the concatenated layer from which the model is the same.