I have a simple dataset, something like:
data = [
[[1,2], [3,4], [5,6], [7,8], [9,10]],
[[10, 20], [30, 40], [50, 60], [70, 80], [90, 100]],
[[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8], [0.9, 0.10]],
[[0.01, 0.02], [0.03, 0.04], [0.05, 0.06], [0.07, 0.08], [0.09, 0.010]],
[[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]
]
where each input is quite different from the other.
Each input corresponds to a different class:
classes = ['A', 'B', 'C', 'D', 'E']
I am training LSTM as follows:
# Build Model
model = Sequential()
model.add(LSTM(units=units, activation='tanh',
batch_input_shape=(1, None, n_features),
dropout=dropout, recurrent_dropout=recurrent_dropout,
return_sequences=return_sequences, stateful=stateful))
model.add(Dropout(dropout_))
self.model.add(Dense(n_classes, activation='softmax'))
self.model.compile(loss='categorical_crossentropy',
optimizer=optimizer, metrics=['accuracy'])
# Train
model.fit(x=np.array(data), y=np.array(classes), batch_size=1)
I could achieve 100% accuracy on the training set. However, if I try to predict it on sub-sequence it fails tremendously to know to which class the sub-sequence belongs to! Such as:
[[1,2], [3,4]]
[[10, 20], [30, 40], [50, 60]]
[[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]]
[[0.01, 0.02], [0.03, 0.04]]
[[11, 12], [13, 14]]
Applying different parameters for the above: units, dropout ...etc. To fix overfitting (if any), did not work.
Summary:
Prediction on sub-sequences never exceeded 60% accuracy regardless of number of epochs trained, parameters tuning and different optimizers and learning rates used..etc!
I want to move from this to a bigger dataset, but I am unable to make this work first.
You should first split your dataset into train and validation (develop) datasets. Maybe with a test split as well. Then measure the performance on the validation dataset.
You could also augment your dataset with the sub-sequence examples and train/validate on those.