Recurrent Neural Networks (RNN) with different sequences length inputs

Question

I have different length input texts, from few character to hundred words so I decided to use different MAX_LENGTH for each batch instead of fix MAX_LENGTH for all batches (obviously shorter MAX_LENGTH for smaller text).

After googling I saw this thread in Keras github page which gave the following solution:

Sequences should be grouped together by length, and segmented manually into batches by that length before being sent to Keras.

If I use this trick, I'll guess there is no way to shuffle data at training time and it might lead to overfitting.

I saw many disscution in Kaggle that use this trick. I want to know is there any other solution for this problem?

Answer 1

There is the solution of padding your data with a dummy value, so all your input sequences have the same length.

Suppose you have these two sequences:

• [1,2,3,1,2], you keep it as [1,2,3,1,2]
• [1,3,2,3], you pad it with zeros until it reaches the desired length: [1,3,2,3,0]

Then you start your model with a Masking layer.

This will automatically ignore the additional length in the samples.