Search code examples
keraslstmword-embeddingzero-padding

Train LSTM with custom padding in each batch size


I am training a RNN for text classification. I want to train the model with a batch size X. However, for each batch, I want to create a zero padding where word_padding_length = maximum string length in each batch - current_word_length. I have tried search but was not able to find anything related to this. This should happen during when I fit the model.


Solution

  • you cannot change the input tensor shape during the training process. In your case, you want to have:
    (batch_size, len1, num_features) -> (batch_size, len2, num_features) for each steps.

    You can have sth similar to this:

    from keras.models import Model
    from keras.layers import Input, LSTM
    
    x = Input(shape=(None,128))
    hidden = LSTM(32)(x)
    model = Model(x, hidden)
    
    for batch in batch_lst:
         model.train_on_batch(batch)
    

    Noted that Input has shape of (None, 128), which means variable batch_size, variable time_steps, and fixed num_feature=128

    Moreover, you might consider using masking layer to ignore all your padding values, so that you can have one tensor input for whole training set while the model performance is not affected by padding.