Use Tensorflow LSTM PTB example for scoring sentences

Question

I try to use an example LSTM, trained according to Tensorflow LSTM example. This example allows to get perplexity on whole test set. But I need to use the trained model to score (get loglikes) of each sentence separately (to score hypotheses of STT decoder output). I modified reader a bit and used code:

mtests=list()
with tf.name_scope("Test"):        
    for test_data_item in test_data:
      test_input.append(PTBInput(config=eval_config, data=test_data_item, name="TestInput"))   
    with tf.variable_scope("Model", reuse=True, initializer=initializer):
      for test_input_item in test_input:
        mtests.append(PTBModel(is_training=False, config=eval_config,
                     input_=test_input_item))
sv = tf.train.Supervisor(logdir=FLAGS.model_dir)

with sv.managed_session() as session:
  checkpoint=tf.train.latest_checkpoint(FLAGS.model_dir)      
  sv.saver.restore(session, checkpoint)
  sys.stderr.write("model restored\n") 

  for mtest in mtests:      
    score, test_perplexity = run_epoch_test(session, mtest)
    print(score)

So, using that code, I get score of each sentence independently. If I pass 5 sentences, it works ok. But if I pass 1k sentences to this code, it works extremely slow and uses a lot of memory, because I create 1k models mtest. So, could you tell me another way to reach my goal? Thank you.

Answer 1

It seems like the model can take a batch of inputs, which is set to 20 in all cases by default. You should be able to feed a larger batch of sentences to one test model to get the output for all of them without having to create multiple models instances. This probably involves some experimenting with the reader, which you are already familiar with.