I'm currently using HuggingFace's T5 implementation for text generation purposes. More specifically, I'm using the T5ForConditionalGeneration to solve a text classification problem as generation.
The model's performance is overall very satisfactory after training, but what I am wondering is how I can get the logits for generation?
I'm currently performing inference as is suggested in the documentation via model.generate(**tokenizer_outputs), but this simply outputs the IDs themselves without anything else.
The reason why I want the logits is because I want to measure the model's confidence of generation. I'm not 100% certain if my approach is correct, but I'm thinking that if I can get the logit values of each generated token and average them, I could get the overall confidence score of the generated sequence.
Would anybody know how I could do this? Thanks.
I was struggling with this because I wasn't familiar with how the Transformers library works, but after looking at the source code all you have to do is set the arguments output_scores and return_dict_in_generate to True.
For more information, take a look at the method transformers.generation.utils.GenerationMixin.generate.