Is there a GPT-2 implementation that allows me to fine-tune and prompt for text completion?

Question

I wish to to fine-tune a GPT-2 implementation on some text data. I then want to use this model to complete a text prompt. I can do the first part easily enough using Max Woolf's gpt-2-simple implementation. And Neil Shepherd's fork of OpenAI allows for GPT-2 to be trained on new data and completes text.

However, my corpus is too small to train on and not get gibberish back. Is there any way I can combine the two functions? Ideally, I'd like to be able to do this via a python interface (as opposed to CLI), as I'd like to use pandas for data cleaning and what-have-you. Thanks.

Answer 1

Huggingface's Transformers package has a GPT-2 implementation (including pre-trained models) for PyTorch and TensorFlow. You can easily work with them in Python.

Fine-tuning of GPT-2, however, requires a lot of memory and I am not sure is you will be able to do the full backpropagation on that. In that case, you fine-tune just a few highest layers.