Search code examples
machine-learningtorchhuggingface-transformers

Huge memory usage when running huggingface transformers run_language_modeling.py with GPT2

I tried to test run_language_modeling.py with a small test file and it run out of memory after using more than 32 GB of RAM. Why does it need so much RAM or what am I doing wrong?

Command line:

python run_language_modeling.py --output_dir foo --model_type gpt2 --model_name_or_path gpt2 --do_train --train_data_file test.txt --no_cuda --eval_data_file test.txt

Testfile size: 29600 bytes, 546 lines.

With the original OpenAI implementation I have no problem to run the training script.

Solution

  • Their new run_clm.py script does not have the problem and the old script seems now to be unsupported anyway.