Search code examples
pythonpytorchhuggingface-transformershuggingface-tokenizers

How to disable TOKENIZERS_PARALLELISM=(true | false) warning?

I use pytorch to train huggingface-transformers model, but every epoch, always output the warning:

The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)

How to disable this warning?

Solution

  • Set the environment variable to the string "false"

    either by

    TOKENIZERS_PARALLELISM=false

    in your shell

    or by:

    import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"

    in the Python script