Search code examples
tokenizehuggingface

Libretranslate (+ Huggingface Transformers) - Cannot translate text: Error(s) in loading state_dict for Tokenizer: Missing key(s) in state_dict:

Python 3.10.6, Libretranslate 1.3.10, on Ubuntu 22.04 Installed libretranslate with pip install libretranslate, ran it with libretranslate --host 0.0.0.0 --port 5001. Everything worked as expected. Note: I ran pip as root, which gave me a warning to not use pip as root but I ignored it at that point. Then I created various screens with screen -S libretranslateX. Started running libretranslate on a total of four screens, each with a different port. Everything works. One day later, I check if everything still works, but now when I try to translate a text, I always get this error message

Cannot translate text: Error(s) in loading state_dict for Tokenizer: Missing key(s) in state_dict: "mwt_clf.weight", "mwt_clf.bias", "mwt_clf2.weight".

I stopped all services. Reinstalled libretranslate with pip install --force-reinstall --no-cache-dir libretranslate. Uninstalled with pip uninstall libretranslate. Repeated everything in a venv. I tried older libretranslate versions. It doesn't work. What's strange is that the libretranslate /detect endpoint still works, it's just the /translate endpoint which is returning the error.

The browser console returns this error: POST http://example.com:5001/translate 500 (INTERNAL SERVER ERROR)

I haven't found anything related to this error anywhere. Hence I come to you for help. Any feedback will be greatly appreciated!

Edit: Apparently, this isn't just a problem of libretranslate but for huggingface transformers as well. I have installed the following packages on the same server in order to work with huggingface models: "pinferencia[streamlit]" transformers[torch] sentencepiece nltk stanza and git+https://github.com/PrithivirajDamodaran/Parrot_Paraphraser.git When I try to work with these models, I currently get the same error message I also get on libretranslate

Solution

  • Found the problem and resolved it. During libretranslate installation there was a dependency error message from pip, saying

    argostranslate 1.8.0 requires stanza==1.1.1

    I installed stanza 1.1.1 manually with pip install stanza==1.1.1 but unfortunately that caused the state_dict Tokenizer problem. After checking the version on another machine which didn't have that problem, I installed stanza 1.5.0 (which should have come automatically with it), which gave me the dependency error again, but both libretranslate and huggingface work again now.