Search code examples
pythonpipnlpanacondaspacy

Loading en_core_web_sm results in AttributeError: module 'transformers' has no attribute 'BertTokenizerFast'


I (a beginner at programming and anything pc related) am clueless how to solve the following problem:

I had spacy 3.7.2 installed, including en_core_web_sm. Running the code

nlp = spacy.load("en_core_web_sm")

resulted in the traceback mentioned in the title question.

I tried downgrading spacy to version 3.6.1 (via pip install), then ran the code again and got the traceback:

RegistryError: [E892] Unknown function registry: 'vectors'. Available names: architectures, augmenters, batchers, callbacks, cli, datasets, displacy_colors, factories, initializers, languages, layers, lemmatizers, loggers, lookups, losses, misc, models, ops, optimizers, readers, schedules, scorers, tokenizers

Same happend with spacy version 3.6.0.

Some other person ran my code with spacy 3.4.3 and that worked. So tried downgrading spacy to that, but got the AttributeError again.

I noticed, that on my PC (windows) under ...anaconda3\Lib\site-packages there were still folders with en_core_web_sm version 3.7.2 (despite the downgrading). I randomly deleted those folders. Then installed spacy again, and also tried to manually install en_core_web_sm with python -m spacy download en_core_web_sm-3.4.3

But got the AttributeError in cmd. Tried installing different older/newer versions of spacy as well as en_core_web_sm but can't successfully install the latter.

So now I can't even load en_core_web_sm since it's not installed. Does anyone have an idea what else I could try to install en_core_web_sm?


Solution

  • I think that there is older version of the transformers in your global environment that cause the problem.

    To avoid version conflict, create a new virtual environment using conda :

     conda create -n myenv
    
    

    Activate myenv:

    conda activate myenv
    

    install scipy check the instalation page:

    conda install -c conda-forge spacy
    

    Download en_core_web_sm :

    python -m spacy download en_core_web_sm
    

    Now you can run your code.