Search code examples
spacyhuggingface-transformersspacy-transformers

How can I use a model trained with the transformers trainer in the spaCy pipeline?

I have trained and saved a BERT model for NER with the transformers trainer. Is it possible to use this model in the spaCy pipeline?

I have tried with spacy.load('path\to\model'), but get the following error:

OSError: [E053] Could not read meta.json

How can I use this model in the spacy pipeline?

Solution

  • spaCy uses spacy-transformers to wrap HuggingFace Transformers, but it only allows using the models as a source of features, it doesn't allow using task-specific heads like NER. So there's not an easy way to use this in spaCy.

    If you want to load your model as a source of features, see the guide here on how to specify your filename.

    Another option is to load your HuggingFace model separately and wrap it as a spaCy component, but that's kind of involved and usually not very useful.