Search code examples
pythonnlpspyder

How to use the DeBERTa model by He et al. (2022) on Spyder?


I have recently successfully analyzed text-based data using sentence transformers based on the BERT model. Inspired by the book by Kulkarni et al. (2022), my code looked like this:

# Import SentenceTransformer
from sentence_transformers import SentenceTransformer
# use paraphrase-MiniLM-L12-v2 pre trained model
sbert_model = SentenceTransformer('paraphrase-MiniLM-L12-v2')
# My text
x='The cat cought the mouse'
# get embeddings for each question
sentence_embeddings_BERT= sbert_model.encode(x)

I would like to do the same using the DeBERTa model but can't get it running. I managed to load the model, but how to apply it?

import transformers 

from transformers import DebertaTokenizer, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")
model = AutoModel.from_pretrained("microsoft/deberta-v3-base")                    

sentence_embeddings_deBERTa= model(x)

The last line does not run, error message is:

AttributeError: 'str' object has no attribute 'size'

Any experienced DeBERTa users out there?

Thanks Pat


Solution

  • Welcome to SO ;) When you call encode() method it would tokenize the input then encode it to the tensors a transformer model expects, then pass it through model architecture. When you're using transformers you must do the steps manually.

    from transformers import DebertaTokenizer, DebertaModel
    import torch
    # downloading the models
    tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
    model = DebertaModel.from_pretrained("microsoft/deberta-base")
    # tokenizing the input text and converting it into pytorch tensors
    inputs = tokenizer(["The cat cought the mouse", "This is the second sentence"], return_tensors="pt", padding=True)
    # pass through the model 
    outputs = model(**inputs)
    
    print(outputs.last_hidden_state.shape)
    

    Lastly, you must know what kind of output you're supposed to work with.