I have recently successfully analyzed text-based data using sentence transformers based on the BERT model. Inspired by the book by Kulkarni et al. (2022), my code looked like this:
# Import SentenceTransformer
from sentence_transformers import SentenceTransformer
# use paraphrase-MiniLM-L12-v2 pre trained model
sbert_model = SentenceTransformer('paraphrase-MiniLM-L12-v2')
# My text
x='The cat cought the mouse'
# get embeddings for each question
sentence_embeddings_BERT= sbert_model.encode(x)
I would like to do the same using the DeBERTa model but can't get it running. I managed to load the model, but how to apply it?
import transformers
from transformers import DebertaTokenizer, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")
model = AutoModel.from_pretrained("microsoft/deberta-v3-base")
sentence_embeddings_deBERTa= model(x)
The last line does not run, error message is:
AttributeError: 'str' object has no attribute 'size'
Any experienced DeBERTa users out there?
Thanks Pat
Welcome to SO ;)
When you call encode()
method it would tokenize
the input then encode it to the tensors a transformer model expects, then pass it through model architecture. When you're using transformers
you must do the steps manually.
from transformers import DebertaTokenizer, DebertaModel
import torch
# downloading the models
tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
model = DebertaModel.from_pretrained("microsoft/deberta-base")
# tokenizing the input text and converting it into pytorch tensors
inputs = tokenizer(["The cat cought the mouse", "This is the second sentence"], return_tensors="pt", padding=True)
# pass through the model
outputs = model(**inputs)
print(outputs.last_hidden_state.shape)
Lastly, you must know what kind of output you're supposed to work with.