Search code examples
pythontorchtqdm

sentence-transformers progress bar can't disable

We're using sentence transformers to batch encode large text

model = SentenceTransformer('/app/resources/all-MiniLM-L6-v2') 
embeddings = []
for obj in data:
    text = obj['text']
    row = {'id': obj['id'],'embedding' : np.ndarray.tolist(model.encode(text)}
    embeddings.append(row)

Which under the covers is using tqdm to log and produces these spammy batch logs

enter image description here

Since we are not interacting with tqdm directly, we're looking for a global way to disable these. From related thread it was suggested that it can be done via an env. variable

export TQDM_DISABLE=1/False/None

This page confirms that this should be valid. In fact, I was able to change the color on the progress bar via: export TQDM_COLOUR='red'

enter image description here

Yet the TQDM_DISABLE doesn't work. I've tried many hacks from other answers - but the only thing that works is completely suppressing logging to warning level, which I don't want to do. Is there any way to just turn off tqdm logs entirely reliably?

Solution

  • Sentence Transformers provides that option. You can implement it like this:model.encode(text, show_progress_bar = False) You can find more information about this option here: Sentence Transformer Documentation. Check out the encode() method.