huggingface-tokenizers Examples and Free Source Code

Strange results with huggingface transformer[marianmt] translation of larger text...

python translation huggingface-transformers huggingface-tokenizers

Suppress HuggingFace logging warning: "Setting `pad_token_id` to `eos_token_id`:{eos_token_id} ...

huggingface-transformers huggingface-tokenizers

Can not load the safetensors huggingface model in DJL in Java...

java huggingface-transformers word-embedding huggingface-tokenizers djl

How does one set the pad token correctly (not to eos) during fine-tuning to avoid model not predicti...

machine-learning pytorch huggingface-transformers huggingface huggingface-tokenizers

Hugging face tokenizer cannot load files properly...

python nlp huggingface-tokenizers

Question about data_collator throwing a key error in Hugging face...

python pytorch nlp huggingface-transformers huggingface-tokenizers

How to add EOS when training T5?...

machine-learning huggingface-transformers huggingface huggingface-tokenizers huggingface-trainer

Setting padding token as eos token when using DataCollatorForLanguageModeling from HuggingFace...

pytorch huggingface-transformers huggingface-tokenizers huggingface huggingface-datasets

Tokenizer.from_file() HUGGINFACE : Exception: data did not match any variant of untagged enum ModelW...

json nlp huggingface-transformers huggingface-tokenizers huggingface

PanicException: AddedVocabulary bad split AFTER adding tokens to BertTokenizer...

huggingface-transformers huggingface-tokenizers huggingface-trainer

How do I increase max_new_tokens...

huggingface-transformers langchain huggingface-tokenizers llama huggingface-hub

How to know which words are encoded with unknown tokens in HuggingFace BertTokenizer?...

huggingface-transformers huggingface-tokenizers

How to set eos_token_id in llama3 in HuggingFaceLLM?...

large-language-model huggingface huggingface-tokenizers llama-index llama3

How do we add/modify the normalizer in a pretrained Huggingface tokenizer?...

python nlp large-language-model huggingface-tokenizers

Huggingface - Finetuning in Tensorflow with custom datasets...

tensorflow huggingface-transformers transfer-learning huggingface-tokenizers fine-tuning

AutoTokenizer.from_pretrained took forever to load...

python huggingface-transformers huggingface-tokenizers

Embedding of LLM vs custom embeddings...

huggingface-transformers embedding large-language-model huggingface-tokenizers retrieval-augmented-generation

Huggingface tokenizer not able to load model after upgrading python to 3.10...

python-3.x collections jupyter-notebook python-3.10 huggingface-tokenizers

Huggingface pretrained model's tokenizer and model objects have different maximum input length...

nlp huggingface-transformers huggingface-tokenizers sentence-transformers

Transformers v4.x: Convert slow tokenizer to fast tokenizer...

python nlp huggingface-transformers huggingface-tokenizers

Using a custom trained huggingface tokenizer...

python huggingface-transformers huggingface-tokenizers huggingface huggingface-hub

Huggingface tokenizer has two ids for the same token...

huggingface-transformers huggingface-tokenizers

How to resolve ValueError: You should supply an encoding or a list of encodings to this method that ...

nlp huggingface-transformers huggingface-tokenizers peft

Huggingface Tokenizer not adding the padding tokens...

python python-3.x huggingface-transformers huggingface-tokenizers machine-translation

How to stop at 512 tokens when sending text to pipeline? HuggingFace and Transformers...

deep-learning huggingface-transformers huggingface huggingface-tokenizers

How can I push a custom tokenizer to HuggingFace Hub?...

huggingface-tokenizers

How to run a NLP+Transformers LLM on low memory GPUs?...

python nlp gpu huggingface-transformers huggingface-tokenizers

Truncating a training dataset so that it fits exactly within the context window...

huggingface-transformers bert-language-model huggingface-tokenizers

How to truncate input in the Huggingface pipeline?...

huggingface-transformers huggingface-tokenizers

In HuggingFace tokenizers: how can I split a sequence simply on spaces?...

split tokenize huggingface-transformers huggingface-tokenizers