Search code examples
kerasnlpkeras-layerhuggingface-transformers

biobert for keras version of huggingface transformers

(also posted in https://github.com/dmis-lab/biobert/issues/98)

Hi, does anyone know how to load biobert as a keras layer using the huggingface transformers (version 2.4.1)? I tried several possibilities but none of these worked. All that I found out is how to use the pytorch version but I am interested in the keras layer version. Below are two of my attempts (I saved the biobert files into folder "biobert_v1.1_pubmed").

Attempt 1:

biobert_model = TFBertModel.from_pretrained('bert-base-uncased')
biobert_model.load_weights('biobert_v1.1_pubmed/model.ckpt-1000000')

Error message:

AssertionError: Some objects had attributes which were not restored:
    : ['tf_bert_model_4/bert/embeddings/word_embeddings/weight']
    : ['tf_bert_model_4/bert/embeddings/position_embeddings/embeddings']
   (and many more lines like above...)

Attempt 2:

biobert_model = TFBertModel.from_pretrained("biobert_v1.1_pubmed/model.ckpt-1000000", config='biobert_v1.1_pubmed/bert_config.json')

Error message:

NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights).

Any help appreciated! My experience with huggingface's transformers library is almost zero. I also tried to load the following two models but it seems they only support the pytorch version.

Solution

  • Might be a bit late but I have found a not so elegant fix to this problem. The tf bert models in the transformers library can be loaded with a PyTorch save file.

    Step 1: Convert the tf checkpoint to a Pytorch save file with the following command (more here: https://github.com/huggingface/transformers/blob/master/docs/source/converting_tensorflow_models.rst)

    transformers-cli convert --model_type bert\
  --tf_checkpoint=./path/to/checkpoint_file \
  --config=./bert_config.json \
  --pytorch_dump_output=./pytorch_model.bin

    Step 2: Make sure to combine the following files in a directory

    • config.json - bert config file (must be renamed from bert_config.json!)
    • pytorch_model.bin - the one we just converted
    • vocab.txt - bert vocab file

    Step 3: Load model from the directory we just created

    model = TFBertModel.from_pretrained('./pretrained_model_dir', from_pt=True)

    There is actually also an argument "from_tf" which, according to the documentation should work with tf style checkpoints but I can't get it to work. See: https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained