How to load a finetuned vision llm model? Moondream model case

Question

I've finetuned the moondream model for my own image inference case. I used the finetune script provided on the official repository. It all went well after learning how to use the script for my dataset.

The script generated the following files when i've "save_pretrained()":

• config.json;
• generation_config.json;
• model.safetensors

now i want to load it and make some inference with some images. how can i do it? I've tried using many ways but i haven't managed to do it, for instance:

model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=model_root_dir,
    use_safetensors=True,
    state_dict=load_file(model_path),
    config=model_config,  # Provide a configuration file for the model
    trust_remote_code=True,
)

but i'm getting errors like:

AttributeError: module 'transformers_modules.vikhyatk.moondream2.fb2293ab7450beb1dae536b069f5966becf58e5c.moondream' has no attribute 'Moondream'

How can i solve this error?

Answer 1

The issue probably arises due to significant structural changes made to the moondream model's config and renaming in the HF repository, as seen in this commit

https://huggingface.co/vikhyatk/moondream2/commit/05d640e6da70c37b2473e0db8fef0233c0709ce4

If you’ve saved your checkpoint using save_pretrained(), it no longer align with the new structure of the repository.

Here’s how I solved the problem temporarly. Maybe there needs to be a new "Legacy" moondream model to solve these changes in the long term.

Workaround: Copy the repository content from HuggingFace a commit prior to the breaking changes into the directory where your saved model resides.

Remove any remote references in config.json
For example, replace:

"AutoModelForCausalLM": "vikhyatk/moondream2--moondream.MoondreamModel"

with:

"AutoModelForCausalLM": "moondream.MoondreamModel"

If the model complains about missing .py files, paste them into the local directory (in my case this was in the hugginface cache folder).

Now, load the model with the updated files:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "myawesomemodel",
    trust_remote_code=True,
)

I know this isn't a perfect solution but maybe it helps.

Let me know if you have a better one!