Using locally deployed llm with langchain's openai llm wrapper

Question

I have deployed llm model locally which follows openai api schema. As it's endpoint follows openai schema, I don't want to write separate inference client.

Is there any way we can utilize existing openai wrapper by langchain to do inference for my localhost model.

I checked there is a openai adapter by langchain, but it seems like it require provider, which again I have to write separate client.

Overall goal it to not write any redundant code as it's already been maintained by langchain and may change with time. We can modify our api wrt openai and it works out of the box.

Your suggestion is appreciated.

Answer 1

It turns out you can utilize existing ChatOpenAI wrapper from langchain and update openai_api_base with the url where your llm is running which follows openai schema, add any dummy value to openai_api_key can be any random string but is necessary as they have validation for this and finally set model_name to whatever model you've deployed.

Rest other params are same as what you'll set in openai, if you want you can set it.

from langchain_community.chat_models import ChatOpenAI

llm = ChatOpenAI(
        openai_api_base="http://<host-ip>:<port>/<v1>",
        openai_api_key="dummy_value",
        model_name="model_deployed")