Support for locally deployed LLM using LM Studio

[kellerj]

How do you specify the OpenAI endpoint URL? I would like to use a locally running LLM (using the OpenAI API syntax during initial experimentation and development.). Thanks.

[tirumaraiselvan]

Hi @kellerj

Which LLM are you using and how are you deploying it locally - using Ollama or anything like that?

[kellerj]

I was using LM Studio, which provides the OpenAI style endpoints. I've used it with some other tools by setting their environment variables to point to those localhost URLs rather than the ChatGPT endpoints. Ollama is also an option, I just have not used it as much.

[seanparkross]

Forwarded to team.

[tirumaraiselvan]

Supporting all major LLMs (including locally hosted ones) is definitely in the roadmap but not in the short-term.

As of date, we only support OpenAI and Anthropic.