Implementing a "stream"-like Model's response render

[fguillen]

I am using langchainrb to build a RAG Rails-based app. Thanks for the library! :)

In my road map, I feature showing the Model responses as it builds them, "chunk" by "chunk," to offer a more dynamic experience to the User.

What would be the approach to do this using langchainrb?

[andreibondarev]

@fguillen Which LLM are you using? You should be able to stream the response back.

[fguillen]

Hi @andreibondarev, I am using openai (gpt-4o). To be honest, I haven't tried to do it yet. But I haven't seen any note about this in the README.

I am using the assistant mode:

assistant = Langchain::Assistant.new(
  llm: llm,
  thread: thread,
  instructions: "You are a Meteorologist Assistant that is able to pull the weather for any location",
  tools: [
    Langchain::Tool::Weather.new(api_key: ENV["OPEN_WEATHER_API_KEY"])
  ]
)

assistant.add_message content: "How about San Diego, CA?"
assistant.run(auto_tool_execution: true)

And I am unsure how to capture the chunks from the model's response.

[andreibondarev]

So you can stream a response directly doing something like (with any LLM class):

llm.chat(messages: [{role:"user", content:"hello"}]) do |chunk|
  puts chunk
end

But I'll create a ticket to add this to the Assistant class.

[fguillen]

Thanks for the support @andreibondarev !