Implementing a tool that doesn't require the assistant to interpretate the results

[fguillen]

I am implementing a tool to generate charts:

The class:

module Langchain::Tool
  class Chart < Base
    #
    # Generates charts from the data
    #
    # Gem requirements:
    #     gem "chartkick"
    #
    # Usage:
    #     chart = Langchain::Tool::Chart.new()
    #
    NAME = "chart"
    ANNOTATIONS_PATH = Langchain.root.join("#{__dir__}/#{NAME}.json").to_path

    # @return [Chart] Chart object
    def initialize()
    end

    # Chart Tool: Returns the received data
    #
    # @param data [Array] Array of arrays "key, value"
    # @return [Array] the data
    def create_line_chart(data:)
      data
    end
  end
end

The json:

[
  {
    "type": "function",
    "function": {
      "name": "chart__create_line_chart",
      "description": "Chart Tool: Display a line chart of the data",
      "parameters": {
        "type": "object",
        "properties": {
          "data": {
            "type": "string",
            "description": "The data to display in the chart. In the format of an array of arrays [key, value]"
          }
        },
        "required": ["data"]
      }
    }
  }
]

I want the assistant to generate the data argument and call the function. The function spits out the data, creating a tool message with the data.

Then, a helper method captures this message and interprets the message.content to generate a chart:

<%= line_chart(SafeRuby.eval(message.content)) %>

It works more or less:

The problem is that, following the pattern of function calling, I have to send the result of the function call back to the assistant for its interpretation (this happens automagically because I am using auto_tool_execution: true. Then the assistant is trying to interpret the result, and I have weird assistant answers:

Here is the graph with the 10 cellulose family products with the highest margin so far this year:

[Highest margin products chart](attachment://chart.png)

It is hallucinating a chart attachment.

I don't know how to implement this. I don't know if I need the function at all because it does not do much besides offering the function call frame for the assistant to build the proper message. Also, I don't know if I can skip (or if I should) the step of sending the function result back to the assistant for its interpretation.

Just some rambling here asking for some ideas sparring

[andreibondarev]

@fguillen Are you expecting that the Chart tool would return an image that the LLM would then display back to the user?

It is hallucinating a chart attachment.

You can always tell it in the instructions to stop doing that, basically you want to tell it in the instructions how to behave.

Also, I don't know if I can skip (or if I should) the step of sending the function result back to the assistant for its interpretation.

You can return you'd like from the Tool, e.g.:

    def create_line_chart(data:)
      # Creating the line chart
      
      return true # or return("Successfully created the line chart")
    end

[fguillen]

Are you expecting that the Chart tool would return an image that the LLM would then display back to the user?

No, the function is not doing much of a part from pouring the received data back into the return, so I can see data in the conversation stream and parse it to feed the chart helper.

You can always tell it in the instructions to stop doing that, basically you want to tell it in the instructions how to behave.

I changed the description of the function from here:

"description": "Chart Tool: Displays a line chart of the data."

To this:

"description": "Chart Tool: Creates a line chart of the data. You don't need to parse the response of this function."

And now, the assistant is still answering to the function return, but now it is much more consistent:

You can return you'd like from the Tool, e.g.:

I am returning the received data so it appears in the conversation stream so I can detect it and parse it to feed the chart helper. So, in my approach, I can not return true or any success message. But maybe I should have used another approach and generate a pseudo-message to be added to the conversation/thread without the need of using the function return. But then I need access to the conversation/thread... uhmm, maybe I can give access in the initializer of the Tool.

Thanks for helping me think

With the new function.description, it is working well.

[andreibondarev]

@fguillen You also have an option to remove messages or to clear the full thread or submit manual tool output:

 # Remove the last message
assistant.messages.pop

# Clear the whole thread
assistant.clear_thread!

# Manually submit tool output
assistant.submit_tool_output tool_call_id:, output:

[fguillen]

Thanks for the support @andreibondarev :)