How do I get ai to interpret the tools response?

[lilouartz]

For context, I am building an AI agent to assist with supplement search (https://pillser.com/ask).

User should be able to ask a question like "Find supplement that contains the most vitamin B"

Then ai should pull supplement data from the database, interpret results, and return the response to the user.

This is my code:

const stream = await streamText({
  messages: coreMessages,
  model: openai('gpt-4o'),
  system: systemPrompt.content,
  tools: {
    getSupplements: tool({
      description: 'Finds Pillser products that match the keywords.',
      execute: async ({ keywords }) => {
        console.log('getSupplements', keywords);

        return [
          {
            name: 'B-100 Complex',
            url: 'https://pillser.com/supplements/b-100-complex-415',
          },
        ];
      },
      parameters: z.object({
        keywords: z.array(z.string()),
      }),
    }),
  },
});

and this is what I am seeing in the stream:

coreMessages [
  {
    role: 'user',
    content: 'suggest me a few products that contain Vitamin B'
  }
]
getSupplements [ 'Vitamin B' ]
{
  type: 'tool-call',
  toolCallId: 'call_',
  toolName: 'getSupplements',
  args: { keywords: [ 'Vitamin B' ] }
}
{
  type: 'tool-result',
  toolCallId: 'call_',
  toolName: 'getSupplements',
  args: { keywords: [ 'Vitamin B' ] },
  result: [
    {
      name: 'B-100 Complex',
      url: 'https://pillser.com/supplements/b-100-complex-415'
    }
  ]
}
{
  type: 'finish',
  finishReason: 'tool-calls',
  logprobs: undefined,
  usage: { promptTokens: 124, completionTokens: 17, totalTokens: 141 }
}

It looks like whatever is returned from tool-result is immediately passed back to the client without feeding that back into the AI to augment the text-response.

How do I use the pulled data to generate the response?

It looks like in case of client-side, I should be using addToolResult, but I want all of this done server-side.

I would expect stream object to have addToolResult, but that does not exist.

[zvictor]

I feel you! Generally speaking, AI SDK lacks a lot more flexibility on the handling of tool calls.
I have created an issue trying to cover, but your use case is not there. Maybe you would want to add your use case to it: #2102

[lilouartz]

I ended up just migrating to the native openai SDK.

Here is the code that is powering https://pillser.com/ask

const stream = openai.beta.chat.completions
    .runTools({
      messages: chatSessionMessages.map((chatSessionMessage) => {
        if (chatSessionMessage.role === 'assistant') {
          return {
            content: chatSessionMessage.content,
            role: 'assistant',
            tool_calls: chatSessionMessage.toolCalls ?? undefined,
          };
        }

        if (chatSessionMessage.role === 'tool') {
          return {
            content: chatSessionMessage.content,
            role: 'tool',
            tool_call_id: chatSessionMessage.toolCallId,
          };
        }

        if (chatSessionMessage.role === 'user') {
          return {
            content: chatSessionMessage.content,
            role: 'user',
          };
        }

        if (chatSessionMessage.role === 'system') {
          return {
            content: chatSessionMessage.content,
            role: 'system',
          };
        }

        throw new Error('Invalid role');
      }),
      model: 'gpt-4o',
      stream: true,
      tools: [
        {
          function: {
            description: 'Finds products that match the keywords.',
            function: async ({ keywords }) => {
              return [];
            },
            name: 'getSupplements',
            parameters: {
              properties: {
                keywords: {
                  items: {
                    type: 'string',
                  },
                  type: 'array',
                },
              },
              type: 'object',
            },
            parse: parseJson,
          },
          type: 'function',
        } as RunnableToolFunction<{ keywords: string[] }>,
      ],
    })
    .on('message', async (message) => {
      if (message.role === 'assistant') {
        // ... update assistant response
      }

      if (message.role === 'tool') {
        // ... record tool response
      }
    })
    .on('content', async (delta) => {
      content += delta;

      send({
        data: stringifyJson(
          UpdateEventZodSchema.parse({
            content,
            ended: false,
            started: true,
          }),
        ),
        event: 'message',
      });

      // ...
    })
    .on('abort', async () => {
      log('openai:stream:abort');

      // ... mark conversation as interrupted
    })
    .on('end', async () => {
      log('openai:stream:end');

      send({
        data: stringifyJson(
          UpdateEventZodSchema.parse({
            content,
            ended: true,
            started: true,
          }),
        ),
        event: 'message',
      });

      // ... mark conversation as ended
    });

  request.signal.addEventListener('abort', () => {
    log('request.signal:abort');

    stream.abort();
  });

[zvictor]

I ended up just migrating to the native openai SDK.

I feel you, again! I am really considering rewriting my whole app because of how opaque AI SDK is. And keep in mind I am someone who already has put a lot effort into porting gemini-chatbot to the Anthropic's provider, so by this time I would have expected to be feeling way more comfortable with tweeking it.

I will still give it one more chance this week, though. Maybe keeping the UI and parts of the RSC might be ideal for me, and then the actual LLM calls could be done using other libs.

P.s: thanks for sharing your code ❤️

[punkpeye]

Equally confused by this. @zvictor @lilouartz did you ever find a way using Vercel's AI SDK?

[iGmainC]

I achieved this.

But I feel there are still some minor issues.

[nsenthilkumar]

check out the docs here which should help. You have to set your maxRoundTrips

https://sdk.vercel.ai/docs/ai-sdk-ui/chatbot-with-tool-calling

[iGmainC]

This solved the problem, awesome