cache control for claude

[hyusetiawan]

hey all, I see the guide here for cache control, I have a couple of follow up questions:

1. Does that mean we can't use system property and can only use messages as shown in the example? like we can't do this:

 const response = await streamText({
      model,
      temperature: 0,
      maxSteps: 10, // allow up to 5 steps
      system: `this is my system message but where to put the cache control?`

2. Is there an example of how to do cache control for the tools? The tools are typed with the first level keys to be any string and can contain the tool function definition

Thank you!

[lgrammel]

1. yes you need messages to use cache control for the system prompt (you can still use system: if the caching is in the user msgs)
2. you can set the cache annotation on the tool msg

[hyusetiawan]

could you elaborate on point 1, specifically using system:? So is it because the way Claude prompt caching works, as in everything prior to the cache control object will be cached, so system and tools are automatically cached as long as the cache control happens to be ordered just prior to message of type "user"?

Do you happen to have an example for cache annotation on the tool msg? I see this https://sdk.vercel.ai/docs/foundations/prompts#tool-messages but i don't think it is right

[lgrammel]

Examples are here: https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic#example-cache-control

For tools you would just the experimental_providerMetadata on the tool msg

[hyusetiawan]

Sorry but I'm still unsure what is a tool message? Like tools: property? Or inside an array of messages as role assistant and somehow type id tools?

…

On Thu, Sep 19, 2024 at 8:44 AM Lars Grammel ***@***.***> wrote: Examples are here: https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic#example-cache-control For tools you would just the experimental_providerMetadata on the tool msg — Reply to this email directly, view it on GitHub <#3062 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIDQCCQ24JULI3EA7HAXTDZXLWPBAVCNFSM6AAAAABOPMVWRKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRZGUYTMNA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lgrammel]

Tool messages are one of the core message types: https://sdk.vercel.ai/docs/foundations/prompts#tool-messages

[hyusetiawan]

It seems this is the message to call and its result? But i want to cache the tools definition as shown here https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching#caching-tool-definitions

…

On Thu, Sep 19, 2024 at 9:21 AM Lars Grammel ***@***.***> wrote: Tool messages are one of the core message types: https://sdk.vercel.ai/docs/foundations/prompts#tool-messages — Reply to this email directly, view it on GitHub <#3062 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIDQCC5LW4ONW66LZ27VKLZXL2XDAVCNFSM6AAAAABOPMVWRKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRZGU2TANQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lgrammel]

Ah I see. We would need to add that. In the meantime, would having a system msg and caching that work for you?

[hyusetiawan]

it does not stop development but it will stop production deployment because we have a lot of tools. I hate to ask but is there a very rough timeline? Because it will probably take us 2 months to deployment to prod.

[hyusetiawan]

@lgrammel just following up to conclude, would this be added to the roadmap?