[computer use] Counting tokens with new count_tokens() method?

[Quasimondo]

In trying to get a better picture of how many tokens each message that Claude sends needs during computer use I am trying to use the new token count endpoint which is right now in beta.

This works fine for regular text messages and responses, but I am struggling with the tool use:

According to the docs for computer use (https://docs.anthropic.com/en/docs/build-with-claude/computer-use) this is how a regular request gets sent:

import anthropic

client = anthropic.Anthropic()

response = client.beta.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    tools=[
        {
          "type": "computer_20241022",
          "name": "computer",
          "display_width_px": 1024,
          "display_height_px": 768,
          "display_number": 1,
        },
        {
          "type": "text_editor_20241022",
          "name": "str_replace_editor"
        },
        {
          "type": "bash_20241022",
          "name": "bash"
        }
    ],
    messages=[{"role": "user", "content": "Save a picture of a cat to my desktop."}],
    betas=["computer-use-2024-10-22"],
)
print(response)

The docs for the token counting endpoint (https://docs.anthropic.com/en/docs/build-with-claude/token-counting) give this example for tool use:

import anthropic

client = anthropic.Anthropic()

response = client.beta.messages.count_tokens(
    betas=["token-counting-2024-11-01"],
    model="claude-3-5-sonnet-20241022",
    tools=[
        {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "input_schema": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    }
                },
                "required": ["location"],
            },
        }
    ],
    messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}]
)

print(response.json())

The thing that I have not figured out yet is how to translate the computer-use tools into the correct format to query the token count, since this approach does not work:

import anthropic

client = anthropic.Anthropic()
response = client.beta.messages.count_tokens(
        betas=["token-counting-2024-11-01"],
        model="claude-3-5-sonnet-20241022",
        tools=[
            {
              "type": "computer_20241022",
              "name": "computer",
              "display_width_px": 1024,
              "display_height_px": 768,
              "display_number": 1,
            },
            {
              "type": "text_editor_20241022",
              "name": "str_replace_editor"
            },
            {
              "type": "bash_20241022",
              "name": "bash"
            }
        ],
        messages=[{"role": "user", "content": "Save a picture of a cat to my desktop."}],
    )

    print(response.json())

anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'tools.0.type: Extra inputs are not permitted'}}

[Quasimondo]

Duh - I realize my mistake - both beta flags must be added:

import anthropic

client = anthropic.Anthropic()
response = client.beta.messages.count_tokens(
        betas=["token-counting-2024-11-01","computer-use-2024-10-22"],
        model="claude-3-5-sonnet-20241022",
        tools=[
            {
              "type": "computer_20241022",
              "name": "computer",
              "display_width_px": 1024,
              "display_height_px": 768,
              "display_number": 1,
            },
            {
              "type": "text_editor_20241022",
              "name": "str_replace_editor"
            },
            {
              "type": "bash_20241022",
              "name": "bash"
            }
        ],
        messages=[{"role": "user", "content": "Save a picture of a cat to my desktop."}],
    )

    print(response.json())

{"input_tokens":1953}

[Quasimondo]

Well, maybe this will be useful for someone else.

[Quasimondo]

Sorry for reopening this, but I realize that it is not really clear to me how to correctly count tokens for tool useage in the context of computer use:

This is a typical entry from the requests that uses the bash tool

[
  {
    "role": "assistant",
    "content": [
      {
        "type": "text",
        "text": "I'll use the bash tool to echo \"hello\":"
      },
      {
        "type": "tool_use",
        "name": "bash",
        "input": {
          "command": "echo \"hello\""
        }
      }
    ]
  }
]

This is the result of that tools use:

{
  "content": [
    {
      "type": "tool_result",
      "content": [
        {
          "type": "text",
          "text": "hello"
        }
      ],
      "tool_use_id": "toolu_01AUudSxQG4DytRGBbD8guvJ",
      "is_error": false
    }
  ],
  "role": "user"
}

It is not really clear to me which of these values go where in the client.beta.messages.count_tokens() call - do I put them into the messages part? Do I pass certain parameters or encode them all into the "content" string? Do the tool_use and the tool_result go into two separate calls? A simple code example would be really very nice to have - thank you!

[Quasimondo]

After a lot of try and error I figured it out - The "secret" is that for tools one has to pass in the assistants "tool_use" and the users(!) "tool_result" together in the message. Also the "tool_use_id" has to be passed as "id":

response = client.beta.messages.count_tokens(
        betas=["token-counting-2024-11-01","computer-use-2024-10-22"],
        model="claude-3-5-sonnet-20241022",
        tools=[
            {
              "type": "computer_20241022",
              "name": "computer",
              "display_width_px": 1024,
              "display_height_px": 768,
              "display_number": 1,
            },
            {
              "type": "text_editor_20241022",
              "name": "str_replace_editor"
            },
            {
              "type": "bash_20241022",
              "name": "bash"
            }
        ],
        messages=[{"role": "assistant", "content": [
           {
           "type": "tool_use",
           "name": "bash",
           "id": "toolu_01AUudSxQG4DytRGBbD8guvJ",
           "input": {"command": "echo \"hello\""}}]},
           
           {"role": "user", 
            "content": [{
              "type": "tool_result",
              "content": [
                {
                  "type": "text",
                  "text": "hello"
                }
              ],
              "tool_use_id": "toolu_01AUudSxQG4DytRGBbD8guvJ",
              
            }]}])

{"input_tokens":2019}

[Quasimondo]

One more observation - removing tools that are not used in that call from the "tool" list gives a different count:

response = client.beta.messages.count_tokens(
        betas=["token-counting-2024-11-01","computer-use-2024-10-22"],
        model="claude-3-5-sonnet-20241022",
        tools=[
           {
              "type": "bash_20241022",
              "name": "bash"
            }
        ],
        messages=[{"role": "assistant", "content": [
           {"type": "tool_use","name": "bash","id": "toolu_01AUudSxQG4DytRGBbD8guvJ","input": {"command": "echo \"hello\""}}]},
           
           {"role": "user", "content": [{
              "type": "tool_result",
              "content": [
                {
                  "type": "text",
                  "text": "hello"
                }
              ],
              "tool_use_id": "toolu_01AUudSxQG4DytRGBbD8guvJ",
            
            }]}
           
           
        ])

{"input_tokens":640}

• You have to leave at least one tool in there - submitting tools=[] will result in an error.
• For counting the tokens used in the context of the actual request of an ongoing thread one probably has to subtract the "tax" for the "tools" object since it will probably only be tallied once for all (e.g. when you use bash 3x it only costs once)
• the "is_error":False can actually be omitted, but it doesn't make a different in the tally