How to return number of tokens used when executing flow as a function?

[teohsinyee]

I load a flow and use it as a function in code successfully. But I can't find guidelines on how to return the prompt and completion tokens count. I knew that I can checked it through tracing UI by just calling start_trace() but I don't want UI, I just want a JSON where it includes all the tokens tracing.

f = load_flow("../../examples/flows/standard/web-classification/")
f(url="sample_url")

Reference: https://github.com/microsoft/promptflow/blob/7e96f020db860f8974c81111aa2bae49fc95c398/docs/how-to-guides/execute-flow-as-a-function.md

[D-W-]

Hi @teohsinyee , flow-as-a-function is designed to execute flow with minimal extra logic so does not support returning token count. If you need to get token count, you can use flow run instead. You can get token count in flow run's properties.system_metrics.

Here's example code for it:

run = pf.run(flow=flow, data=data, stream=True)
print(run.properties["system_metrics"])

[teohsinyee]

Hi @D-W- , I've tried to execute using flow run as suggested and able to get token count from properties object. However, for the pf.run parameter named data, is there an alternative that can read a text rather than a file path? Because in my use case, I will just retrieve the text from SQL and pass to pf.run. Which my 'data' parameter will not be a file.

  # Set flow path and run input data
  flow = "standard/web-classification" # set the flow directory
  data= "standard/web-classification/data.jsonl" # set the data file

  # create a run, stream it until it's finished
  base_run = pf.run(
      flow=flow,
      data=data,
      stream=True,
      # map the url field from the data to the url input of the flow
      column_mapping={"url": "${data.url}"},
  )

[D-W-]

Currently only file is supported for data parameter. We have an issue tracking support passing list of dicts in data parameter: #3207

[teohsinyee]

Hi @D-W- , could you advise that is it possible to integrate tiktoken library to count the tokens when executing flow as a function?
I noticed MSFT advisor in MS Forum tried to do it with Azure OpenAI, but not with Promptflow; Can we achieve the same by just tweaking the provided code by the advisor?

import tiktoken
import openai
import os
openai.api_type = "azure"
openai.api_version = "2023-03-15-preview" 
openai.api_base = os.getenv("OPENAI_API_BASE")  # Your Azure OpenAI resource's endpoint value .
openai.api_key = os.getenv("OPENAI_API_KEY")

system_message = {"role": "system", "content": "You are a helpful assistant."}
max_response_tokens = 250
token_limit= 4096
conversation=[]
conversation.append(system_message)

def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301"):
    encoding = tiktoken.encoding_for_model(model)
    num_tokens = 0
    for message in messages:
        num_tokens += 4  # every message follows <im_start>{role/name}\n{content}<im_end>\n
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":  # if there's a name, the role is omitted
                num_tokens += -1  # role is always required and always 1 token
    num_tokens += 2  # every reply is primed with <im_start>assistant
    return num_tokens

while(True):
    user_input = input("")     
    conversation.append({"role": "user", "content": user_input})
    conv_history_tokens = num_tokens_from_messages(conversation)

    while (conv_history_tokens+max_response_tokens >= token_limit):
        del conversation[1] 
        conv_history_tokens = num_tokens_from_messages(conversation)
        
    response = openai.ChatCompletion.create(
        engine="gpt-35-turbo", # The deployment name you chose when you deployed the ChatGPT or GPT-4 model.
        messages = conversation,
        temperature=.7,
        max_tokens=max_response_tokens,
    )

    conversation.append({"role": "assistant", "content": response['choices'][0]['message']['content']})
    print("\n" + response['choices'][0]['message']['content'] + "\n")