How can we leverage azure prompt flow for evaluation of llm application with another framework?

[HuskyDanny]

I am exploring the framework or cloud service for evaluating llm application.

Our application leverages semantic kernel and sentence-transformers package for RAG scenario. I have used RAGAS for evaluation but only locally. Then I found the Azure Prompt Flow might be a good cloud service for our evaluation tasks. After some playing around, I have the following questions:

1. We have enabled MSI/OAuth for azure open ai service, but I do not see the connection option, which only supports keys. Do we have an update on the auth mechanism?
2. How can we plug our repo into the studio? Preferably, we want to establish the connection to the repo and can reference it as a module, so that I can execute not only the planner and functions but the implementation of those in our application to evaluate the whole flow.

[cloga]

Hi Husky Danny,
Thanks for reaching out.

Prompt flow is working to enhancing the evaluation experience, you can bring your LLM application code and use prompt flow to do evaluation on it, in this part it is framework agnostic.

Specific to your question:

1. Currently we are doing meid_token auth type support for AOAI connection. But the token handle logic only handled by system in prompt flow built-in tool. In your case, you may use your own tool or python tool, so the token handling logic in your code, you didn't need rely on meid_token AOAI connection. For example, you can put dummy key in key based AOAI connection or using custom connection.
   In promptflow runtime, you can use azure default credential to get token: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity

2. Prompt flow support to build your own custom tool and use it as module in prompt flow, you can find detail here: https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-custom-tool-package-creation-and-usage?view=azureml-api-2