Suggestions and critiques

[alanmun]

Having issues getting set up with this app. I have a personal project and a work project I'd really like to use it for, the premise of this app is really cool! I want a fast way to just submit a bunch of prompt variations, and then control the temperature and other such parameters, and just see what effect each prompt has on the getting me towards the desired behavior of my LLM. For my personal project, it would be really cool if I could feed it different preambles (system prompts) and see which one gets my LLM to behave like the "game master" I am trying to get it to be for my LLM game.

For context, I am following: https://docs.agenta.ai/advanced_guides/custom_applications#using-the-configuration-in-the-application to set Agenta up with an existing app.

requirements.txt

I use pipenv to manage my environment. With that comes Pipfile and Pipfile.lock. So I don't need a requirements.txt in my project. This app would be better if it didn't require requirements.txt. Currently to get started I seem to have to create one.

agenta init

This command is scaring me. I ran it and it overwrote my .gitignore for my project... Its not supposed to do that is it?

It sounds like from this picture that the assumption is for it to be ran in the root of your project. If I do that, my gitignore is trashed and replaced with other stuff. I managed to continue from here by doing git restore .gitignore, but I just don't think the init process should touch this file.

I view agenta as a tool... but it seems like agenta views itself as the application. I want to use agenta as a testing tool for finding out how to best optimize my LLM App. I like the idea of Agenta creating "variants" or "versions" of my app to compare against, but I don't want agenta to be part of the LLM App, or the app itself. A lot of the build process gives me the feeling that it is trying to be the app itself. I feel like this is the wrong way to go about things and it is why I am running into so many issues. My gitignore shouldn't be touched by an LLM App testing tool. I shouldn't have to change the way I manage dependencies or manage packages in my 12 month+ old project. Shouldn't Agenta be agnostic about these things?

If it helps, I'll share the way I was hoping to use agenta, at least for getting started. Maybe then it will be more clear as to what I was hoping to get from agenta:

I created a top level file called evaluate.py, it looks like:

import agenta as ag
from shared.constants import UserType
from agent.custom_agent import setup

ag.init()


@ag.entrypoint
def test_with_agenta(prompt: str):
    REDACTED = True  # REDACTED
    test_account = REDACTED # ? REDACTED is a filled out test account with REDACTED 
    REDACTED = UserType.REDACTED
    primary_agent = setup([], test_account, REDACTED, REDACTED)

    return primary_agent.invoke({"input": prompt})

This is an existing LangChain LLM App. primary_agent is an AgentExecutor that accepts the test queries in the input key. My hope is to have this file dedicated to testing with agenta. This way I leave the rest of the app architecture alone. I feel that there is a good separation of concerns here, if I can keep agenta to just this file, it doesn't need to be interwoven into my tool builder component, or my agent component, or my chat history management component, etc. It just needs to feed various prompts into my existing system and then compare them for me against expected answers and such.

[mmabrouk]

Thank you for taking the time and writing this feedback @alanmun

First thanks for noticing the issue with .gitignore. That was a significant oversight, and we'll fix it asap.

Now, let's address the crux of the matter. We're exploring ways to reduce friction when integrating Agenta with an existing code base. I'll explain what we're aiming for and the constraints we face, and I welcome your thoughts and suggestions:

Our primary goal is to enable you to iterate on the prompts and parameters of your application via a UI (simplifying the workflow and facilitating collaboration with non-technical team members). Ideally, you'd add a couple of lines to your code base (similar to your example), run a single command, and your application is integrated with Agenta: You will have a playground where you can edit these parameters and see live the effect, an evaluation suite where you can batch run stuff, create sweeps, and an observability view where you can see all the logs/traces of the interaction with your system in the different configs that you have created.

For this to work, we need a way to 1) modify your application configuration from the UI, and 2) invoke your application logic from the UI. The latter presents the greatest challenge and is the cause of the current friction. Our solution so far is the agenta serve command, which builds an image from your code base, creates a container, and then generates an endpoint. This allows our UI to invoke your code base, providing all the other capabilities.

We're yet to find a better solution, but we're certainly trying to minimize the friction and changes as much as possible. For example, we're considering the use of Nixpacks which work with pipfile, pyproject.toml, in addition to requirements.txt. We're also looking to create specific langchain integrations to simplify the process.

Here's how we envision it:

• You would still run agenta init (but we won't alter your .gitignore)
• You would create an [app.py/evaluate.py](http://app.py/evaluate.py) similar to your current one, then run agenta serve app.py. Alternatively, you could add a GitHub action to your repository to automate this each time you alter the code.
• You get all the Agenta tools for free: a playground for testing, an evaluation suite, observability, endpoints, versioning, etc.

Would love to hear your thoughts about this. If you're okay with it, I'd love also to schedule a short call to understand your use case.

[alanmun]

That vision sounds good to me. I'm down for a call, might help in case I am not understanding how best to use agenta.