-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
apollo: Remove nested adaptor functions in AI assistant #130
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Hunter Achieng <[email protected]>
Signed-off-by: Hunter Achieng <[email protected]>
@@ -0,0 +1,966 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I ask you to rename both notebooks to something more meanigful? issue 97 is too broad.
Maybe like:
test_nested_state
test_related_questions
"outputs": [], | ||
"source": [ | ||
"evaluate_answer_template = \"\"\"\n", | ||
"For the given Question and Answer pair, evaluate whether the answer at least briefly answers the question (True), or if it entirely fails to answer the question (False).\n", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For your test, I would argue that we don;t need to check whether it answers the question. It's sufficient just to look for that nested state argument.
I'd also argue you can do this with regular python rather than using AI? But I'll leave that up to you
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"V1 success rate: 0.8888888888888888, V2 success rate: 0.9\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The success rate here on v1 is way too high!
I think that means you'll need to manually figure out questions which showcase the issue. Or maybe generate many more questions.
We can only solve the problem if we have a good dataset of questions which showcase the issue :)
Signed-off-by: Hunter Achieng <[email protected]>
@josephjclark I have fixed the requested changes. For the |
Signed-off-by: Hunter Achieng <[email protected]>
Signed-off-by: Hunter Achieng <[email protected]>
Signed-off-by: Hunter Achieng <[email protected]>
Short Description
The AI assistant currently gives code snippets for job writing that have nested adaptor functions which goes away from the openFn job writing architectural requirement.
Fixes #97
Implementation Details
A more detailed breakdown of the changes, including motivations (if not provided in the issue).
AI Usage
Please disclose how you've used AI in this work (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy