Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apollo: Remove nested adaptor functions in AI assistant #130

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

hunterachieng
Copy link

@hunterachieng hunterachieng commented Dec 5, 2024

Short Description

The AI assistant currently gives code snippets for job writing that have nested adaptor functions which goes away from the openFn job writing architectural requirement.

Fixes #97

Implementation Details

A more detailed breakdown of the changes, including motivations (if not provided in the issue).

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

  • Code generation (copilot but not intellisense)
  • Learning or fact checking
  • Strategy / design
  • Optimisation / refactoring
  • Translation / spellchecking / doc gen
  • Other
  • I have not used AI

You can read more details in our Responsible AI Policy

Signed-off-by: Hunter Achieng <[email protected]>
@@ -0,0 +1,966 @@
{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I ask you to rename both notebooks to something more meanigful? issue 97 is too broad.

Maybe like:

test_nested_state
test_related_questions

"outputs": [],
"source": [
"evaluate_answer_template = \"\"\"\n",
"For the given Question and Answer pair, evaluate whether the answer at least briefly answers the question (True), or if it entirely fails to answer the question (False).\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For your test, I would argue that we don;t need to check whether it answers the question. It's sufficient just to look for that nested state argument.

I'd also argue you can do this with regular python rather than using AI? But I'll leave that up to you

"name": "stdout",
"output_type": "stream",
"text": [
"V1 success rate: 0.8888888888888888, V2 success rate: 0.9\n"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The success rate here on v1 is way too high!

I think that means you'll need to manually figure out questions which showcase the issue. Or maybe generate many more questions.

We can only solve the problem if we have a good dataset of questions which showcase the issue :)

@hunterachieng
Copy link
Author

@josephjclark I have fixed the requested changes. For the v1 and v2 issue, they might look the same since I am using the same Job_guide in both. I had to import the existing one. Maybe if we edit is when we can notice a difference?

Signed-off-by: Hunter Achieng <[email protected]>
Signed-off-by: Hunter Achieng <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

job chat: claude issues
2 participants