Agents planning #31702

aymeric-roucher · 2024-06-28T17:26:13Z

What does this PR do?

This adds an optional planning steps to agents, to let them insert steps of higher-level planning to their workflow.

It also:

improves several aspects in the Python code interpreter, particularly to handle pandas series.
removes an AgentType behavior where int and floats or other outputs would be converted to AgentType, which made them less accessible.

Fix infinite loops in python interpreter

HuggingFaceDocBuilderDev · 2024-06-28T18:28:11Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

aymeric-roucher · 2024-07-10T08:34:38Z

src/transformers/agents/agents.py

+                    **BASE_PYTHON_TOOLS.copy(),
+                    **self.toolbox.tools,
+                },
+                custom_tools=self.custom_tools,


This change is made because I needed to separate two sets of tools:

user-defined tools that are static (at least they do not change within one call to run) but can be changed in-between run calls

agent-defined tools that will have to change within one run (thus the different handling in python interpreter)

aymeric-roucher · 2024-07-10T08:35:05Z

src/transformers/agents/prompts.py

@@ -365,7 +365,118 @@ def download_prompt(prompt_or_repo_id, agent_name, mode="run"):
 6. Don't name any new variable with the same name as a tool: for instance don't name a variable 'final_answer'.
 7. Never create any notional variables in our code, as having these in your logs might derail you from the true variables.
 8. You can use imports in your code, but only from the following list of modules: <<authorized_imports>>
-9. Don't give up! You're in charge of solving the task, not providing directions to solve it.
+9. The state persists between code executions: so if in one step you've created variables or imported modules, these will all persist.


This precision helps e.g. with matplotlib plots that persist between code snippets.

Super nice prompt!

aymeric-roucher · 2024-07-10T08:57:17Z

src/transformers/agents/python_interpreter.py

@@ -75,8 +85,8 @@ def get_iterable(obj):
        raise InterpreterError("Object is not iterable")


-def evaluate_unaryop(expression, state, tools):
-    operand = evaluate_ast(expression.operand, state, tools)
+def evaluate_unaryop(expression, state, static_tools, custom_tools):


We add the distinction between static_tools and custom_tools everywhere: static_tools are the ones that the interpreter is not allowed to overwrite or create new ones (like base Python functions or user-defined tools), whereas custom_tools are the ones it can modify or extend.

Ok, understood! Can you clarify that in the methods' docstrings?

aymeric-roucher · 2024-07-10T08:58:11Z

src/transformers/agents/python_interpreter.py

    ops = [type(op) for op in condition.ops]

    result = True
    current_left = left

    for op, comparator in zip(ops, comparators):
        if op == ast.Eq:
-            result = result and (current_left == comparator)


This change allows to support pandas "==" statement between a Series and an int for instance

aymeric-roucher · 2024-07-10T09:00:51Z

src/transformers/agents/agents.py

+            f"""
+The code blob you used is invalid: due to the following error: {e}
+This means that the regex pattern {pattern} was not respected: make sure to include code with the correct pattern, for instance:
+Thoughts: Your thoughts


We remind the correct pattern to guide the LLM towards correct formatting.

aymeric-roucher · 2024-07-10T09:02:42Z

src/transformers/agents/agents.py

@@ -655,7 +687,7 @@ def provide_final_answer(self, task) -> str:
        except Exception as e:
            return f"Error in generating final llm output: {e}."

-    def run(self, task: str, stream: bool = False, **kwargs):
+    def run(self, task: str, stream: bool = False, reset: bool = True, **kwargs):


This reset option is provisory, it will be refined later to help multi-turn conversation with an user.

aymeric-roucher · 2024-07-10T09:03:15Z

src/transformers/agents/agents.py

@@ -726,6 +764,91 @@ def direct_run(self, task: str, **kwargs):

        return final_answer

+    def planning_step(self, task, is_first_step=False, iteration: int = None):


We tried to keep this planning step as simple as possible for now.

LysandreJik

No problem for me to merge this!

As the agents framework grows, it would be nice to complete the docstrings to hve each method be understood better; for example the methods planning_step, provide_final_answer, direct_run, stream_run, etc, are they supposed to be run by the user or by the Agent?

Clarifying this via their docstring would help a lot regarding the accessibility of the framework

LysandreJik · 2024-07-16T06:21:49Z

src/transformers/agents/python_interpreter.py

@@ -75,8 +85,8 @@ def get_iterable(obj):
        raise InterpreterError("Object is not iterable")


-def evaluate_unaryop(expression, state, tools):
-    operand = evaluate_ast(expression.operand, state, tools)
+def evaluate_unaryop(expression, state, static_tools, custom_tools):


Ok, understood! Can you clarify that in the methods' docstrings?

LysandreJik · 2024-07-16T06:22:31Z

src/transformers/agents/prompts.py

@@ -365,7 +365,118 @@ def download_prompt(prompt_or_repo_id, agent_name, mode="run"):
 6. Don't name any new variable with the same name as a tool: for instance don't name a variable 'final_answer'.
 7. Never create any notional variables in our code, as having these in your logs might derail you from the true variables.
 8. You can use imports in your code, but only from the following list of modules: <<authorized_imports>>
-9. Don't give up! You're in charge of solving the task, not providing directions to solve it.
+9. The state persists between code executions: so if in one step you've created variables or imported modules, these will all persist.


Super nice prompt!

* Allow planning for agents

aymeric-roucher added 2 commits June 28, 2024 17:01

Allow planning for agents

fc5c978

Fix infinite loops in python interpreter

Clarify structure

31583a7

aymeric-roucher force-pushed the agents-planning branch from 1dce68f to 75d1129 Compare June 28, 2024 17:29

Fix code quality issues

8efac17

aymeric-roucher force-pushed the agents-planning branch from 75d1129 to 8efac17 Compare June 28, 2024 17:50

aymeric-roucher added the Agents label Jun 28, 2024

Merge branch 'main' into agents-planning

75d411c

aymeric-roucher added 3 commits July 1, 2024 19:40

Merge branch 'main' into agents-planning

87739cd

Don't convert floats or ints to str

b12c3c9

Merge branch 'main' into agents-planning

42cd4ab

aymeric-roucher requested review from ArthurZucker and LysandreJik July 8, 2024 18:30

Correct tools

50cad81

aymeric-roucher force-pushed the agents-planning branch from e00836b to 50cad81 Compare July 8, 2024 18:33

aymeric-roucher added 2 commits July 9, 2024 15:58

Fix interpreter errors

9285705

Update prompt to add mention of persistent state

cbba80d

aymeric-roucher force-pushed the agents-planning branch from 6dabac6 to cbba80d Compare July 9, 2024 17:48

aymeric-roucher commented Jul 10, 2024

View reviewed changes

Rename tools to static_tools for clarity

7263052

aymeric-roucher force-pushed the agents-planning branch from ef0c543 to 7263052 Compare July 10, 2024 08:59

aymeric-roucher commented Jul 10, 2024

View reviewed changes

LysandreJik approved these changes Jul 16, 2024

View reviewed changes

Add clarifying docstrings

c4aebd0

aymeric-roucher merged commit b381880 into main Jul 22, 2024
20 checks passed

aymeric-roucher deleted the agents-planning branch July 22, 2024 08:49

MHRDYN7 pushed a commit to MHRDYN7/transformers that referenced this pull request Jul 23, 2024

Agents planning (huggingface#31702)

3ee4326

* Allow planning for agents

zucchini-nlp pushed a commit to zucchini-nlp/transformers that referenced this pull request Jul 24, 2024

Agents planning (huggingface#31702)

24283fe

* Allow planning for agents

itazap pushed a commit that referenced this pull request Jul 25, 2024

Agents planning (#31702)

a5d8c2a

* Allow planning for agents

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Agents planning #31702

Agents planning #31702

aymeric-roucher commented Jun 28, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Jun 28, 2024

aymeric-roucher Jul 10, 2024

aymeric-roucher Jul 10, 2024

LysandreJik Jul 16, 2024

aymeric-roucher Jul 10, 2024

LysandreJik Jul 16, 2024

aymeric-roucher Jul 10, 2024

aymeric-roucher Jul 10, 2024

aymeric-roucher Jul 10, 2024

aymeric-roucher Jul 10, 2024

LysandreJik left a comment

LysandreJik Jul 16, 2024

LysandreJik Jul 16, 2024

		@@ -726,6 +764,91 @@ def direct_run(self, task: str, **kwargs):

		return final_answer

		def planning_step(self, task, is_first_step=False, iteration: int = None):

Agents planning #31702

Agents planning #31702

Conversation

aymeric-roucher commented Jun 28, 2024 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Jun 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aymeric-roucher commented Jun 28, 2024 •

edited

Loading