Merge pull request #151 from deepset-ai/b-multimodal-agent

Make changes on the multimodal agent example
deepset-ai · Nov 11, 2024 · fd8ca58 · fd8ca58
2 parents 049a331 + 37c9e13
commit fd8ca58
Showing 1 changed file with 79 additions and 7 deletions.
diff --git a/notebooks/multimodal_agent_with_fastrag_haystack.ipynb b/notebooks/multimodal_agent_with_fastrag_haystack.ipynb
@@ -499,19 +499,19 @@
         "## Output Format\n",
         "\n",
         "If you need to make a tool call, your responses should follow this structure:\n",
-        "```\n",
+        "\n",
         "Thought: [your reasoning process, decide whether you need a tool or not]\n",
         "Tool: [tool name]\n",
         "Tool Input: [the input to the tool, in a JSON format representing the kwargs (e.g. {{\"input\": \"hello world\"}})]\n",
         "Observation: [tool response]\n",
-        "```\n",
+        "\n",
         "Based on the tool response, you need decide whether you need another more information. If so, make another tool call with the same structure.\n",
         "\n",
         "If you have enough information to answer the question without using any more tools, you MUST give your answer to the user question with \"Final Answer:\" and respond in the following format:\n",
-        "```\n",
+        "\n",
         "Thought: [your reasoning process, decide whether you need a tool or not]\n",
         "Final Answer: [final answer to the human user's question after observation]\n",
-        "```\n",
+        "\n",
         "\"\"\"\n",
         "prompt_template = {\"system\":[{\"role\": \"system\", \"content\": agent_prompt}], \"chat\":[{'role': 'user', 'content': 'Question: {query}\\nThought: '}]}"
       ]
@@ -653,7 +653,27 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"What is the fat content of the protein bar?\")"
+        "agent_response = multimodal_agent.run(\"What is the fat content of the protein bar?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "629ab8e0",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to find out the fat content of a protein bar.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"protein bar\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Protein bar with chocolate peanut butter nutrition facts per bar (50g)\n",
+        "\n",
+        "Thought:Thought: I have found the fat content of the protein bar.\n",
+        "Final Answer: The fat content of the protein bar is 8 grams.\n",
+        "```"
       ]
     },
     {
@@ -777,7 +797,27 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"How many calories do cheerios have?\")"
+        "agent_response = multimodal_agent.run(\"How many calories do cheerios have?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5ccd0f36",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to find out the calorie content of cheerios.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"cheerios\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Cheerios (cereal) nutrition facts per 1 cup\n",
+        "\n",
+        "Thought:Thought: I have found the calorie content of cheerios.\n",
+        "Final Answer: The calorie content of cheerios is 120 calories per cup.\n",
+        "```"
       ]
     },
     {
@@ -918,7 +958,39 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"Which one has more protein, protein bar or yogurt?\")"
+        "agent_response = multimodal_agent.run(\"Which one has more protein, protein bar or yogurt?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "4d5938ed",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to compare the protein content of a protein bar and yogurt.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"protein bar\"}}\n",
+        "Observation:\n",
+        "Observation: I have already used this Tool with this Tool Input. I will use the information I already have to respond.\n",
+        "Thought:Thought: I need to find out the protein content of yogurt.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"yogurt\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Greek style yoghurt nutrition facts per serving\n",
+        "\n",
+        "Thought:Thought: I have found the protein content of yogurt.\n",
+        "Final Answer: The protein content of yogurt is 18 grams per cup.\n",
+        "\n",
+        "Comparing the two:\n",
+        "- Protein bar: 14 grams\n",
+        "- Yogurt: 18 grams\n",
+        "\n",
+        "Thought: The yogurt has more protein than the protein bar.\n",
+        "Final Answer: Yogurt has more protein than the protein bar.\n",
+        "```"
       ]
     },
     {