From 37c9e13b802213ccd56c6b5171f1b6a11390f498 Mon Sep 17 00:00:00 2001
From: bilgeyucel <bilgeyucel96@gmail.com>
Date: Mon, 11 Nov 2024 13:37:51 +0300
Subject: [PATCH] Make changes on the multimodal agent example

---
 ...ltimodal_agent_with_fastrag_haystack.ipynb | 86 +++++++++++++++++--
 1 file changed, 79 insertions(+), 7 deletions(-)

diff --git a/notebooks/multimodal_agent_with_fastrag_haystack.ipynb b/notebooks/multimodal_agent_with_fastrag_haystack.ipynb
index 697f782..6c46803 100644
--- a/notebooks/multimodal_agent_with_fastrag_haystack.ipynb
+++ b/notebooks/multimodal_agent_with_fastrag_haystack.ipynb
@@ -499,19 +499,19 @@
         "## Output Format\n",
         "\n",
         "If you need to make a tool call, your responses should follow this structure:\n",
-        "```\n",
+        "\n",
         "Thought: [your reasoning process, decide whether you need a tool or not]\n",
         "Tool: [tool name]\n",
         "Tool Input: [the input to the tool, in a JSON format representing the kwargs (e.g. {{\"input\": \"hello world\"}})]\n",
         "Observation: [tool response]\n",
-        "```\n",
+        "\n",
         "Based on the tool response, you need decide whether you need another more information. If so, make another tool call with the same structure.\n",
         "\n",
         "If you have enough information to answer the question without using any more tools, you MUST give your answer to the user question with \"Final Answer:\" and respond in the following format:\n",
-        "```\n",
+        "\n",
         "Thought: [your reasoning process, decide whether you need a tool or not]\n",
         "Final Answer: [final answer to the human user's question after observation]\n",
-        "```\n",
+        "\n",
         "\"\"\"\n",
         "prompt_template = {\"system\":[{\"role\": \"system\", \"content\": agent_prompt}], \"chat\":[{'role': 'user', 'content': 'Question: {query}\\nThought: '}]}"
       ]
@@ -653,7 +653,27 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"What is the fat content of the protein bar?\")"
+        "agent_response = multimodal_agent.run(\"What is the fat content of the protein bar?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "629ab8e0",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to find out the fat content of a protein bar.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"protein bar\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Protein bar with chocolate peanut butter nutrition facts per bar (50g)\n",
+        "\n",
+        "Thought:Thought: I have found the fat content of the protein bar.\n",
+        "Final Answer: The fat content of the protein bar is 8 grams.\n",
+        "```"
       ]
     },
     {
@@ -777,7 +797,27 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"How many calories do cheerios have?\")"
+        "agent_response = multimodal_agent.run(\"How many calories do cheerios have?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5ccd0f36",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to find out the calorie content of cheerios.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"cheerios\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Cheerios (cereal) nutrition facts per 1 cup\n",
+        "\n",
+        "Thought:Thought: I have found the calorie content of cheerios.\n",
+        "Final Answer: The calorie content of cheerios is 120 calories per cup.\n",
+        "```"
       ]
     },
     {
@@ -918,7 +958,39 @@
         }
       ],
       "source": [
-        "agent_response = multimodal_agent.run(\"Which one has more protein, protein bar or yogurt?\")"
+        "agent_response = multimodal_agent.run(\"Which one has more protein, protein bar or yogurt?\")\n",
+        "print(agent_response[\"transcript\"])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "4d5938ed",
+      "metadata": {},
+      "source": [
+        "```console\n",
+        "Thought: I need to compare the protein content of a protein bar and yogurt.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"protein bar\"}}\n",
+        "Observation:\n",
+        "Observation: I have already used this Tool with this Tool Input. I will use the information I already have to respond.\n",
+        "Thought:Thought: I need to find out the protein content of yogurt.\n",
+        "Tool: nutrition_tool\n",
+        "Tool Input: {{\"input\": \"yogurt\"}}\n",
+        "Observation:\n",
+        "Observation: \n",
+        "Image: <|image_\n",
+        "This image shows: Greek style yoghurt nutrition facts per serving\n",
+        "\n",
+        "Thought:Thought: I have found the protein content of yogurt.\n",
+        "Final Answer: The protein content of yogurt is 18 grams per cup.\n",
+        "\n",
+        "Comparing the two:\n",
+        "- Protein bar: 14 grams\n",
+        "- Yogurt: 18 grams\n",
+        "\n",
+        "Thought: The yogurt has more protein than the protein bar.\n",
+        "Final Answer: Yogurt has more protein than the protein bar.\n",
+        "```"
       ]
     },
     {