diff --git a/examples/function_calling/fireworks_functions_information_extraction.ipynb b/examples/function_calling/fireworks_functions_information_extraction.ipynb index ad5601a..e926fbe 100644 --- a/examples/function_calling/fireworks_functions_information_extraction.ipynb +++ b/examples/function_calling/fireworks_functions_information_extraction.ipynb @@ -17,9 +17,9 @@ "source": [ "# Summarize Anything - Information Extraction via [Fireworks Function Calling](https://readme.fireworks.ai/docs/function-calling)\n", "\n", - "This is inspired by awesome colab notebook by [Deepset](https://colab.research.google.com/github/anakin87/notebooks/blob/main/information_extraction_via_llms.ipynb). Checkout there OSS LLM Orchestration framework [haystack](https://haystack.deepset.ai/).\n", + "This is inspired by awesome colab notebook by [Deepset](https://colab.research.google.com/github/anakin87/notebooks/blob/main/information_extraction_via_llms.ipynb). Check out there OSS LLM Orchestration framework [haystack](https://haystack.deepset.ai/).\n", "\n", - "In this experiment, we will use function calling ability of [Fireworks Function Calling](https://readme.fireworks.ai/docs/function-calling) model to generate structured information from unstrucutred data.\n", + "In this experiment, we will use function calling ability of [Fireworks Function Calling](https://readme.fireworks.ai/docs/function-calling) model to generate structured information from unstructured data.\n", "\n", "🎯 Goal: create an application that, given a text (or URL) and a specific structure provided by the user, extracts information from the source.\n", "\n", @@ -55,6 +55,16 @@ "id": "w6TPh-3lOTDP" } }, + { + "cell_type": "markdown", + "source": [ + "## Setup\n", + "Let's install the dependencies needed for the demo first and import any dependencies needed." + ], + "metadata": { + "id": "CuitKPhdd5Ze" + } + }, { "cell_type": "code", "source": [ @@ -82,6 +92,17 @@ "from IPython.display import HTML, display" ] }, + { + "cell_type": "markdown", + "source": [ + "## Setup your API Key\n", + "\n", + "In order to use the Fireworks AI function calling model, you must first obtain Fireworks API Keys. If you don't already have one, you can one by following the instructions [here](https://readme.fireworks.ai/docs/quickstart)." + ], + "metadata": { + "id": "pLVxmW58eIIw" + } + }, { "cell_type": "code", "source": [ @@ -105,7 +126,9 @@ "source": [ "## Introduction\n", "\n", - "The [documentation](https://readme.fireworks.ai/docs/function-calling) for FW function calling details the API we can use to specify the list of tools/functions available to the model. We will use the described API to test out the structured response usecase." + "The [documentation](https://readme.fireworks.ai/docs/function-calling) for FW function calling details the API we can use to specify the list of tools/functions available to the model. We will use the described API to test out the structured response usecase.\n", + "\n", + "Before we can begin, let's give the function calling model a go with a simple toy example and examine it's output." ] }, { @@ -138,7 +161,7 @@ "messages = [\n", " {\n", " \"role\": \"system\",\n", - " \"content\": \"You are a helpful assistant with access to tools. Use them wisely and don't image parameter values\",\n", + " \"content\": \"You are a helpful assistant with access to tools. Use them wisely and don't imagine parameter values\",\n", " },\n", " {\n", " \"role\": \"user\",\n", @@ -208,13 +231,16 @@ "id": "4zvaVXFcmy2v" }, "source": [ - "All good! ✅" + "The model outputs the function that should be called along with arguments under the `tool_calls` field. This field contains the arguments to be used for calling the function as JSON Schema and the `name` field contains the name of the function to be called.\n", + "\n", + "\n", + "The output demonstrates a sample input & output to function calling model. All good! ✅" ] }, { "cell_type": "markdown", "source": [ - "# Document Retrieval & Clean Up\n", + "## Document Retrieval & Clean Up\n", "\n", "Before we can get started with extracting the right set of information. We need to first obtaint the document given a url & then clean it up. For cleaning up HTML, we will use [BeautifulSoup](https://beautiful-soup-4.readthedocs.io/en/latest/)." ], @@ -272,6 +298,21 @@ "execution_count": null, "outputs": [] }, + { + "cell_type": "markdown", + "source": [ + "## Setup Information Extraction using Function Calling\n", + "\n", + "After we have obtained clean data from a html page given a url, we are going to send this data to function calling model. Along with sending the cleaned html, we are also going to send it the schema in which we expect the model to produce output. This schema is sent under the tool specification of chat completion call.\n", + "\n", + "For this notebook, we use the `animal_info_tools` schema to extract information from species info pages of [Rain Forest Alliance](https://www.rainforest-alliance.org/). There are several attributes about the animal we want the model to extract from the web page e.g. `weight`, `habitat`, `diet` etc. Additionally, we specify some attributes as `required` forcing the model to always output this information regardless of the input. Given, we would be supplying the model with species information pages, we expect this information to be always present.\n", + "\n", + "**NOTE** We set the temperature to 0.0 to get reliable and consistent output across calls. In this particular example, we want the model to produce the right answer rather than creative answer." + ], + "metadata": { + "id": "sGHBcM90gNTH" + } + }, { "cell_type": "code", "source": [ @@ -391,7 +432,9 @@ { "cell_type": "markdown", "source": [ - "### Let's learn about Capybara" + "### Let's learn about Capybara\n", + "\n", + "Given the schema, we expect the model to produce some basic information like `weight`, `habitat`, `diet` & `predators` for Capybara. You can visit the [webpage](https://www.rainforest-alliance.org/species/capybara/) to see the source of the truth." ], "metadata": { "id": "0kVJ8IfSI-Dx" @@ -426,6 +469,15 @@ } ] }, + { + "cell_type": "markdown", + "source": [ + "You can see the model correctly identifies the correct weight - `100 lbs` for the Capybara even though the webpage mentions the weight in `kgs` too. It also identifies the correct habitat etc. for the animal. " + ], + "metadata": { + "id": "iiAq5zoSiy9C" + } + }, { "cell_type": "markdown", "source": [ @@ -591,13 +643,11 @@ ] }, { - "cell_type": "code", + "cell_type": "markdown", "source": [], "metadata": { "id": "cJ3baFWLQg6L" - }, - "execution_count": null, - "outputs": [] + } } ], "metadata": { @@ -616,4 +666,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file diff --git a/examples/function_calling/fireworks_functions_qa.ipynb b/examples/function_calling/fireworks_functions_qa.ipynb index ec20a18..0fefb43 100644 --- a/examples/function_calling/fireworks_functions_qa.ipynb +++ b/examples/function_calling/fireworks_functions_qa.ipynb @@ -17,13 +17,25 @@ "id": "71a43144" }, "source": [ - "# Structure answers with Fireworks functions\n", + "# Structured answers with Fireworks functions\n", "\n", - "Fireworks (FW) function calling model allows has the ability to produced structured responses. This is often useful in question answering when you want to not only get the final answer but also supporting evidence, citation, etc.\n", + "Several real world applications of LLM require them to respond in a strucutred manner. This structured response could look like `JSON` or `YAML`. For e.g. answering research questions using arxiv along with citations. Instead of parsing the entire LLM response and trying to figure out the actual answer of the LLM vs the citations provided by the LLM, we can use function calling ability of the LLMs to answer questions in a structured way.\n", "\n", - "In this notebook we show how to use an LLM chain which uses FW functions as part of an overall retrieval pipeline." + "In this notebook, we demonstrate structured response generation ability of the Fireworks function calling model. We will build an application that can answer questions (along with citations) regarding the State of the Union speech of 2022." ] }, + { + "cell_type": "markdown", + "source": [ + "# Setup\n", + "\n", + "Install all the dependencies and import the required python modules." + ], + "metadata": { + "id": "-7tAxHrBp4IQ" + }, + "id": "-7tAxHrBp4IQ" + }, { "cell_type": "code", "source": [ @@ -38,7 +50,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "id": "f059012e", "metadata": { "id": "f059012e" @@ -51,6 +63,18 @@ "import openai" ] }, + { + "cell_type": "markdown", + "source": [ + "## Download & Clean the Content\n", + "\n", + "We are going to download the content using the python package `requests` and perform minor cleanup by removing several newlines. Even minimal cleanup should be good enough to obtain good results with the model." + ], + "metadata": { + "id": "tgbH6j3Lp-_x" + }, + "id": "tgbH6j3Lp-_x" + }, { "cell_type": "code", "source": [ @@ -62,7 +86,7 @@ "id": "IcIybYoE35ro" }, "id": "IcIybYoE35ro", - "execution_count": 3, + "execution_count": null, "outputs": [] }, { @@ -75,9 +99,21 @@ "id": "xTeisbO_4UI7" }, "id": "xTeisbO_4UI7", - "execution_count": 4, + "execution_count": null, "outputs": [] }, + { + "cell_type": "markdown", + "source": [ + "## Setup your API Key\n", + "\n", + "In order to use the Fireworks AI function calling model, you must first obtain Fireworks API Keys. If you don't already have one, you can one by following the instructions [here](https://readme.fireworks.ai/docs/quickstart)." + ], + "metadata": { + "id": "XBfEwDuiqQMT" + }, + "id": "XBfEwDuiqQMT" + }, { "cell_type": "code", "source": [ @@ -91,9 +127,21 @@ "id": "ZlTFlhtB5baq" }, "id": "ZlTFlhtB5baq", - "execution_count": 5, + "execution_count": null, "outputs": [] }, + { + "cell_type": "markdown", + "source": [ + "## Define the Structure\n", + "\n", + "Let's define the strucutre in which we want our model to responsd. The JSON structure for function calling follows the conventions of [JSON Schema](https://json-schema.org/). Here we define a structure with `answer` and `citations` field." + ], + "metadata": { + "id": "JoHfdVFlqbjN" + }, + "id": "JoHfdVFlqbjN" + }, { "cell_type": "code", "source": [ @@ -129,13 +177,26 @@ "id": "Zj-9l4m283b4" }, "id": "Zj-9l4m283b4", - "execution_count": 6, + "execution_count": null, "outputs": [] }, + { + "cell_type": "markdown", + "source": [ + "## Perform Sanity Test\n", + "\n", + "Let's perform a sanity test by querying the speech for some basic information. This would ensure that our model setup is working correctly and the document is being processed correctly." + ], + "metadata": { + "id": "4tz7bwV-qset" + }, + "id": "4tz7bwV-qset" + }, { "cell_type": "code", "source": [ - "messages = [\n", + "mp\n", + "essages = [\n", " {\"role\": \"system\", \"content\": f\"You are a helpful assistant who is given document with following content: {clean_content}.\"\n", " \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Please reply in succinct manner and be truthful in the reply.\"},\n", " {\"role\": \"user\", \"content\": \"What did the president say about Ketanji Brown Jackson?\"}\n", @@ -145,7 +206,7 @@ "id": "LcnDoz7H8jjE" }, "id": "LcnDoz7H8jjE", - "execution_count": 7, + "execution_count": null, "outputs": [] }, { @@ -163,7 +224,7 @@ "id": "ENX3Fgcd_JfZ" }, "id": "ENX3Fgcd_JfZ", - "execution_count": 8, + "execution_count": null, "outputs": [] }, { @@ -179,7 +240,7 @@ "outputId": "ee5a8472-167e-4167-f716-94378d8bd333" }, "id": "0WzRJ5PgFAXc", - "execution_count": 9, + "execution_count": null, "outputs": [ { "output_type": "stream", @@ -225,7 +286,7 @@ "id": "bF-o87oxD05g" }, "id": "bF-o87oxD05g", - "execution_count": 10, + "execution_count": null, "outputs": [] }, { @@ -261,7 +322,7 @@ "id": "wYGPiSXfAysM" }, "id": "wYGPiSXfAysM", - "execution_count": 11, + "execution_count": null, "outputs": [] }, { @@ -277,7 +338,7 @@ "outputId": "dcce465f-8d89-4937-f17c-4906dc142dcd" }, "id": "UkWhQ4hPFMc_", - "execution_count": 12, + "execution_count": null, "outputs": [ { "output_type": "stream", @@ -351,7 +412,7 @@ "id": "JxsGWpUcIGan" }, "id": "JxsGWpUcIGan", - "execution_count": 13, + "execution_count": null, "outputs": [] }, { @@ -381,7 +442,7 @@ "id": "g-CYqzXIIUIl" }, "id": "g-CYqzXIIUIl", - "execution_count": 14, + "execution_count": null, "outputs": [] }, { @@ -399,7 +460,7 @@ "id": "RylJZ8BiIewx" }, "id": "RylJZ8BiIewx", - "execution_count": 15, + "execution_count": null, "outputs": [] }, { @@ -415,7 +476,7 @@ "outputId": "d3018c86-21dd-4b40-9a38-7f14bbec05cd" }, "id": "qi_gNf-qI-CG", - "execution_count": 16, + "execution_count": null, "outputs": [ { "output_type": "stream", @@ -476,4 +537,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/examples/function_calling/fireworks_langchain_tool_usage.ipynb b/examples/function_calling/fireworks_langchain_tool_usage.ipynb index 35ef7b0..2276467 100644 --- a/examples/function_calling/fireworks_langchain_tool_usage.ipynb +++ b/examples/function_calling/fireworks_langchain_tool_usage.ipynb @@ -25,70 +25,11 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "s40suaVseAN6", - "outputId": "8e535339-b1e4-404f-cf9d-99bd5b9037cc" + "id": "s40suaVseAN6" }, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (0.1.5)\n", - "Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (1.11.1)\n", - "Requirement already satisfied: langchain_openai in /usr/local/lib/python3.10/dist-packages (0.0.5)\n", - "Collecting langchainhub\n", - " Downloading langchainhub-0.1.14-py3-none-any.whl (3.4 kB)\n", - "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n", - "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.24)\n", - "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.3)\n", - "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n", - "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.6.4)\n", - "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.33)\n", - "Requirement already satisfied: langchain-community<0.1,>=0.0.17 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.17)\n", - "Requirement already satisfied: langchain-core<0.2,>=0.1.16 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.1.18)\n", - "Requirement already satisfied: langsmith<0.1,>=0.0.83 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.86)\n", - "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.23.5)\n", - "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.10.14)\n", - "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.31.0)\n", - "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n", - "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n", - "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n", - "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.26.0)\n", - "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.0)\n", - "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)\n", - "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai) (4.9.0)\n", - "Requirement already satisfied: tiktoken<0.6.0,>=0.5.2 in /usr/local/lib/python3.10/dist-packages (from langchain_openai) (0.5.2)\n", - "Collecting types-requests<3.0.0.0,>=2.31.0.2 (from langchainhub)\n", - " Downloading types_requests-2.31.0.20240125-py3-none-any.whl (14 kB)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n", - "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.4)\n", - "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n", - "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (3.6)\n", - "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.0)\n", - "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (3.20.2)\n", - "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (0.9.0)\n", - "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.11.17)\n", - "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.2)\n", - "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n", - "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain) (2.4)\n", - "Requirement already satisfied: packaging<24.0,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain) (23.2)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)\n", - "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\n", - "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken<0.6.0,>=0.5.2->langchain_openai) (2023.12.25)\n", - "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain) (1.0.0)\n", - "Installing collected packages: types-requests, langchainhub\n", - "Successfully installed langchainhub-0.1.14 types-requests-2.31.0.20240125\n" - ] - } - ], + "outputs": [], "source": [ "!pip install langchain openai langchain_openai langchainhub" ] @@ -123,6 +64,8 @@ "\n", "For solving our math equations, we are going to use the recently released [Mixtral MoE](https://mistral.ai/news/mixtral-of-experts/). For all of our inferencing needs we are going to use the [Fireworks Inference Service](https://fireworks.ai/models).\n", "\n", + "In order to use the Fireworks AI function calling model, you must first obtain Fireworks API Keys. If you don't already have one, you can one by following the instructions [here](https://readme.fireworks.ai/docs/quickstart). Replace `YOUR_FW_API_KEY` with your obtained key.\n", + "\n", "**NOTE:** It's important to set temperature to 0.0 for the function calling model because we want reliable behaviour in routing." ], "metadata": { @@ -211,7 +154,7 @@ "metadata": { "id": "50g_NBE_Atn5" }, - "execution_count": 36, + "execution_count": null, "outputs": [] }, { @@ -239,7 +182,7 @@ "id": "A18JJlKaOgLL", "outputId": "5f0ce2fe-b2ad-4404-8394-f825fd123d71" }, - "execution_count": 23, + "execution_count": null, "outputs": [ { "output_type": "stream", @@ -279,7 +222,7 @@ "id": "qu0HFN35OuRo", "outputId": "89d47217-4b03-4830-add7-03b8b32b61a2" }, - "execution_count": 37, + "execution_count": null, "outputs": [ { "output_type": "stream", @@ -335,15 +278,6 @@ "metadata": { "id": "XFrp6ckCOyYt" } - }, - { - "cell_type": "code", - "source": [], - "metadata": { - "id": "uNHOAmqfQT33" - }, - "execution_count": null, - "outputs": [] } ] -} +} \ No newline at end of file diff --git a/examples/function_calling/fw_autogen_image_gen.ipynb b/examples/function_calling/fw_autogen_image_gen.ipynb index 44f2c18..399a7bc 100644 --- a/examples/function_calling/fw_autogen_image_gen.ipynb +++ b/examples/function_calling/fw_autogen_image_gen.ipynb @@ -28,7 +28,7 @@ }, "outputs": [], "source": [ - "!pip install pyautogen openai" + "!pip install pyautogen openai fireworks-ai" ] }, { @@ -48,13 +48,14 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 13, "metadata": { "id": "zAuynYvM2c3w" }, "outputs": [], "source": [ "import autogen\n", + "import asyncio\n", "import os\n", "from IPython import get_ipython\n", "from typing_extensions import Annotated\n", @@ -64,17 +65,32 @@ "from pathlib import Path\n", "from openai import OpenAI\n", "from matplotlib import pyplot as plt\n", - "import cv2" + "import cv2\n", + "import fireworks.client\n", + "from fireworks.client.image import ImageInference, Answer" ] }, + { + "cell_type": "markdown", + "source": [ + "## Setup\n", + "\n", + "In order to use the Fireworks AI function calling model, you must first obtain Fireworks API Keys. If you don't already have one, you can one by following the instructions [here](https://readme.fireworks.ai/docs/quickstart). Replace FW_API_KEY with your obtained key." + ], + "metadata": { + "id": "ddA-KVZdMoN6" + } + }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": { "id": "FD9w-Ztt3Ssm" }, "outputs": [], "source": [ + "FW_API_KEY = \"YOUR_API_KEY\"\n", + "\n", "config_list = [\n", " {\n", " \"model\": \"accounts/fireworks/models/firefunction-v1\",\n", @@ -82,13 +98,28 @@ " \"base_url\": \"https://api.fireworks.ai/inference/v1\",\n", " \"temperature\": 0.0\n", " }\n", - "]\n", - "OPENAI_API_KEY = \"YOUR_OPENAI_API_KEY\"" + "]" ] }, + { + "cell_type": "markdown", + "source": [ + "## Configure Tools\n", + "\n", + "For this notebook, we are going to use 2 sets of tools\n", + "1. **Image Generation** - We will use [StableDiffusion XL](https://fireworks.ai/models/fireworks/stable-diffusion-xl-1024-v1-0) model on Fireworks platform to generate images for us given the prompt. The tool itself would save the file to a randomly generated file name.\n", + "2. **Show Image** - This tool, given a valid file path, will display the image.\n", + "\n", + "\n", + "Using the AutoGen framework we demonstrate the co-operative nature of agents working with each other to accomplish a complex task. This tutorial can be extended to perform more complicated tasks such as generating stock price charts etc." + ], + "metadata": { + "id": "ZX3i43aUOBUf" + } + }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 28, "metadata": { "id": "5G_iwAz_-NvY" }, @@ -117,38 +148,42 @@ "\n", "@user_proxy.register_for_execution()\n", "@chatbot.register_for_llm(name=\"generate_and_save_images\", description=\"Function to paint, draw or illustrate images based on the users query or request. Generates images from a given query using OpenAI's DALL-E model and saves them to disk. Use the code below anytime there is a request to create an image.\")\n", - "def generate_and_save_images(query: Annotated[str, \"A natural language description of the image to be generated\"], image_size: Annotated[str, \"The size of the image to be generated. (default is '1024x1024')\"] = \"1024x1024\") -> List[str]:\n", + "async def generate_and_save_images(query: Annotated[str, \"A natural language description of the image to be generated\"], image_size: Annotated[str, \"The size of the image to be generated. (default is '1024x1024')\"] = \"1024x1024\") -> List[str]:\n", " \"\"\"\n", " :param query: A natural language description of the image to be generated.\n", " :param image_size: )\n", " :return: A list of filenames for the saved images.\n", " \"\"\"\n", - "\n", - " client = OpenAI(api_key=OPENAI_API_KEY) # Initialize the OpenAI client\n", - " response = client.images.generate(model=\"dall-e-3\", prompt=query, n=1, size=image_size) # Generate images\n", + " fireworks.client.api_key = FW_API_KEY\n", + " inference_client = ImageInference(model=\"stable-diffusion-xl-1024-v1-0\")\n", "\n", " # List to store the file names of saved images\n", " saved_files = []\n", "\n", - " # Check if the response is successful\n", - " if response.data:\n", - " for image_data in response.data:\n", - " # Generate a random UUID as the file name\n", - " file_name = str(uuid.uuid4()) + \".png\" # Assuming the image is a PNG\n", - " file_path = Path(file_name)\n", - "\n", - " img_url = image_data.url\n", - " img_response = requests.get(img_url)\n", - " if img_response.status_code == 200:\n", - " # Write the binary content to a file\n", - " with open(file_path, \"wb\") as img_file:\n", - " img_file.write(img_response.content)\n", - " print(f\"Image saved to {file_path}\")\n", - " saved_files.append(str(file_path))\n", - " else:\n", - " print(f\"Failed to download the image from {img_url}\")\n", + "\n", + " file_name = str(uuid.uuid4()) + \".jpg\" # Assuming the image is a JPG\n", + " file_path = Path(file_name)\n", + "\n", + " # Generate an image using the text_to_image method\n", + " answer : Answer = await inference_client.text_to_image_async(\n", + " prompt=query,\n", + " cfg_scale=7,\n", + " height=1024,\n", + " width=1024,\n", + " sampler=None,\n", + " steps=30,\n", + " seed=0,\n", + " safety_check=False,\n", + " output_image_format=\"JPG\",\n", + " # Add additional parameters here as necessary\n", + " )\n", + "\n", + " if answer.image is None:\n", + " raise RuntimeError(f\"No return image, {answer.finish_reason}\")\n", " else:\n", - " print(\"No image data found in the response!\")\n", + " answer.image.save(file_path)\n", + " print(f\"Image saved to {file_path}\")\n", + " saved_files.append(str(file_path))\n", "\n", " # Return the list of saved files\n", " return {\"path\": saved_files[0]}\n", @@ -163,33 +198,90 @@ " plt.imshow(img)\n", " plt.axis(\"off\")\n", " plt.show()\n", - " return \"\"\n", - "\n" + " return \"\"\n" ] }, + { + "cell_type": "markdown", + "source": [ + "## Initiating Chat\n", + "\n", + "Now we will use the `initiate_chat` functionality to give our AutoGen bot a complex task to accomplish. For this particular task - we ask it to paint an image of ethiopian coffee and show it's image." + ], + "metadata": { + "id": "lLBZp9LTSYDl" + } + }, { "cell_type": "code", - "execution_count": null, + "execution_count": 29, "metadata": { - "id": "11CtFkFWNv_l" + "id": "11CtFkFWNv_l", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "d3ff7dff-8524-4f0b-cc8b-ac09d538ebc5" }, - "outputs": [], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "user_proxy (to chatbot):\n", + "\n", + "paint and show an image of a glass of ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery.\n", + "\n", + "--------------------------------------------------------------------------------\n", + "chatbot (to user_proxy):\n", + "\n", + " \n", + "***** Suggested tool Call (call_F18NYriOOks268hACJ1inp9V): generate_and_save_images *****\n", + "Arguments: \n", + "{\"query\": \"a glass of ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery\", \"image_size\": \"1024x1024\"}\n", + "*****************************************************************************************\n", + "\n", + "--------------------------------------------------------------------------------\n", + "\n", + ">>>>>>>> EXECUTING ASYNC FUNCTION generate_and_save_images...\n", + "Image saved to aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg\n", + "user_proxy (to chatbot):\n", + "\n", + "user_proxy (to chatbot):\n", + "\n", + "***** Response from calling tool \"call_F18NYriOOks268hACJ1inp9V\" *****\n", + "{\"path\": \"aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg\"}\n", + "**********************************************************************\n", + "\n", + "--------------------------------------------------------------------------------\n", + "chatbot (to user_proxy):\n", + "\n", + " Sure, here is the result of your request. I have generated an image of a glass of Ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery. \n", + "\n", + "![Ethiopian Coffee](aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg)\n", + "\n", + "TERMINATE.\n", + "\n", + "--------------------------------------------------------------------------------\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "ChatResult(chat_history=[{'content': 'paint and show an image of a glass of ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery.', 'role': 'assistant'}, {'content': ' ', 'tool_calls': [{'id': 'call_F18NYriOOks268hACJ1inp9V', 'function': {'arguments': '{\"query\": \"a glass of ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery\", \"image_size\": \"1024x1024\"}', 'name': 'generate_and_save_images'}, 'type': 'function', 'index': 0}], 'role': 'assistant'}, {'content': '{\"path\": \"aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg\"}', 'tool_responses': [{'tool_call_id': 'call_F18NYriOOks268hACJ1inp9V', 'role': 'tool', 'content': '{\"path\": \"aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg\"}'}], 'role': 'tool'}, {'content': ' Sure, here is the result of your request. I have generated an image of a glass of Ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery. \\n\\n![Ethiopian Coffee](aace2257-dd6f-45bb-83dd-75c4fe0cf4c7.jpg)\\n\\nTERMINATE.', 'role': 'user'}], summary='', cost=({'total_cost': 0, 'accounts/fireworks/models/firefunction-v1': {'cost': 0, 'prompt_tokens': 1161, 'completion_tokens': 174, 'total_tokens': 1335}}, {'total_cost': 0, 'accounts/fireworks/models/firefunction-v1': {'cost': 0, 'prompt_tokens': 606, 'completion_tokens': 101, 'total_tokens': 707}}), human_input=[])" + ] + }, + "metadata": {}, + "execution_count": 29 + } + ], "source": [ "# start the conversation\n", - "user_proxy.initiate_chat(\n", + "await user_proxy.a_initiate_chat(\n", " chatbot,\n", " message=\"paint and show an image of a glass of ethiopian coffee, freshly brewed in a tall glass cup, on a table right in front of a lush green forest scenery.\",\n", ")" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "rR_HDevWQPQT" - }, - "outputs": [], - "source": [] } ], "metadata": { @@ -206,4 +298,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file