sdk/python/endpoints/online/llm/langchain/1_langchain_basic_deploy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Langchain Integration into Azure Machine Learning (AzureML)\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "* A basic understanding of Machine Learning and Large Language Models\n",
    "* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "* An Azure Machine Learning Workspace, Azure Key Vault, and Azure Container Registry\n",
    "* An OpenAI API Key which can be found in User Settings in OpenAI\n",
    "\n",
    "**Motivations** - The Langchain framework allows for rapid development of applications powered by large language models. This sample creates a chat bot application backed by a large language model and deploys the application to AzureML.\n",
    "\n",
    "**Outline** - \n",
    "1. Prepare the required resources\n",
    "2. Deploy the app to an **AzureML Managed Online Endpoint**\n",
    "3. Test\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning\n",
    "Before deploy, you could test the langchain app locally. We are using [Langchain ChatGPT plugin](https://python.langchain.com/en/latest/modules/agents/tools/examples/chatgpt_plugins.html) as an example app here. Execute the code below to try out. You can inspect the [simple_agent_app_test.py](../src/langchain/simple_agent_app_test.py) to see the implementation itself. It's a langchain ZERO_SHOT_REACT_DESCRIPTION agent with Klarna plugin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_API_TYPE = \"openai\"  # 'azure' or 'openai'\n",
    "OPENAI_API_KEY = \"<OPENAI-API-KEY>\"\n",
    "\n",
    "# required for OpenAI API\n",
    "OPENAI_ORG_ID = \"\"\n",
    "OPENAI_MODEL_ID = \"gpt-3.5-turbo\"\n",
    "\n",
    "# required for Azure OpenAI API\n",
    "AZURE_OPENAI_API_ENDPOINT = \"<AOAI endpoint>\"\n",
    "AZURE_OPENAI_API_DEPLOYMENT_NAME = \"<deployment-name>\"\n",
    "\n",
    "# set to env var for the langchain code to consume\n",
    "%env OPENAI_API_KEY=$OPENAI_API_KEY\n",
    "%env OPENAI_API_TYPE=$OPENAI_API_TYPE\n",
    "%env OPENAI_MODEL_ID=$OPENAI_MODEL_ID\n",
    "%env OPENAI_ORG_ID=$OPENAI_ORG_ID\n",
    "%env AZURE_OPENAI_API_ENDPOINT=$AZURE_OPENAI_API_ENDPOINT\n",
    "%env AZURE_OPENAI_API_DEPLOYMENT_NAME=$AZURE_OPENAI_API_DEPLOYMENT_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Set workspace details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter details of your AML workspace\n",
    "SUBSCRIPTION_ID = \"<SUBSCRIPTION_ID>\"\n",
    "RESOURCE_GROUP = \"<RESOURCE_GROUP>\"\n",
    "AML_WORKSPACE_NAME = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Login to your Azure account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticate clients\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    AzureCliCredential,\n",
    ")\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential(additionally_allowed_tenants=[\"*\"])\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    credential = InteractiveBrowserCredential(additionally_allowed_tenants=[\"*\"])\n",
    "\n",
    "# If login doesn't work above, uncomment the code below and login using device code\n",
    "# !az login --use-device-code"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Managed Online Endpoint\n",
    "### 2.1 Create Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a endpoint\n",
    "import datetime\n",
    "\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    ")\n",
    "\n",
    "from azure.ai.ml import (\n",
    "    MLClient,\n",
    ")\n",
    "\n",
    "time = str(datetime.datetime.now().strftime(\"%m%d%H%M%f\"))\n",
    "online_endpoint_name = f\"aml-llm-lc-demo-{time}\"\n",
    "\n",
    "# get a handle to the workspace\n",
    "ml_client = MLClient(credential, SUBSCRIPTION_ID, RESOURCE_GROUP, AML_WORKSPACE_NAME)\n",
    "\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"online endpoint for Langchain server\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()\n",
    "\n",
    "print(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Store the API Key in KeyVault\n",
    "\n",
    "The below code is modelled after the example notebook [online-endpoints-keyvault.ipynb](../../managed/online-endpoints-keyvault.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Import Keyvault Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.mgmt.keyvault import KeyVaultManagementClient\n",
    "from azure.keyvault.secrets import SecretClient\n",
    "from azure.mgmt.keyvault.models import (\n",
    "    VaultCreateOrUpdateParameters,\n",
    "    VaultProperties,\n",
    "    Sku,\n",
    ")\n",
    "from azure.mgmt.keyvault.models import AccessPolicyEntry, Permissions, SecretPermissions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create a Keyvault Management client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEYVAULT_NAME = f\"llmdemokv{time}\"\n",
    "KV_OPENAI_KEY = \"OPENAI-API-KEY\"\n",
    "\n",
    "keyvault_mgmt_client = KeyVaultManagementClient(\n",
    "    credential=credential, subscription_id=SUBSCRIPTION_ID\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Get your Object Id\n",
    "The `oid` in your JWT access token represents the Object ID of the current user or Service Principal logged into the Azure CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, base64\n",
    "\n",
    "cli_credential = AzureCliCredential()\n",
    "token = cli_credential.get_token(\"https://management.azure.com\").token\n",
    "user_or_sp_object_id = json.loads(base64.b64decode(token.split(\".\")[1] + \"===\")).get(\n",
    "    \"oid\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Define an AccessPolicy for the Endpoint and the current user\n",
    "\n",
    "Allow the endpoint to get secrets in the keyvault and allow all secret permissions for the current user or Service Principal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_access_policy = AccessPolicyEntry(\n",
    "    tenant_id=endpoint.identity.tenant_id,\n",
    "    object_id=endpoint.identity.principal_id,\n",
    "    permissions=Permissions(secrets=[SecretPermissions.GET]),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_or_sp_access_policy = AccessPolicyEntry(\n",
    "    tenant_id=endpoint.identity.tenant_id,\n",
    "    object_id=user_or_sp_object_id,\n",
    "    permissions=Permissions(secrets=[SecretPermissions.ALL]),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Create the Keyvault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keyvault = keyvault_mgmt_client.vaults.begin_create_or_update(\n",
    "    vault_name=KEYVAULT_NAME,\n",
    "    resource_group_name=RESOURCE_GROUP,\n",
    "    parameters=VaultCreateOrUpdateParameters(\n",
    "        location=endpoint.location,\n",
    "        properties=VaultProperties(\n",
    "            tenant_id=endpoint.identity.tenant_id,\n",
    "            sku=Sku(name=\"Standard\", family=\"A\"),\n",
    "            access_policies=[endpoint_access_policy, user_or_sp_access_policy],\n",
    "        ),\n",
    "    ),\n",
    ").result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Add your OPENAI_API_KEY to the Keyvault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEYVAULT_URL = f\"https://{KEYVAULT_NAME}.vault.azure.net\"\n",
    "\n",
    "secret_client = SecretClient(credential=credential, vault_url=KEYVAULT_URL)\n",
    "secret = secret_client.set_secret(name=KV_OPENAI_KEY, value=OPENAI_API_KEY)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Deploy to Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    "    Environment,\n",
    "    CodeConfiguration,\n",
    ")\n",
    "\n",
    "KEYVAULT_URL = f\"https://{KEYVAULT_NAME}.vault.azure.net\"\n",
    "\n",
    "env = Environment(\n",
    "    conda_file=\"deployments/env.yml\",\n",
    "    image=\"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest\",\n",
    ")\n",
    "\n",
    "deployment_name = f\"deploy-{time}\"\n",
    "lc_deployment = ManagedOnlineDeployment(\n",
    "    name=deployment_name,\n",
    "    environment=env,\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"../src\", scoring_script=\"langchain/simple_agent_score.py\"\n",
    "    ),\n",
    "    request_settings=OnlineRequestSettings(request_timeout_ms=60000),\n",
    "    environment_variables={\n",
    "        \"OPENAI_API_KEY\": f\"keyvaultref:{KEYVAULT_URL}/secrets/{KV_OPENAI_KEY}\",\n",
    "        \"OPENAI_API_TYPE\": OPENAI_API_TYPE,\n",
    "        \"OPENAI_MODEL_ID\": OPENAI_MODEL_ID,\n",
    "        \"OPENAI_ORG_ID\": OPENAI_ORG_ID,\n",
    "        \"AZURE_OPENAI_API_ENDPOINT\": AZURE_OPENAI_API_ENDPOINT,\n",
    "        \"AZURE_OPENAI_API_DEPLOYMENT_NAME\": AZURE_OPENAI_API_DEPLOYMENT_NAME,\n",
    "    },\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    instance_type=\"Standard_F2s_v2\",\n",
    "    instance_count=1,\n",
    ")\n",
    "ml_client.online_deployments.begin_create_or_update(lc_deployment).result()\n",
    "\n",
    "endpoint.traffic = {deployment_name: 100}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Test\n",
    "Now endpoint has been deployed, let's test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, json\n",
    "from urllib.parse import urlsplit\n",
    "\n",
    "url_parts = urlsplit(endpoint.scoring_uri)\n",
    "url = url_parts.scheme + \"://\" + url_parts.netloc\n",
    "\n",
    "token = ml_client.online_endpoints.get_keys(name=online_endpoint_name).primary_key\n",
    "headers = {\"Authorization\": \"Bearer \" + token, \"Content-Type\": \"application/json\"}\n",
    "payload = json.dumps(\n",
    "    {\"question\": \"what are the top 5 results for womens t shirts on klarna?\"}\n",
    ")\n",
    "\n",
    "response = requests.post(f\"{url}/score\", headers=headers, data=payload)\n",
    "print(f\"Response:\\n\", response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Clean up resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=online_endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Delete the Keyvault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    }
   ],
   "source": [
    "keyvault_mgmt_client.vaults.delete(RESOURCE_GROUP, KEYVAULT_NAME)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}