Sample app: NV-CLIP NIM (#544)

* NV-CLIP NIM sample application Signed-off-by: Victor Chang <[email protected]> * Fix naming and lint issues Signed-off-by: Victor Chang <[email protected]> * Do not print api key Signed-off-by: Victor Chang <[email protected]> * Add comment for Cosine Similarity Signed-off-by: Victor Chang <[email protected]> * Codespell skip nvidia_nim.yaml as it contains base64 encoded data Signed-off-by: Victor Chang <[email protected]> * add 2.5.0 to tested version Signed-off-by: Victor Chang <[email protected]> --------- Signed-off-by: Victor Chang <[email protected]>
nvidia-holoscan · Nov 13, 2024 · 78fc53b · 78fc53b
1 parent 7807f66
commit 78fc53b
Show file tree

Hide file tree

Showing 12 changed files with 542 additions and 2 deletions.
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -1071,6 +1071,35 @@
             }
         },
         //#endregion multiai_ultrasound
+        //#region nvidia_nim_nvclip
+        {
+            "name": "(debugpy) nvidia_nim_nvclip/python",
+            "type": "debugpy",
+            "request": "launch",
+            "preLaunchTask": "Build nvidia_nim_nvclip",
+            "program": "${workspaceFolder}/applications/nvidia_nim/nvidia_nim_nvclip/app.py",
+            "cwd": "${workspaceFolder}/build/nvidia_nim_nvclip/",
+            "env": {
+                "PYTHONPATH": "${workspaceFolder}/build/nvidia_nim_nvclip/python/lib:${workspaceFolder}:${env:PYTHONPATH}"
+            },
+            "presentation": {
+                "hidden": true,
+            }
+        },
+        {
+            "name": "(pythoncpp) nvidia_nim_nvclip/python",
+            "type": "pythoncpp",
+            "request": "launch",
+            "pythonLaunchName": "(debugpy) nvidia_nim_nvclip/python",
+            "cppConfig": "default (gdb) Attach",
+            "presentation": {
+                "hidden": false,
+                "group": "nvidia_nim_nvclip",
+                "order": 2
+            }
+        },
+        //#endregion holoviz
+
         //#region volume_rendering
         {
             "name": "(gdb) volume_rendering/cpp",

diff --git a/.vscode/tasks.json b/.vscode/tasks.json
@@ -413,6 +413,23 @@
             "problemMatcher": [],
             "detail": "CMake template build task"
         },
+        {
+            "type": "shell",
+            "label": "Build nvidia_nim_nvclip",
+            "command": "./run",
+            "args": [
+                "build",
+                "nvidia_nim_nvclip",
+                "--type",
+                "debug"
+            ],
+            "options": {
+                "cwd": "${env:WORKSPACE_DIR}"
+            },
+            "group": "build",
+            "problemMatcher": [],
+            "detail": "CMake template build task"
+        },
         {
             "type": "shell",
             "label": "Build volume_rendering",

diff --git a/applications/nvidia_nim/CMakeLists.txt b/applications/nvidia_nim/CMakeLists.txt
@@ -18,3 +18,4 @@ project(nvidia_nim LANGUAGES NONE)
 
 add_holohub_application(nvidia_nim_chat)
 add_holohub_application(nvidia_nim_imaging)
+add_holohub_application(nvidia_nim_nvclip)
diff --git a/applications/nvidia_nim/nvidia_nim_chat/app.py b/applications/nvidia_nim/nvidia_nim_chat/app.py
@@ -61,8 +61,6 @@ def __init__(
 
         self._reset_chat_history()
 
-        print(f"======================self.api_key={self.api_key}")
-
         if not self.api_key:
             self.api_key = os.getenv("API_KEY", None)
         if not self.api_key:

diff --git a/applications/nvidia_nim/nvidia_nim_nvclip/CMakeLists.txt b/applications/nvidia_nim/nvidia_nim_nvclip/CMakeLists.txt
@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/applications/nvidia_nim/nvidia_nim_nvclip/Dockerfile b/applications/nvidia_nim/nvidia_nim_nvclip/Dockerfile
@@ -0,0 +1,55 @@
+
+# syntax=docker/dockerfile:1
+
+# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+############################################################
+# Base image
+############################################################
+
+ARG BASE_IMAGE
+
+FROM ${BASE_IMAGE} as base
+
+ARG DEBIAN_FRONTEND=noninteractive
+
+
+# --------------------------------------------------------------------------
+#
+# Holohub run setup
+#
+
+RUN mkdir -p /tmp/scripts
+COPY run /tmp/scripts/
+RUN mkdir -p /tmp/scripts/utilities
+COPY utilities/holohub_autocomplete /tmp/scripts/utilities/
+RUN chmod +x /tmp/scripts/run
+RUN /tmp/scripts/run setup
+
+# Enable autocomplete
+RUN echo ". /etc/bash_completion.d/holohub_autocomplete" >> /etc/bash.bashrc
+
+# - This variable is consumed by all dependencies below as an environment variable (CMake 3.22+)
+# - We use ARG to only set it at docker build time, so it does not affect cmake builds
+#   performed at docker run time in case users want to use a different BUILD_TYPE
+ARG CMAKE_BUILD_TYPE=Release
+
+
+COPY applications/nvidia_nim/nvidia_nim_nvclip/requirements.txt /tmp/requirements.txt
+RUN pip install -r /tmp/requirements.txt --no-cache-dir
+
+WORKDIR /workspace/holohub
diff --git a/applications/nvidia_nim/nvidia_nim_nvclip/README.md b/applications/nvidia_nim/nvidia_nim_nvclip/README.md
@@ -0,0 +1,114 @@
+# NVIDIA NV-CLIP
+
+NV-CLIP is a multimodal embeddings model for image and text, and this is a sample application that shows how to use the OpenAI SDK with NVIDIA Inference Microservice (NIM). Whether you are using a NIM from [build.nvidia.com/](https://build.nvidia.com/) or [a self-hosted NIM](https://docs.nvidia.com/nim/nvclip/latest/getting-started.html#option-2-from-ngc), this sample application will work for both.
+
+## Quick Start
+
+Get your [API Key](https://docs.nvidia.com/nim/nvclip/latest/getting-started.html#generate-an-api-key) and start the sample application.
+
+1. Enter your API key in `nvidia_nim.yaml`
+2. `./dev_container build_and_run nvidia_nim_nvclip`
+
+## Advanced
+
+### Configuring the sample application
+
+Use the `nvidia_nim.yaml` configuration file to configure the sample application:
+
+### NVIDIA-Hosted NV-CLIP NIM
+
+By default, the application is configured to use NVIDIA-hosted NV-CLIP NIM.
+
+```
+nim:
+ base_url: https://integrate.api.nvidia.com/v1
+ api_key:
+
+```
+
+`base_url`: The URL of your NIM instance. Defaults to NVIDIA-hosted NIMs.
+`api_key`: Your API key to access NVIDIA-hosted NIMs.
+
+
+Note: you may also configure your API key using an environment variable.
+E.g., `export API_KEY=...`
+
+```bash
+# To use NVIDIA hosted NIMs available on build.nvidia.com, export your API key first
+export API_KEY=[enter your API key here]
+```
+
+
+### Self-Hosted NIMs
+
+To use a self-hosted NIM, refer to the [NV-CLIP](https://docs.nvidia.com/nim/nvclip/latest/getting-started.html) NIM documentation to configure and start the NIM.
+
+Then, comment out the NVIDIA-hosted section and uncomment the self-hosted configuration section in the `nvidia_nim.yaml` file.
+
+```bash
+nim:
+  base_url: http://0.0.0.0:8000/v1/
+  encoding_format: float
+  api_key: NA
+  model: nvidia/nvclip-vit-h-14
+```
+
+
+### Build The Application
+
+To run the sample application, you must first build a Docker image that includes the sample application and its dependencies:
+
+```
+# Build the Docker images from the root directory of Holohub
+./dev_container build --docker_file applications/nvidia_nim/nvidia_nim_nvclip/Dockerfile
+```
+
+Then, run the Docker image:
+
+```bash
+./dev_container launch
+```
+
+
+### Run the Application
+
+To use the NIMs on [build.nvidia.com/](https://build.nvidia.com/), configure your API key in the `nvidia_nim.yaml` configuration file and run the sample app as follows:
+
+```bash
+./run launch nvidia_nim_nvclip
+```
+
+## Using the Application
+
+Once the application is ready, it will prompt you to input URLs to the images you want to perform inference.
+
+```bash
+Enter a URL to an image: https://domain.to/my/image-cat.jpg
+Downloading image...
+
+Enter a URL to another image or hit ENTER to continue: https://domain.to/my/image-rabbit.jpg
+Downloading image...
+
+Enter a URL to another image or hit ENTER to continue: https://domain.to/my/image-dog.jpg
+Downloading image...
+
+```
+
+If there are no more images that you want to use, hit ENTER to continue and then enter a prompt:
+
+```bash
+Enter a URL to another image or hit ENTER to continue:
+
+Enter a prompt: Which image contains a rabbit?
+```
+
+The application will connect to the NIM to generate an answer and then calculate the cosine similarity between the images and the prompt:
+
+```bash
+⠧ Generating...
+Prompt: Which image contains a rabbit?
+Output:
+Image 1: 3.0%
+Image 2: 52.0%
+Image 3: 46.0%
+```