-
Notifications
You must be signed in to change notification settings - Fork 70
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
gRPC-based Endoscopy Tool Tracking application
Signed-off-by: Victor Chang <[email protected]>
- Loading branch information
Showing
20 changed files
with
1,435 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 19 additions & 0 deletions
19
applications/distributed/grpc/grpc_endoscopy_tool_tracking/CMakeLists.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| cmake_minimum_required(VERSION 3.20) | ||
| project(grpc_endoscopy_tool_tracking LANGUAGES NONE) | ||
|
|
||
| add_subdirectory(cpp) |
77 changes: 77 additions & 0 deletions
77
applications/distributed/grpc/grpc_endoscopy_tool_tracking/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # Endoscopy Tool Tracking Application with gRPC | ||
|
|
||
| This application demonstrates how to offload heavy workloads to a remote Holoscan application using gRPC. | ||
|
|
||
| ## Overview | ||
|
|
||
| In this sample application, we divided the Endoscopy Tool Tracking application into a server and client application where the two communicate via gRPC. | ||
|
|
||
| The client application inputs a video file and streams the video frames to the server application. The server application handles the heavy workloads of inferencing and post-processing of the video frames. It receives the video frames, processes each frame through the endoscopy tool tracking pipeline, and then streams the results to the client. | ||
|
|
||
|  | ||
| *Endoscopy Tool Tracking Application with gRPC* | ||
|
|
||
| From the diagram above, we can see that both the App Cloud (the server) and the App Edge (the client) are very similar to the standalone [Endoscopy Tool Tracking](../../../endoscopy_tool_tracking/) application. This section will only describe the differences; for details on inference and post-processing, please refer to the link above. | ||
|
|
||
| On the client side, we provided two examples, one using a single fragment and another one using two fragments. When comparing the client side to the standalone [Endoscopy Tool Tracking](../../../endoscopy_tool_tracking/) application, the differences are the queues and the gRPC client. We added the following: | ||
| - **Outgoing Requests** operator (`GrpcClientRequestOp`): It converts the video frames (GXF entities) received from the *Video Stream Replayer* operator into `EntityRequest` protobuf messages and queues each frame in the *Request Queue*. | ||
| - **gRPC Service & Client** (`EntityClientService` & `EntityClient`): The gRPC Service is responsible for controlling the life cycle of the gRPC client. The client connects to the remote gRPC server and then sends the requests found in the *Request Queue*. When it receives a response, it converts it into a GXF entity and queues it in the *Response Queue*. | ||
| - **Incoming Responses** operator (`GrpcClientResponseOp`): This operator is configured with an `AsynchronousCondition` condition to check the availability of the *Response Queue*. When notified of available responses in the queue, it dequeues each item and emits each to the output port. | ||
|
|
||
| The App Cloud (the server) application consists of a gRPC server and a few components for managing Holoscan applications. When the server receives a new remote procedure call in this sample application, it launches a new instance of the Endoscopy Tool Tracking application. This is facilitated by the `ApplicationFactory` used for application registration. | ||
|
|
||
| Under the hood, the Endoscopy Tool Tracking application here inherits a custom base class (`HoloscanGrpcApplication`) which manages the `Request Queue` and the `Response Queue` as well as the `GrpcServerRequestOp` and `GrpcServerResponseOp` operators for receiving requests and serving results, respectively. When the RPC is complete, the instance of the Endoscopy Tool Tracking application is destroyed and ready to serve the subsequent request. | ||
|
|
||
|
|
||
| ## Requirements | ||
|
|
||
| ### Data | ||
|
|
||
| [📦️ (NGC) Sample App Data for AI-based Endoscopy Tool Tracking](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara-holoscan/resources/holoscan_endoscopy_sample_data) | ||
|
|
||
| The data is automatically downloaded when building the application. | ||
|
|
||
| ## Building and Running gRPC Endoscopy Tool Tracking Application | ||
|
|
||
| * Building and running the application from the top level Holohub directory: | ||
|
|
||
| ### C++ | ||
|
|
||
| ```bash | ||
| # Start the gRPC Server | ||
| ./dev_container build_and_run grpc_endoscopy_tool_tracking --run_args cloud [--language cpp] | ||
|
|
||
| # Start the gRPC Client | ||
| ./dev_container build_and_run grpc_endoscopy_tool_tracking --run_args edge [--language cpp] | ||
| ``` | ||
|
|
||
|
|
||
| ## Dev Container | ||
|
|
||
| To start the the Dev Container, run the following command from the root directory of Holohub: | ||
|
|
||
| ```bash | ||
| ./dev_container vscode | ||
| ``` | ||
|
|
||
| ### VS Code Launch Profiles | ||
|
|
||
| #### C++ | ||
|
|
||
| The following launch profiles are available: | ||
|
|
||
| - **(compound) grpc_endoscopy_tool_tracking/cpp (cloud & edge)**: Launch both the gRPC server and the client. | ||
| - **(gdb) grpc_endoscopy_tool_tracking/cpp (cloud)**: Launch the gRPC server. | ||
| - **(gdb) grpc_endoscopy_tool_tracking/cpp (edge)**: Launch the gRPC client. | ||
|
|
||
|
|
||
| ## Limitations & Known Issues | ||
|
|
||
| - The connection between the server and the client is controlled by `rpc_timeout`. If no data is received or sent within the configured time, it assumes the call has been completed and hangs up. The `rpc_timeout` value can be configured in the [endoscopy_tool_tracking.yaml](./cpp/endoscopy_tool_tracking.yaml) file with a default of 5 seconds. Increasing this value may help on a slow network. | ||
| - The server can serve one request at any given time. Any subsequent call receives a `grpc::StatusCode::RESOURCE_EXHAUSTED` status. | ||
| - When debugging using the compound profile, the server may not be ready to serve, resulting in errors with the client application. When this happens, open [tasks.json](../../../.vscode/tasks.json), find `Build grpc_endoscopy_tool_tracking (delay 3s)`, and adjust the `command` field with a higher sleep value. | ||
| - The client is expected to exit with the following error. It is how the client application terminates when it completes streaming and displays the entire video. | ||
| ```bash | ||
| [error] [program.cpp:614] Event notification 2 for entity [video_in__outgoing_requests] with id [33] received in an unexpected state [Origin] | ||
| ``` | ||
81 changes: 81 additions & 0 deletions
81
applications/distributed/grpc/grpc_endoscopy_tool_tracking/cpp/CMakeLists.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| cmake_minimum_required(VERSION 3.20) | ||
| project(grpc_endoscopy_tool_tracking CXX) | ||
|
|
||
| find_package(holoscan 2.6 REQUIRED CONFIG | ||
| PATHS "/opt/nvidia/holoscan" "/workspace/holoscan-sdk/install") | ||
|
|
||
| add_executable(grpc_endoscopy_tool_tracking_cloud | ||
| cloud/app_cloud_main.cpp | ||
| cloud/grpc_service.hpp | ||
| cloud/app_cloud_pipeline.hpp | ||
| ) | ||
|
|
||
| add_executable(grpc_endoscopy_tool_tracking_edge | ||
| edge/app_edge_main.cpp | ||
| edge/app_edge.hpp | ||
| edge/video_input_fragment.hpp | ||
| edge/viz_fragment.hpp | ||
| ) | ||
|
|
||
| target_link_libraries(grpc_endoscopy_tool_tracking_cloud | ||
| PRIVATE | ||
| holoscan::core | ||
| holoscan::ops::format_converter | ||
| lstm_tensor_rt_inference | ||
| tool_tracking_postprocessor | ||
| grpc_operators | ||
| ) | ||
|
|
||
| target_link_libraries(grpc_endoscopy_tool_tracking_edge | ||
| PRIVATE | ||
| holoscan::core | ||
| holoscan::ops::holoviz | ||
| holoscan::ops::video_stream_replayer | ||
| grpc_operators | ||
| ) | ||
|
|
||
| # Copy the config to the binary directory | ||
| add_custom_target(grpc_endoscopy_tool_tracking_yaml | ||
| COMMAND ${CMAKE_COMMAND} -E copy_if_different "${CMAKE_CURRENT_SOURCE_DIR}/endoscopy_tool_tracking.yaml" ${CMAKE_CURRENT_BINARY_DIR} | ||
| DEPENDS "endoscopy_tool_tracking.yaml" | ||
| BYPRODUCTS "endoscopy_tool_tracking.yaml" | ||
| ) | ||
| add_dependencies(grpc_endoscopy_tool_tracking_edge grpc_endoscopy_tool_tracking_yaml) | ||
|
|
||
| # Copy the launch script | ||
| add_custom_target(grpc_endoscopy_tool_tracking_launch_sh | ||
| COMMAND ${CMAKE_COMMAND} -E copy_if_different "${CMAKE_CURRENT_SOURCE_DIR}/launch.sh" ${CMAKE_CURRENT_BINARY_DIR} | ||
| DEPENDS "launch.sh" | ||
| BYPRODUCTS "launch.sh" | ||
| ) | ||
| add_dependencies(grpc_endoscopy_tool_tracking_edge grpc_endoscopy_tool_tracking_launch_sh) | ||
|
|
||
| # Default to download datasets | ||
| option(HOLOHUB_DOWNLOAD_DATASETS "Download datasets" ON) | ||
|
|
||
| # Download the endoscopy sample data | ||
| if(HOLOHUB_DOWNLOAD_DATASETS) | ||
| include(holoscan_download_data) | ||
| holoscan_download_data(endoscopy | ||
| URL https://api.ngc.nvidia.com/v2/resources/nvidia/clara-holoscan/holoscan_endoscopy_sample_data/versions/20230222/zip | ||
| DOWNLOAD_NAME holoscan_endoscopy_sample_data_20230222.zip | ||
| URL_MD5 d54f84a562d29ed560a87d2607eba973 | ||
| DOWNLOAD_DIR ${HOLOHUB_DATA_DIR} | ||
| ) | ||
| add_dependencies(grpc_endoscopy_tool_tracking_edge endoscopy_data) | ||
| endif() |
Oops, something went wrong.