-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workspace with ZED for Simulation and Real Camera #25
base: main
Are you sure you want to change the base?
Conversation
Using the create_workspace.sh script, and remove the minimal_pkg as it is not used here.
Command: git clone --recursive https://github.com/stereolabs/zed-ros2-wrapper.git Branch: master Reference: https://github.com/stereolabs/zed-ros2-wrapper/tree/5368f79d200ca89ecb39935751563ef64a4c8bf0
We will use the settings provide by the file `zed_macro.urdf.xacro`.
Gazebo needs the inertial tags for every link. Just as you have <collision> and <visual>, you need to define <inertial>. Reference: https://answers.gazebosim.org/question/21647
Add the usbutils package to enable the `lsusb` command, use this command to verify the connection with ZED. Install the necessary ZED dependencies by running the `rosdep` command to get the package lists.
For convenience, I mount all `/dev` into the container.
Migrate the settings to ROS2, ensuring the topic name aligns with the ZED SDK. Keep in mind that the gazebo plugin only publishes essential data; other data, such as "Gray Image," will need to be published by a separate node (which hasn't been implemented yet). Reference: https://github.com/ros-simulation/gazebo_ros_pkgs/wiki
Implement the `sim_mode` flag to seamlessly toggle between simulation and real modes. (Use Gazebo for simulation.) Command: - Launch ZED node only - ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zed2i sim_mode:=true - Launch ZED node with RVIZ - ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zed2i sim_mode:=true
Considering the information displayed during ZED launch, where the ZED SDK captures frames in 1280x720 and publishes them in 640x360, it would be advisable to configure the Gazebo plugin to publish frames in the same resolution, i.e., 640x360.
Copy the values from the default configuration file of the ZED SDK for accurate adjustment. File path: /usr/local/zed/settings Reference: https://support.stereolabs.com/hc/en-us/articles/360007497173-What-is-the-calibration-file
Due to the variance in the base image for Isaac Sim compared to the original one, I've crafted a new Dockerfile to generate the image for Isaac Sim.
To utilize ros2_bridge, install ROS core locally. Note that we are not employing the official ROS base image, it's necessary to install tzdata as a dependency. Reference: - https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html - https://www.stereolabs.com/docs/isaac-sim/ros2_integration
As the ZED plugin relies on dynamic libraries supplied by the ZED SDK, installing the ZED SDK is essential. However, there's no need to worry about the CUDA version, as we won't be running the SDK within the Isaac Sim container. Reference: - https://github.com/stereolabs/zed-ros2-wrapper/blob/282f879ba05cbbc845e519683a11033c1c2eb6f0/docker/Dockerfile.u22-cu117-humble-devel - https://www.stereolabs.com/docs/isaac-sim/ros2_integration
As we're utilizing the user `ubuntu`, there are several files and folders that require attention regarding permissions and ownership. I attempted to change the ownership within the `/isaac-sim` folder, but the process consistently terminates when executing the commands. I'm uncertain about the reason, so I'll address each case I encounter and post the issue for further investigation. Error message: ``` [Error] [carb.scripting-python.plugin] PermissionError: [Errno 13] Permission denied: '/isaac-sim/kit/exts/omni.gpu_foundation/cache/nv_shadercache' At: /isaac-sim/kit/python/lib/python3.10/os.py(230): makedirs /isaac-sim/kit/exts/omni.gpu_foundation/omni/gpu_foundation_factory/impl/foundation_extension.py(91): setup_shadercache_locations /isaac-sim/kit/exts/omni.gpu_foundation/omni/gpu_foundation_factory/impl/foundation_extension.py(142): on_startup /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(164): _startup_ext /isaac-sim/kit/kernel/py/carb/profiler/__init__.py(85): wrapper /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(224): startup /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension PythonExtension.cpp::startup()(2): <module> [Error] [omni.ext.plugin] [ext: omni.gpu_foundation-0.0.0] Failed to startup python extension. ```
Considering that the folders mounted by Docker are owned by the root, it's necessary to change the ownership to enable reading and writing data inside them.
Error message: ``` [Error] [carb.events.python] PermissionError: [Errno 13] Permission denied: '/isaac-sim/exts/omni.isaac.synthetic_recorder/data/last_config.json' ```
This example is extracted from the ZED tutorial. Refer to the documentation for more comprehensive details. Reference: https://www.stereolabs.com/docs/isaac-sim/ros2_integration
Utilize the `sim_platform` flag to specify the simulation platform. It defaults to Isaac Sim.
Hi, I've resolved the issues we discussed in our private meeting last week. Now, we can simulate ZED in either Gazebo or Isaac Sim, and you can toggle between them using a single flag. I've written a Dockerfile for Isaac Sim, so you can use that container to start up Isaac Sim directly. Alternatively, you can use the Omniverse Launcher to launch the Isaac Sim; however, please note that you need to install ros2-humble and ZED SDK locally to use the Omniverse Launcher. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, thanks for opening this PR!
Most changes look good to me, and I have left some comments. I haven't run the code yet, but will run it during the next review. Thanks!
zed_ws/.gitignore
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
> ```bash= | ||
> # Outside the container | ||
> cd ~/ros2-agv-essentials/zed_ws/docker | ||
> docker compose build zed-isaac-sim |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe setting up a free NGC API key is required before this step
Consider adding this link in the README.
a8b1110
to
dbd12e4
Compare
Closes: #23