Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workspace with ZED for Simulation and Real Camera #25

Open
wants to merge 30 commits into
base: main
Choose a base branch
from

Conversation

YuZhong-Chen
Copy link
Collaborator

Closes: #23

Using the create_workspace.sh script, and remove the minimal_pkg as it is not used here.
We will use the settings provide by the file `zed_macro.urdf.xacro`.
Gazebo needs the inertial tags for every link. Just as you have <collision> and <visual>, you need to define <inertial>.

Reference: https://answers.gazebosim.org/question/21647
Add the usbutils package to enable the `lsusb` command, use this command to verify the connection with ZED.

Install the necessary ZED dependencies by running the `rosdep` command to get the package lists.
For convenience, I mount all `/dev` into the container.
Migrate the settings to ROS2, ensuring the topic name aligns with the ZED SDK.
Keep in mind that the gazebo plugin only publishes essential data; other data, such as "Gray Image," will need to be published by a separate node (which hasn't been implemented yet).

Reference: https://github.com/ros-simulation/gazebo_ros_pkgs/wiki
Implement the `sim_mode` flag to seamlessly toggle between simulation and real modes.
(Use Gazebo for simulation.)

Command:
- Launch ZED node only
    - ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zed2i sim_mode:=true
- Launch ZED node with RVIZ
    - ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zed2i sim_mode:=true
Considering the information displayed during ZED launch, where the ZED SDK captures frames in 1280x720 and publishes them in 640x360, it would be advisable to configure the Gazebo plugin to publish frames in the same resolution, i.e., 640x360.
Copy the values from the default configuration file of the ZED SDK for accurate adjustment.

File path: /usr/local/zed/settings
Reference: https://support.stereolabs.com/hc/en-us/articles/360007497173-What-is-the-calibration-file
@YuZhong-Chen YuZhong-Chen self-assigned this Jan 19, 2024
Due to the variance in the base image for Isaac Sim compared to the original one, I've crafted a new Dockerfile to generate the image for Isaac Sim.
To utilize ros2_bridge, install ROS core locally.
Note that we are not employing the official ROS base image, it's necessary to install tzdata as a dependency.

Reference:
- https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html
- https://www.stereolabs.com/docs/isaac-sim/ros2_integration
As the ZED plugin relies on dynamic libraries supplied by the ZED SDK, installing the ZED SDK is essential. However, there's no need to worry about the CUDA version, as we won't be running the SDK within the Isaac Sim container.

Reference:
- https://github.com/stereolabs/zed-ros2-wrapper/blob/282f879ba05cbbc845e519683a11033c1c2eb6f0/docker/Dockerfile.u22-cu117-humble-devel
- https://www.stereolabs.com/docs/isaac-sim/ros2_integration
As we're utilizing the user `ubuntu`, there are several files and folders that require attention regarding permissions and ownership.

I attempted to change the ownership within the `/isaac-sim` folder, but the process consistently terminates when executing the commands. I'm uncertain about the reason, so I'll address each case I encounter and post the issue for further investigation.

Error message:
```
[Error] [carb.scripting-python.plugin] PermissionError: [Errno 13] Permission denied: '/isaac-sim/kit/exts/omni.gpu_foundation/cache/nv_shadercache'
At:
  /isaac-sim/kit/python/lib/python3.10/os.py(230): makedirs
  /isaac-sim/kit/exts/omni.gpu_foundation/omni/gpu_foundation_factory/impl/foundation_extension.py(91): setup_shadercache_locations
  /isaac-sim/kit/exts/omni.gpu_foundation/omni/gpu_foundation_factory/impl/foundation_extension.py(142): on_startup
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(164): _startup_ext
  /isaac-sim/kit/kernel/py/carb/profiler/__init__.py(85): wrapper
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(224): startup
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>
[Error] [omni.ext.plugin] [ext: omni.gpu_foundation-0.0.0] Failed to startup python extension.
```
Considering that the folders mounted by Docker are owned by the root, it's necessary to change the ownership to enable reading and writing data inside them.
Error message:
```
[Error] [carb.events.python] PermissionError: [Errno 13] Permission denied: '/isaac-sim/exts/omni.isaac.synthetic_recorder/data/last_config.json'
```
This example is extracted from the ZED tutorial. Refer to the documentation for more comprehensive details.

Reference: https://www.stereolabs.com/docs/isaac-sim/ros2_integration
Utilize the `sim_platform` flag to specify the simulation platform. It defaults to Isaac Sim.
@YuZhong-Chen
Copy link
Collaborator Author

Hi, I've resolved the issues we discussed in our private meeting last week. Now, we can simulate ZED in either Gazebo or Isaac Sim, and you can toggle between them using a single flag. I've written a Dockerfile for Isaac Sim, so you can use that container to start up Isaac Sim directly. Alternatively, you can use the Omniverse Launcher to launch the Isaac Sim; however, please note that you need to install ros2-humble and ZED SDK locally to use the Omniverse Launcher.

Copy link
Owner

@j3soon j3soon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for opening this PR!

Most changes look good to me, and I have left some comments. I haven't run the code yet, but will run it during the next review. Thanks!

zed_ws/docker/Dockerfile Show resolved Hide resolved
zed_ws/docker/compose.yaml Show resolved Hide resolved
zed_ws/docker/compose.yaml Show resolved Hide resolved
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All .gitignore entries and .gitkeep files in commit 5bc3e85 would be unnecessary after fixing #31

zed_ws/docker/Dockerfile.isaac-sim Outdated Show resolved Hide resolved
> ```bash=
> # Outside the container
> cd ~/ros2-agv-essentials/zed_ws/docker
> docker compose build zed-isaac-sim
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe setting up a free NGC API key is required before this step

Ref: https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_container.html#container-deployment

Consider adding this link in the README.

@j3soon j3soon force-pushed the main branch 2 times, most recently from a8b1110 to dbd12e4 Compare October 23, 2024 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setup ZED sensor and model for Gazebo Simulation
2 participants