integrations/sony-imx500/

[giscus[bot]]

integrations/sony-imx500/

Learn to export Ultralytics YOLOv8 models to Sony's IMX500 format to optimize your models for efficient deployment.

https://docs.ultralytics.com/integrations/sony-imx500/

[UltralyticsAssistant]

👋 Hello, thank you for your interest in Ultralytics and exploring integrations with the Sony IMX500! 🚀

For those looking to dive deeper into this topic, the Ultralytics Docs are a great starting point. They contain sections dedicated to various integrations, including Sony IMX500, with detailed instructions on how to export your YOLO models efficiently.

If you're here to report a 🐛 bug, please provide a minimum reproducible example. This will help us better understand and assist with the issue you're facing.

For general usage, upgrading to the latest version of Ultralytics might resolve some issues you encounter. Make sure your environment is updated with the latest package:

pip install -U ultralytics

For interactive discussions, consider joining the Ultralytics community on Discord 🎧, our Discourse forum, or participate in threads on our Subreddit.

Our team will assist you soon, but for now, these resources should help you get started! 😊

Environments

Feel free to run YOLO in any of these verified environments with preinstalled dependencies:

• Notebooks with free GPU:
• Google Cloud and Amazon environments as per GCP Quickstart Guide and AWS Quickstart Guide
• Docker Image: with guidance in our Docker Quickstart Guide

Status

Our continuous integration (CI) ensures all models and integrations, including Sony IMX500, are tested across platforms. Check our CI status here for the latest updates:

[alexander-sony]

I ran into some python dependency issues when trying to run the 1st example. i started with python 3.12. ultralytics asks for tensorflow 2.12 which seems to clash with the requirements of the converter tools. Downgrading python to 3.11 is not enough. Tensorflow version must be higher than 2.12.

❯ uv add "model-compression-toolkit==2.1.1" "sony-custom-layers==0.2.0" "tensorflow==2.12.0" "imx500-converter[pt]==3.14.3"
  × No solution found when resolving dependencies for split (python_full_version >= '3.12' and platform_system == 'Darwin' and sys_platform == 'darwin'):
  ╰─▶ Because tensorflow==2.12.0 has no wheels with a matching Python version tag and your project depends on tensorflow==2.12.0, we can conclude that your project's
      requirements are unsatisfiable.

# downgraded python to 3.11

❯ uv add "model-compression-toolkit==2.1.1" "sony-custom-layers==0.2.0" "tensorflow==2.12.0" "imx500-converter[pt]==3.14.3"
  × No solution found when resolving dependencies for split (python_full_version == '3.11.*' and platform_system == 'Darwin' and sys_platform == 'darwin'):
  ╰─▶ Because onnxruntime>=1.17.0,<=1.17.3 depends on numpy>=1.24.2 and only the following versions of onnxruntime are available:
          onnxruntime<=1.17.0
          onnxruntime==1.17.1
          onnxruntime==1.17.3
          onnxruntime>1.18.dev0
      we can conclude that onnxruntime>=1.17.0,<1.18.dev0 depends on numpy>=1.24.2.
      And because uni-pytorch==1.10.0 depends on onnxruntime>=1.17.0,<1.18.dev0, we can conclude that uni-pytorch==1.10.0 depends on numpy>=1.24.2.
      And because imx500-converter[pt]==3.14.3 depends on uni-pytorch==1.10.0 and tensorflow==2.12.0 depends on numpy>=1.22,<1.24, we can conclude that tensorflow==2.12.0
      and imx500-converter[pt]==3.14.3 are incompatible.
      And because your project depends on imx500-converter[pt]==3.14.3 and tensorflow==2.12.0, we can conclude that your project's requirements are unsatisfiable.
...

I tested on ubuntu 22 machine. Found solution after some testing. Here is the project file.

[project]
name = "ultralytics-imx500"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = "~=3.11.0"
dependencies = [
    "imx500-converter[pt]==3.14.3",
    "model-compression-toolkit==2.1.1",
    "sony-custom-layers==0.2.0",
    "tensorflow~=2.14.0",
    "ultralytics>=8.3.31",
]

It could also be good to mention that converter installation at some point might ask for sudo pwd since it requires java.

[glenn-jocher]

@alexander-sony thank you for sharing your solution and insights on resolving the dependency issues. It's great to see the community collaborating to overcome challenges. If you have further questions or need additional assistance, feel free to ask here.

[JochenSchmitz]

Hello Alexander-Sony,
I have the problem with the sudo password. Unfortunately, I don't know where to enter it. I'm not asked either. I'm using the current Ultralytics Docker container on Ubuntu 22. But installing with pip also gives the same error. Here is the complete output. Can you please give me a tip.
Thank you
Jochen

root@5c41d2cb95d6:/datasets/dataset# yolo export model=best.pt format=imx
Ultralytics 8.3.36 🚀 Python-3.11.10 torch-2.5.0+cu124 CPU (Intel Core(TM) i5-9500T 2.20GHz)
WARNING ⚠️ IMX only supports int8 export, setting int8=True.
WARNING ⚠️ INT8 export requires a missing 'data' arg for calibration. Using default 'data=coco8.yaml'.
Model summary (fused): 168 layers, 3,007,208 parameters, 0 gradients, 8.1 GFLOPs

PyTorch: starting from 'best.pt' with input shape (1, 3, 416, 416) BCHW and output shape(s) ((1, 3549, 4), (1, 3549, 8)) (5.9 MB)
/opt/conda/lib/python3.11/site-packages/sony_custom_layers/pytorch/object_detection/nms.py:120: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
@torch.library.impl_abstract(MULTICLASS_NMS_TORCH_OP_QUALNAME)
IMX: export failure ❌ 3.7s: [Errno 2] No such file or directory: 'sudo'
Traceback (most recent call last):
File "/ultralytics/ultralytics/engine/exporter.py", line 1123, in export_imx
raise FileNotFoundError
FileNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/bin/yolo", line 8, in
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/ultralytics/ultralytics/cfg/init.py", line 969, in entrypoint
getattr(model, mode)(**overrides) # default args from model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ultralytics/ultralytics/engine/model.py", line 734, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ultralytics/ultralytics/engine/exporter.py", line 381, in call
f[13], _ = self.export_imx()
^^^^^^^^^^^^^^^^^
File "/ultralytics/ultralytics/engine/exporter.py", line 146, in outer_func
raise e
File "/ultralytics/ultralytics/engine/exporter.py", line 141, in outer_func
f, model = inner_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ultralytics/ultralytics/engine/exporter.py", line 1125, in export_imx
subprocess.run(["sudo", "apt", "install", "-y", "openjdk-17-jdk", "openjdk-17-jre"], check=True)
File "/opt/conda/lib/python3.11/subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/conda/lib/python3.11/subprocess.py", line 1955, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'sudo'
root@5c41d2cb95d6:/datasets/dataset#

[JochenSchmitz]

Hello, I found it myself. Installing openjdk-17-jdk before starting the export prevents the problem.
Best regards
Jochen

[glenn-jocher]

Thank you for sharing your solution, Jochen! Pre-installing openjdk-17-jdk is an excellent approach to prevent the issue. If you have further questions or run into additional challenges, feel free to ask here.

[ARCHANGELOVE]

Hi there it doesnt support for yolov8n-obb model because it shows me
TypeError: decode_bboxes() got an unexpected keyword argument 'xywh'

[glenn-jocher]

@ARCHANGELOVE it seems you're encountering an issue with the decode_bboxes() function in the YOLOv8n-obb model. This error might be due to an unsupported or incorrect argument being passed. Please ensure that you're using the latest version of the Ultralytics package and verify the function's parameters in the documentation. If the problem persists, consider opening an issue with detailed information on your setup and code snippet for further assistance.

[ARCHANGELOVE]

Yep noted but as i know the box decoding for OBB is differ from AABB because the model convertion using yolov8n.pt works fine and the conversion was successful

yolo export model=yolov8n-obb.pt format=imx

i face the error when i convert the obb model

my ultralytics vertion is in latest aswell

[glenn-jocher]

It seems like the decode_bboxes() function currently doesn't support the OBB model conversion for the yolov8n-obb.pt to IMX500 format. This might be due to differences in bounding box representation between OBB and AABB models. Please ensure you're using the latest version of the Ultralytics package and consider checking the latest documentation or opening an issue on GitHub for further assistance.

[marcivanmanalac]

Hello there,

Are there any tips on IMX exporting for later YOLO models such as V11?

[glenn-jocher]

@marcivanmanalac to export later YOLO models like YOLOv11 to IMX format, you can follow a similar process as mentioned for YOLOv8, using the model.export(format="imx") function with INT8 quantization. Ensure your model is well-prepared by training effectively and refer to the Sony IMX500 Export Guide for detailed instructions and deployment steps. Let us know if you need further assistance!

[marcivanmanalac]

Hi Glenn,

Yes, that was what I had attempted in the following format success = model.export(mode=model, format="imx", int8=True)

but I get a �[34m�[1mIMX:�[0m export failure ❌ 0.0s: IMX export is only supported for YOLOv8 detection models error from line 1111 on from export_imx( ).

[glenn-jocher]

@marcivanmanalac the error occurs because IMX export currently supports only YOLOv8 detection models and is not implemented for YOLOv11. Please refer to the Sony IMX500 Export guide for supported workflows. YOLOv11-specific export capabilities may require future updates.

[alexander-sony]

There is a problem with the instructions to install picamera2, it seems to create issues for the imx500 examples in https://github.com/raspberrypi/picamera2/tree/main/examples/imx500. Has it been tested with existing imx500 examples in the picamera2 repo?

git clone -b next https://github.com/raspberrypi/picamera2
cd picamera2 && pip install -e . --break-system-packages

Proper picamera2 installation is with apt, see https://github.com/raspberrypi/picamera2

[glenn-jocher]

@alexander-sony thank you for highlighting this. The installation via pip install -e . in the guide corresponds to the use of the development branch (next) in specific scenarios involving experimental features. For a stable installation, we recommend following the official picamera2 instructions using apt. We'll review the documentation to include clarifications on when each method should be used. Appreciate your feedback!

[alexander-sony]

problem with ultralytics version >8.3.34, the instruction: yolo predict model=yolov8n_imx_model source='https://ultralytics.com/images/bus.jpg' is generating

yolo predict model=yolov8n_imx_model source=datasets/bus.jpg
WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'.
Ultralytics 8.3.35 🚀 Python-3.11.8 torch-2.5.1+cu124 CUDA:0 (NVIDIA RTX 6000 Ada Generation, 48632MiB)
Loading yolov8n_imx_model for ONNX Runtime inference...
Preferring ONNX Runtime AzureExecutionProvider
Loading yolov8n_imx_model/yolov8n_imx.onnx for ONNX IMX inference...

...        

  File "/storage/alex/Projects/tutorials/aitrios-rpi-tutorials-ai-model-training-dev/notebooks/yolo-ultralytics-ppe/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
    return self._sess.run(output_names, input_feed, run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: input for the following indices
 index: 3 Got: 480 Expected: 640
 Please fix either the inputs/outputs or the model.

[glenn-jocher]

@alexander-sony the error indicates a mismatch between the input dimensions expected by the model and the data provided. Ensure the input image matches the expected size (default: 640x640) by resizing or specifying imgsz=640 during the export and prediction processes. Let me know if additional guidance is needed.

[HelloCigar]

Can we do face recognition(not just detection) with this approach?

[glenn-jocher]

@HelloCigar face recognition is out of scope for YOLO models, which are designed for object detection, classification, segmentation, and pose estimation. For facial recognition tasks, consider exploring specialized libraries like FaceNet or Mediapipe. Let me know if you have any YOLO-specific questions!

[HelloCigar]

https://huggingface.co/arnabdhar/YOLOv8-Face-Detection
i've been messing around this model, it does seem to do face recognition from what I've seen. I labeled 3 classes(names) and annotated images. Seems to work fine? But am I missing something? What are the disadvantages of this approach?

[glenn-jocher]

@HelloCigar using YOLOv8 for face classification (labeling faces with names) can work if your dataset is well-annotated. However, this approach is not true "face recognition" as it lacks the specialized feature extraction and matching capabilities of dedicated face recognition models like FaceNet. The main disadvantage is that YOLOv8 may struggle with nuanced identity recognition compared to models tailored for such tasks. Let me know if you need further clarification!

[dave-ct]

Great guide and I managed to get my first model trained and converted to run on the imx500 on a raspberry pi4 for some custom object detection. Initially I trained on the yolov8s but when I tried to export it it did not work. I then trained the yolov8n with the same data and it worked fine (doubled checked and the yolov8s defualt does not export either) Is there a way to get the yolov8s models to export so I can run it on the imx500 with the tools or does it only work for the yolov8n?

I am new to the Yolov8 models and the imx500 so was just wondering (the s version performed much better than the n version on my dataset) as once I get the yolov8n packaged up on the raspberry PI it seems to be only about 2MB is size so with the input tensors would think it could handle a bigger model as I think the imx500 has 8MB available for the model and input tensors.

[glenn-jocher]

@dave-ct thank you for your detailed feedback! Currently, the IMX500 export supports YOLOv8 detection models like YOLOv8n but does not extend to YOLOv8s. This limitation is related to the model's architecture and compatibility with the IMX500 conversion tools. While the IMX500 hardware has sufficient memory, its constraints on model complexity and processing may require further optimizations or updates to support larger models like YOLOv8s in the future. You can stay updated via our documentation or explore adjustments to your dataset to improve YOLOv8n performance.

[HalidSaglam]

Hello @dave-ct , I was able to convert the yolov8n model into a suitable format for the IMX500 and run it successfully as well. However, I couldn’t convert any models other than yolov8n. Have you tried converting any models other than yolov8n, and if so, were you successful?