integrations/ncnn/

[giscus[bot]]

integrations/ncnn/

Uncover how to improve your Ultralytics YOLOv8 model's performance using the NCNN export format that is suitable for devices with limited computation resources.

https://docs.ultralytics.com/integrations/ncnn/?h=ncnn

[nisankarsan]

When using Raspberry Pi, the Onnx model gave slower results than the normal model. Does this NCNN model give faster results when testing it on Raspberry using the weights of the model I trained before?

[pderrenger]

Hey there! 😊

Yes, exporting your model to NCNN format can indeed offer faster inference times on devices like the Raspberry Pi, especially when compared to using the ONNX model directly. NCNN is optimized for high performance on mobile and embedded devices, making it a great choice for scenarios where computational resources are limited.

To export your trained YOLOv8 model to NCNN and test it on your Raspberry Pi, you can follow these steps:

1. First, ensure you have the latest ultralytics package installed:

   pip install ultralytics --upgrade

2. Export your trained model to NCNN format:

   from ultralytics import YOLO
   
   # Load your trained model
   model = YOLO('path/to/your/trained_model.pt')
   
   # Export the model to NCNN format
   model.export(format='ncnn')

3. Once exported, you can deploy the NCNN model on your Raspberry Pi. The performance improvement you'll see depends on various factors, including the complexity of your model and the specific Raspberry Pi model you're using.

Keep in mind that quantization and other optimizations available with NCNN can further enhance performance, so it's worth exploring those options too!

Let us know how it goes, and if you have any more questions! 🚀

[nisankarsan]

Interestingly, this new process cut the training time in half on my MacBook, even though it lacks a GPU. I intend to apply the same process to the Raspberry Pi next. Thanks a lot

[pderrenger]

Hey there! 🎉

That's fantastic to hear about your training time improvements on the MacBook! If you're looking to deploy on a Raspberry Pi next, exporting your model to NCNN could indeed offer better performance, especially for devices with limited computational resources like the Raspberry Pi.

Here's a quick example on how to export your trained model to NCNN format:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_model.pt')

# Export the model to NCNN format
model.export(format='ncnn')

This should create an NCNN model optimized for your Raspberry Pi. Given NCNN's efficiency on mobile and embedded devices, you're likely to see improved inference speeds! 🚀

Let us know how it goes, and if there's anything else we can help with. Happy deploying!

[nisankarsan]

Do I do same conversion for YOLOV5 too? And how can I measure my ncnn model accuracy?

When I tried to work on raspberry pi, I had to do more, which are these steps:
git clone https://github.com/Tencent/ncnn.git
cd ncnn

mkdir -p build
cd build
cmake -DNCNN_OPENMP=ON -DNCNN_STDIO=ON -DNCNN_STRING=ON -DNCNN_PIXEL=ON -DNCNN_PIXEL_ROTATE=ON -DNCNN_PIXEL_AFFINE=ON -DNCNN_VULKAN=OFF ..
make -j$(nproc)
sudo make install

pip3 install --upgrade pip
pip3 install numpy opencv-python-headless

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

[pderrenger]

@nisankarsan hey there! 😊

Yes, you can follow a similar process for converting YOLOv5 models to NCNN format. For measuring the accuracy of your NCNN model, you might consider running inference on a validation set and comparing the predictions against the ground truth annotations. Here's a quick example of how you could do this in Python, assuming you have the necessary setup:

from ultralytics import YOLO

# Load your NCNN model
ncnn_model = YOLO('path/to/your/ncnn_model')

# Run inference on your validation set
results = ncnn_model('path/to/validation/images')

# Compare results.boxes or results.masks etc. with your ground truth annotations

Regarding the additional steps you mentioned for setting up on Raspberry Pi, those are indeed necessary for compiling NCNN from source and preparing your environment. This setup ensures that NCNN is optimized for your specific hardware, potentially leading to better performance. 🚀

Hope this helps, and best of luck with your Raspberry Pi deployment! Let us know if you have any more questions.

[glenn-jocher]

NCNN is crazy fast! On our GitHub macos-14 ARM64 M1 runners it is the fastest format of all. See our GitHub actions for an example speed table at https://github.com/ultralytics/ultralytics/actions/runs/8126355293/job/22209982702

[woerstcase]

Hey there!

I tried and on my Mac M2 Mac the inference time got from around 60ms to 30ms. And in a custom Model even faster.

But now i have the "problem" that the BBoxes are very flickering. Is there a good technique to make the bbox drawing even more smooth.
I tried some approaches like drawing over 5 Frames for e.g. But if i have same classes on the image the bbox starts wandering trough the image.

best regards

[glenn-jocher]

Hey there! 👋

Great to hear about the speed improvements on your Mac M2! For smoothing out the flickering BBoxes, one approach is to apply a moving average or some form of temporal smoothing to the bounding box coordinates across frames. This helps in making the position more stable. A simple example could be:

# Assuming bbox is a list of bounding box coordinates for the current frame
# prev_bboxes is a list of previous bounding boxes (could store N previous states)

smoothed_bbox = current_bbox * alpha + prev_bbox * (1 - alpha)

alpha is a weighting factor that determines the balance between the current and previous frame's information (e.g., alpha=0.6). This approach assumes prev_bbox is the bounding box coordinate from the previous frame. Adjust alpha based on your preference for smoothness vs. responsiveness.

If you're handling multiple objects of the same class, consider incorporating an object tracker that assigns unique IDs to each object. This way, you can apply the smoothing individually to each tracked object, preventing the "wandering" effect across objects of the same class.

Hope this helps! Let us know if you have any more questions. 😊

Best regards

[woerstcase]

Worked for me! With my custom Model the inference time is around 15ms!

This is my smoothing function:

#Smoothing function
def smooth_bbox(current_bbox, prev_bbox):
if prev_bbox is None:
return current_bbox
smoothed_bbox = [(c * alpha + p * (1 - alpha)) for c, p in zip(current_bbox, prev_bbox)]
return smoothed_bbox

....#later in the code i use it like this

    for box, confidence, cls in zip(boxes, confidences, classes):
        x1, y1, x2, y2 = map(int, box[:4])  # Extract bounding box coordinates
        class_name = names[int(cls)]
        
        # Generate a unique key for each bbox assuming no tracking ID available
        bbox_key = f"{x1}_{y1}_{x2}_{y2}"
        
        # Apply smoothing
        current_bbox = [x1, y1, x2, y2]
        if bbox_key in prev_bboxes:
            smoothed_coords = smooth_bbox(current_bbox, prev_bboxes[bbox_key])
        else:
            smoothed_coords = current_bbox
        
        # Update prev_bboxes
        prev_bboxes[bbox_key] = smoothed_coords
        
        # Calculate the center of the smoothed bbox
        center_x = int((smoothed_coords[0] + smoothed_coords[2]) / 2)
        center_y = int((smoothed_coords[1] + smoothed_coords[3]) / 2)

best regards

[glenn-jocher]

Hey there!

Great to see your impressive inference times, and thanks for sharing your smoothing function! 🚀 It looks like a solid approach. To address the flickering and "wandering" of bounding boxes for objects of the same class, you might consider integrating an object tracking algorithm like SORT or DeepSORT that assigns unique IDs to each object. This way, you can ensure that the smoothing you apply is object-specific.

Here's a small tweak to include object tracking by simply creating a unique key based on the object's class and center coordinates, assuming you'd implement tracking to assign IDs:

# Assuming bbox_key generation after object tracking assigns a unique track_id
bbox_key = f"{class_name}_{track_id}"

# Use this key for previous bbox retrieval and smoothing application

This tweak can help keep the smoothing consistent across consecutive frames for the same object, reducing "wandering."

Hope this helps make your bboxes smooth as silk! Let me know if you have other questions or if there's more we can help with.

Happy coding! 😊

Best regards

[abdullah000123]

Can we directly export model in ncnn while training or we only can convert model which i obtained after training ie converting from pt to ncnn if we can convert directly while training whats is benefit in case of accuracy and also give me code if we can export directly while training model

[pderrenger]

@abdullah000123 you can only export a YOLOv8 model to NCNN format after training is complete. Direct export during training is not supported. Exporting post-training ensures that the model is fully trained and optimized before conversion, which does not impact accuracy. For exporting, you can use the following command after training:

from ultralytics import YOLO
model = YOLO("yolov8n.pt")
model.export(format="ncnn")

For more details, please refer to the NCNN export guide.

[amrboef66]

Hellooo, im trying to run interference on a raspberry Pi 5, now indeed NCNN has the fastest time, but i need it to be even quicker, so i tried reducing the image format from 640 to 480, since my model does not require such a large image size, but when i do that it misidentifies and puts bounding boxes everywhere on screen. How do i reduce image size while still allowing the model to run accurately?

Also, do you have any thoughts on how i can make it quicker, perhaps a different method? I am already using Nano model and int8 quantisation.

Thank you so much!

[glenn-jocher]

@amrboef66 to improve accuracy with a smaller image size, consider fine-tuning your model on images of the desired size. Additionally, you might explore using a more efficient model architecture or further optimizing your inference pipeline.

[trungnb34]

Hi everyone
NCNN can be inferred from the int8 model
I have a question. when converting Yolo model to int8 . Can ultralytics support it and Does the accuracy decrease much?
Thanks

[glenn-jocher]

@trungnb34 yes, Ultralytics supports exporting YOLO11 models to NCNN with INT8 quantization using the export(format="ncnn", int8=True) option. Accuracy may decrease depending on the task and dataset, as quantization reduces precision to optimize performance. For details, refer to the Export to NCNN guide.

[robert-hardman]

Hello
ultralytics==8.3.53
Like it doesn't support int8 yet

File "C:\Users***\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 253, in call
validate_args(fmt, self.args, fmts_dict["Arguments"][flags.index(True) + 1])
File "C:\Users***AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 144, in validate_args
assert arg in valid_args, f"ERROR \u274c\ufe0f argument '{arg}' is not supported for format='{format}'"
^^^^^^^^^^^^^^^^^
AssertionError: ERROR \u274c\ufe0f argument 'int8' is not supported for format='ncnn'

[glenn-jocher]

@robert-hardman thank you for raising this issue. Currently, INT8 quantization is not supported for NCNN exports in Ultralytics. For updates on supported features, keep an eye on our Export documentation. Let us know if you have further questions!