models/rtdetr/

[giscus[bot]]

models/rtdetr/

Discover the features and benefits of RT-DETR, Baidu’s efficient and adaptable real-time object detector powered by Vision Transformers, including pre-trained models.

https://docs.ultralytics.com/models/rtdetr/

[pabentse]

Would love OBB support for RT-DETR!

[glenn-jocher]

Hey there! 👋

Thanks for your interest in OBB support for RT-DETR! Currently, RT-DETR is optimized for axis-aligned bounding boxes, but we're always looking to improve. I'll definitely bring this up with the team to consider for future updates. 🚀

In the meantime, if you're working on a project that requires OBB, you might want to explore using the rtdetr-l.pt or rtdetr-x.pt models with some custom post-processing to approximate OBB. Here's a quick example of how you might load and use these models:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR('rtdetr-l.pt')

# Run inference on an image
results = model('path/to/bus.jpg')

# Post-process results here to approximate OBB...

Stay tuned for updates, and feel free to reach out if you have more questions or ideas! 😊

[pabentse]

What would be the point of approximating OBB in post-process?
And how's the weather over there

[glenn-jocher]

Approximating OBB in post-process can help achieve better alignment for objects that aren't axis-aligned, enhancing detection accuracy in certain applications. As for the weather, it's always changing! If you have more questions about RT-DETR, feel free to ask.

[amarnathreddy0201]

What is the default optimizer RT-DETR use?

[glenn-jocher]

Hey there! ✨ RT-DETR primarily uses the AdamW optimizer, a popular choice for training deep learning models effectively, especially Vision Transformers like RT-DETR. It helps in managing weight decay smoothly, contributing to the model's high accuracy and performance. Here’s a small snippet on how you might set it during training, though with RT-DETR and our API, it's all handled under the hood:

optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)

If you have more questions or need further assistance, feel free to ask! 🚀

[amarnathreddy0201]

Dear glenn,
I want RT-DETR information regarding what are the default parameters?
documentation of RT-DETR for fine running parameters.

Thanks,
Reddy

[glenn-jocher]

@amarnathreddy0201 hey Reddy,

Thanks for your interest in RT-DETR! 🚀 For default parameters and fine-tuning, you can primarily rely on the documentation we've provided. Here's a brief overview:

Default Parameters for RT-DETR:

• Model Scales: We offer RT-DETR-L and RT-DETR-X with respective 53.0% and 54.8% AP on COCO val2017.
• Inference Speed: RT-DETR-L runs at 114 FPS and RT-DETR-X at 74 FPS on a T4 GPU.

Fine-Tuning Parameters:
When fine-tuning RT-DETR models, you can adjust the usual training parameters like:

• epochs: Number of training epochs.
• imgsz: Size of the training images.

Example code for training:

from ultralytics import RTDETR

# Load a model
model = RTDETR('rtdetr-l.pt')

# Fine-tune
results = model.train(data='your_dataset.yaml', epochs=100, imgsz=640)

For RT-DETR, leveraging its Vision Transformers architecture allows for efficient feature processing, making it superb for real-time scenarios. If there's anything specific you're looking for or additional help needed, feel free to ask!

[AadilMdhusain]

Hello there!!🖐️🖐️,
I implemented the RTDETR model by training it on a custom dataset that I got on roboflow containing 1000 images. I used it to perform object detection on live feed that I got from my webcam. It took around 800-1000 ms for inference on each frame due to which I got a laggy feed which was not optimal for the project that I am working on. I saw a video on youtube of a person who implemented the same model and got around 40-50 ms inference time for each frame. The code that I implemented is-

`import cv2
from ultralytics import RTDETR

video_path = 0
cap = cv2.VideoCapture(video_path)

window_width = 640
window_height = 480
cv2.namedWindow(" model Test", cv2.WINDOW_NORMAL)
cv2.resizeWindow(" model Test", window_width, window_height)
model = RTDETR('best.pt')

while cap.isOpened():

#read a frame from the video
[]
success, frame = cap.read()

if success: 
    results = model.predict(frame, conf=0.5)
    annotated_frame = results[0].plot()
    #display the frame
    cv2.imshow(" model Test", annotated_frame)

    #Break the loop if q is pressed
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break
else:
    #Break the loop if the end of the video is reached
    break
#Release the video capture object and close the display window

cap.release()
cv2.destroyAllWindows()
#print`

Here best.pt is the model that I have trained and currently am using. I don't know why I am getting such high inference time. My computer specifications are intel i5 processor and Nvidia gtx 1650 gpu. Is it because my Computer isn't powerful enought to run the model at satisfactory rate or is it because I have trained my model on less data, I have zero idea. Your insight on this will be highly appreciated and would really help me in completing theis project on time.
Thank you

[AadilMdhusain]

Heyy,
So yeah I was trying to run your given code yolo export model=best.pt format engine after training but the given code wasn't working. So I searched and found the page on export (https://docs.ultralytics.com/modes/export/#key-features-of-export-mode). So instead I ran the code model.export(format='onnx') and it worked. Now on the page in the tip box recommended me to save model in ONNX format as it runs on 3x CPU speed. But after exporting it, I wasn't able to deploy or run the model in the code that I pasted on the first comment. So I assume that there is another way to run model saved in onnx format. Can you help me out on how to run the model that I saved in onnx format on a video from the webcam. And also I tried to run the code model.export(format='engine'), but the given error came-

TensorRT: export failure ❌ 119.2s: failed to load ONNX file: runs/detect/train/weights/best.onnx
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-16d684fc9521> in <cell line: 1>()
----> 1 model.export(format='engine')

I only am able to save it in onnx format and not in .engine format as you specified. Would you know the reason behind it. And also can you provide me the way to implement a model saved in onnx format.

I am sorry for bombarding you with so many questions but I only have a few days left to complete my project and need your help urgently. Thank you.

[pabentse]

I'm also unable to export RT-DETR to both TensorRT and CoreML.

[pderrenger]

Hey there! 🚀

It looks like you're hitting a snag while trying to export the RT-DETR models to both TensorRT and CoreML formats. For running models in an ONNX format after export, you can utilize the onnxruntime or similar libraries that support ONNX. Here's a quick example on how you might approach it for a webcam feed:

import cv2
import onnxruntime as ort
import numpy as np

# Initialize ONNX runtime session for the model
session = ort.InferenceSession('path/to/your/model.onnx')

# Setup webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    # Read frame
    ret, frame = cap.read()
    if not ret:
        break
    
    # Preprocess your frame (resize, normalization etc.) according to your model's requirements
    # Example preprocessing:
    input_image = cv2.resize(frame, (640, 640))
    input_image = np.transpose(input_image, (2, 0, 1))  # Change data layout from HWC to CHW
    input_image = np.expand_dims(input_image, axis=0).astype(np.float32)
    
    # Run inference
    input_name = session.get_inputs()[0].name
    results = session.run(None, {input_name: input_image})
    
    # Postprocess the results and display
    # Note: customize the postprocessing and display part according to your model outputs and requirements
    # For now, displaying the original frame
    cv2.imshow('frame', frame)
    
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Regarding the issue with exporting to TensorRT (.engine), the error suggests an issue loading the ONNX model, which is an intermediate step in the conversion to TensorRT format. Make sure:

• Your environment has the necessary dependencies for ONNX and TensorRT conversion.
• The ONNX model has been exported correctly and is compatible with the TensorRT conversion tools.

If you're experiencing difficulties with a specific format, continuing in ONNX is a valid approach, especially given its good CPU inference speed-up as mentioned in our docs.

Sorry for any confusion, and hope this helps you move forward with your project! Feel free to reach out again with any more questions or issues. And no worries about the questions; we're here to help. Good luck! 🍀

[AadilMdhusain]

Is the reason why I'm not able to export in .engine format is because I'm training my project in google colab?? Does it have the necessary dependencies for ONNX to TensorRT conversion? and also I have done some modifications to the code that you gave me above and its working fine, but not giving accurate results. It is detecting a box even when there is no face on the screen. You can see the code below:

import cv2
import onnxruntime as ort
import numpy as np

# Initialize ONNX runtime session for the model
session = ort.InferenceSession('best.onnx')

# Setup webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    # Read frame
    ret, frame = cap.read()
    if not ret:
        break

    # Preprocess your frame (resize, normalization etc.) according to your model's requirements
    # Example preprocessing:
    input_image = cv2.resize(frame, (640, 640))
    input_image = np.transpose(input_image, (2, 0, 1))  # Change data layout from HWC to CHW
    input_image = np.expand_dims(input_image, axis=0).astype(np.float32)

    # Run inference
    input_name = session.get_inputs()[0].name
    results = session.run(None, {input_name: input_image})

    # Extract object detection results
    detections = results[0][0] # Assuming batch size is 1

    # Find the detection with the highest confidence
    max_confidence_detection = max(detections, key=lambda x: x[4])

    # Draw bounding box and class label for the detection with the highest confidence
    x_min, y_min, x_max, y_max, confidence, class_id = max_confidence_detection

    # Convert coordinates from normalized values to pixel values
    height, width = frame.shape[:2]
    x_min, y_min, x_max, y_max = int(x_min * width), int(y_min * height), int(x_max * width), int(y_max * height)

    # Draw bounding box
    cv2.rectangle(frame, (x_min, y_min), (x_max, y_max), (0, 255, 0), 2)

    # Draw class label
    class_label = f"Class {int(class_id)}"  # You can replace this with actual class labels
    cv2.putText(frame, f"{class_label} ({confidence:.2f})", (x_min, y_min - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

    # Display the annotated frame
    cv2.imshow("Model Test", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

  cap.release()
  cv2.destroyAllWindows()

The given code is giving bounding boxes over my webcam live feed, but not accurate bounding boxes. Can you tell me the changes to be made here so that I can get accurate bounding boxes? Thanks in advance. 😊

[pderrenger]

Hey there! 🙌

Indeed, exporting models to .engine format requires specific dependencies that Google Colab might not have by default, specifically for TensorRT. That could be why you're facing issues with the .engine export. Colab environments do support ONNX, which explains why exporting to ONNX worked smoothly.

Your modified code snippet for inference using ONNX is a great start. However, the issue with inaccurate bounding boxes might relate to how detections are processed and visualized. The snippet seems to pick the detection with the highest confidence without filtering by a confidence threshold. Adding a threshold filter and refining how detections are selected and displayed could enhance accuracy.

Regarding running ONNX models, you did it correctly by switching to the ONNX Runtime. Keep in mind, every model might require slightly different preprocessing and postprocessing. Double-checking these details according to the specific needs of your model will help ensure accurate detections are visualized.

For TensorRT (model.export(format='engine')), this process is generally more straightforward on local machines with NVIDIA GPUs and proper TensorRT setup. It can be challenging on Colab due to the mentioned dependency requirements and setup configurations. The error suggests trouble loading the ONNX file before conversion, which might include missing installation steps or version incompatibilities in Colab.

If you're set on using TensorRT for its performance benefits, consider setting up the environment on a local system where you have control over the necessary installations and configurations.

Keep up the great work, and no worries about the questions; it's all part of the learning process! Feel free to reach out if you have more questions. 😊

[PhucPhamSy]

Hi all,
Could you please let me know how to use all parameters in the appendix of the original paper, "https://arxiv.org/pdf/2304.08069"?
Thanks!

[glenn-jocher]

Hey there! 👋

We're really glad you're interested in diving deep into the parameters detailed in the original paper. Specifically, for utilizing parameters as mentioned in the RT-DETR paper ("DETRs Beat YOLOs on Real-time Object Detection"), I recommend focusing on the key components such as the Efficient Hybrid Encoder, IoU-aware Query Selection, and adaptability in Inference Speed adjustments that make RT-DETR efficient.

Details on these components can be extrapolated to parameter usage within your model through our Python API. For instance, when initializing the RT-DETR model or adjusting inference speed, you would consider the different scales of pre-trained models available:

from ultralytics import RTDETR

# For a larger model with potentially higher accuracy but lower FPS (Frames Per Second)
model = RTDETR('rtdetr-l.pt')  

# For an extra-large model prioritizing accuracy over speed
model_x = RTDETR('rtdetr-x.pt')

Adjustments for inference speed and applying IoU-aware query selection can directly correlate with your focus during the .train() or .predict() processes, especially understanding what goes behind each configuration setting in these functions.

Unfortunately, we don't provide a direct mapping for every parameter mentioned in the appendix to code-level argument settings as the implementation abstracts many complexities for user convenience and to maintain the API's simplicity. However, understanding the RT-DETR's key features and translating that understanding into practical API usage is a great way to leverage the power of RT-DETR. 😊

For more tailored guidance, always feel free to ask or refer to our official documentation for specific examples and deeper insights.

Happy coding! 🚀

[GaviraghiElia]

Hi guys! When is RT-DETR 18 due to be implemented? Thanks for your work :))

[glenn-jocher]

Hey there! Thanks for reaching out and for your kind words 😊.

The integration of RT-DETR 18 is something we're definitely excited about, but I don't have a specific timeline for its implementation at the moment. Our team is working hard to bring new features and models to you. I recommend keeping an eye on our GitHub repository for the latest updates!

In the meantime, you might find the pre-trained RT-DETR models currently available quite useful. Here's a quick example on how you can use them for inference:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR('rtdetr-l.pt')

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model('path/to/bus.jpg')

Stay tuned for future updates, and thanks again for supporting our work! 🚀

[lange4531]

Hi there!

I am getting nan values during training when finetuning the rtdetr-l model. I therefore want to modify the learning rate to see if that help, however, using optimizer = torch.optim.AdamW(model.parameters(), lr=0.001) does not seem to do anything the model just use whatever ultralytics sets automatically. Any way to force a custom learning rate when doing finetuning?

[pderrenger]

Hi there!

To enforce a custom learning rate during fine-tuning of the rtdetr-l model, you can adjust the learning rate directly in your training script. Make sure to modify the hyperparameters file associated with the training process. Here's how you might adjust the learning rate:

1. Create a YAML file, like custom_hyperparameters.yaml.
2. In this file, set the initial learning rate (lr0):

   lr0: 0.001  # Initial learning rate

3. Pass this file during training:

   yolo train model=rtdetr-l.pt data=coco8.yaml hyperparams=custom_hyperparameters.yaml

This will ensure that your specific learning rate is used during the training process. Let me know if you need any more help! 😊

[lange4531]

It accepts my custom learning rate now and getting much better training performance so thank you a lot!
When I am saving the new pt file I do it like this: torch.save(model_x.state_dict(), 'fine_tuned_rtdetr_1.pt') but it seems Ultralytics does not accept that once I do rtdetr_detection_model = RTDETR('fine_tuned_rtdetr_1.pt').to(device) and I get key error. What is the correct Ultralytics way of saving a new pt file after custom training?

[amarnathreddy0201]

I think you no need to save with torch. It will save pt by default

[pderrenger]

@amarnathreddy0201 hi there!

It looks like the issue is due to saving only the state_dict() of the model which doesn't include other configurations and metadata required for the RT-DETR model to operate correctly within the Ultralytics ecosystem.

To save and load the model the Ultralytics way after fine-tuning, you should use our API to manage model saving, ensuring all necessary config stays intact. Here's how you can do it:

To Save your fine-tuned model:

model.save('fine_tuned_rtdetr_1.pt')

To Load your model later:

from ultralytics import RTDETR
rtdetr_detection_model = RTDETR('fine_tuned_rtdetr_1.pt').to(device)

This approach saves the model with all configurations needed by Ultralytics for proper functioning. Let me know if there’s anything else you need!

[lange4531]

Do you mean doing just "model_x.save('fine_tuned_rtdetrtop6.pt')"? I also tried that (as in the image provided) but I end up saving the old pt file (non fine-tuned version) essentially just giving it a new filename.

[pderrenger]

Hey there! It seems like you're trying to save the fine-tuned model, but facing an issue where it still saves the original pre-trained weights instead. To ensure you're saving the updated weights, please ensure that your training loop has successfully modified the model's parameters before you attempt to save it.

Here’s how you can correctly save the fine-tuned model in Python using the Ultralytics API:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR('rtdetr-l.pt')

# Train the model
results = model.train(data='coco8.yaml', epochs=100, imgsz=640)

# Save the fine-tuned model
model.save('fine_tuned_rtdetrtop6.pt')

This snippet assumes you've retrained the loaded model. Make sure the .train() method successfully runs and updates the model's weights before running model.save(). If the issue persists, there might be a need to check the update status of the weight tensors directly or further debug the training process. Let me know how it goes! 🚀

[SERPENT47]

WARNING ⚠️ no model scale passed. Assuming scale='l'

How to pass scale?

[glenn-jocher]

@SERPENT47 hello! To specify the scale when using the RT-DETR model, you can pass the scale argument directly in your command. For example, if you want to use the large scale model, you would include scale='l' in your function call or command line. Here’s how you can do it:

Python Example

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model with specified scale
model = RTDETR(pretrained="rtdetr-l.pt", scale='l')

CLI Example

yolo predict model=rtdetr-l.pt scale=l source=path/to/bus.jpg

This will ensure that the model uses the 'large' scale configuration. Let me know if you need further assistance! 😊

[SERPENT47]

import os
from ultralytics import RTDETR

batchSize = 20
numberOfEpochs = 20
imageSize = 640
pretrainedModel = 'rtdetr-l'
customDatasetPath = "data.yaml"
trainResultPath = r"C:\Users\USER\Desktop\Results"
trainResultDirectoryName = 'Result[B'+str(batchSize)+'E'+str(numberOfEpochs)+'M'+pretrainedModel+']'
model = RTDETR(pretrainedModel+'.yaml', scale='l')

Traceback (most recent call last):
File "c:\Users\USER\Desktop\1.Mredul\Codes\train_RE-DETR.py", line 12, in
model = RTDETR(pretrainedModel+'.yaml', scale='l')
TypeError: RTDETR.init() got an unexpected keyword argument 'scale'

What am I doing wrong?

[glenn-jocher]

Hello! It looks like there's a misunderstanding in the usage of the RTDETR class constructor. The scale parameter is not a valid argument for initializing the RTDETR model. Instead, you should directly specify the model scale in the model filename when loading it. Here’s how you can correct your code:

from ultralytics import RTDETR

# Correct way to load a COCO-pretrained RT-DETR Large model
model = RTDETR(pretrained='rtdetr-l.pt')

This will correctly load the large-scale RT-DETR model. If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[SERPENT47]

But as you can see in my previous comment that I already specified the scale in the model filename yet I got the warning message.

Also, Isn’t it possible to build a RTDETR model from scratch using .yaml?

[glenn-jocher]

@SERPENT47 hello! Thanks for reaching out with your concerns. 🌟

Regarding the warning about the model scale, it seems there might have been a misunderstanding or a minor glitch. If you've specified the scale correctly in the model filename (like 'rtdetr-l.pt' for the large scale), you shouldn't receive such a warning. It might help to double-check the exact command or code snippet you're using to ensure everything is correctly set up.

As for building an RT-DETR model from scratch using a .yaml file, currently, our pre-trained models are the primary method for utilizing RT-DETR within the Ultralytics framework. Building from scratch typically involves more complex setups and isn't directly supported through a simple .yaml configuration for RT-DETR models. However, you can certainly customize the training process and other parameters using the provided pre-trained models and your specific dataset.

If you need further clarification or assistance, feel free to ask. We're here to help! 😊

[ronit450]

Issue While performing Detection After the model.

I have trained a model on two classes and when I try to run inference on it, it gives me the following error:

325     for c in boxes.cls.unique():
326         n = (boxes.cls == c).sum()  # detections per class

--> 327 log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, "
328 return log_string

File "/home/biosense/anaconda3/envs/rana/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 172, in write_results
log_string += result.verbose()
File "/home/biosense/anaconda3/envs/rana/lib/python3.8/site-packages/ultralytics/engine/results.py", line 327, in verbose
log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, "
KeyError: 2

I have specefically given 2 classes, but still this is the error I am encountering

[pderrenger]

Hello,

It seems like you're encountering a KeyError during inference, which typically occurs if there's a mismatch between the class indices and the labels provided in your dataset configuration. Here are a few steps to troubleshoot and resolve this issue:

1. Check Class Labels: Ensure that your dataset's YAML file correctly defines all class labels and that the indices match those used during training. The class indices should start from 0 and continue sequentially.

2. Model Configuration: Verify that the model configuration (.yaml file used during training) correctly references the number of classes and the path to your dataset's YAML file.

3. Post-Training Modifications: If any modifications were made to the model or dataset labels after training, ensure these are consistent across your training and inference setups.

4. Re-Export the Model: If you've made changes to the dataset or model configuration, consider re-exporting your model to ensure all settings are correctly applied.

If the issue persists after checking these points, could you please provide more details about your training setup and the exact command or script used for inference? This will help in diagnosing the problem more effectively.

Thank you for reaching out, and I look forward to helping you resolve this issue!

[ALMAGEST12]

I'm trying to run the model with coco dataset. But i always got warnings, such as :

• WARNING ⚠️ no model scale passed. Assuming scale='l'.
• WARNING ⚠️ No labels found in C://....
  after that the system make cache file at my labels dir.
  This is my .yaml file :
  train: C:/Users/ASUS/rt-detr_custom/images/train_img/
  val: C:/Users/ASUS/rt-detr_custom/images/val_img/ # directory to val images

COCO json annotation files

train_ann: C:/Users/ASUS/rt-detr_custom/labels/instances_train.json
val_ann: C:/Users/ASUS/rt-detr_custom/labels/instances_val.json

Classes

nc: 12
names:
[
1: 1-motorcycle,
2: 2-car,
3: 3-passenger_car,
4: 4-pickup,
5: 5a-small_bus,
6: 5b-big_bus,
7: 6a-light_truck,
8: 6b-medium_truck,
9: 7a-heavy_truck,
10: 7b-double_truck,
11: 7c-trailer_truck,
12: 8-no_motor_vehicle,
]

I don't know what is the main problem. Your insight on this will be highly appreciated and would really help me in completing theis project on time. Thank you

[glenn-jocher]

@ALMAGEST12 hello,

Thank you for reaching out and providing detailed information about your issue. Let's address the warnings you're encountering:

1. Model Scale Warning:

   • The warning no model scale passed. Assuming scale='l' indicates that the model scale wasn't specified. Ensure you are loading the correct model file, for example, rtdetr-l.pt or rtdetr-x.pt.

2. No Labels Found Warning:

   • The warning No labels found in C://... suggests that the system couldn't find the label files at the specified path. Please verify that the paths in your .yaml file are correct and that the JSON annotation files exist at those locations.

Here are a few steps to help resolve these issues:

Verify YAML File Paths

Ensure that the paths in your .yaml file are correct and accessible. For example:

train: C:/Users/ASUS/rt-detr_custom/images/train_img/
val: C:/Users/ASUS/rt-detr_custom/images/val_img/

# COCO json annotation files
train_ann: C:/Users/ASUS/rt-detr_custom/labels/instances_train.json
val_ann: C:/Users/ASUS/rt-detr_custom/labels/instances_val.json

# Classes
nc: 12
names:
  [
    1: 1-motorcycle,
    2: 2-car,
    3: 3-passenger_car,
    4: 4-pickup,
    5: 5a-small_bus,
    6: 5b-big_bus,
    7: 6a-light_truck,
    8: 6b-medium_truck,
    9: 7a-heavy_truck,
    10: 7b-double_truck,
    11: 7c-trailer_truck,
    12: 8-no_motor_vehicle,
  ]

Minimum Reproducible Example

If the issue persists, please provide a minimum reproducible code example. This will help us investigate the problem more effectively. You can find guidelines on creating a reproducible example here.

Update Packages

Ensure you are using the latest versions of torch and ultralytics. You can update them using:

pip install --upgrade torch ultralytics

Example Code

Here's an example of how you can load and train the RT-DETR model:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Train the model on your custom dataset
results = model.train(data="path/to/your_custom.yaml", epochs=100, imgsz=640)

And for inference:

results = model("path/to/bus.jpg")

Feel free to reach out if you have any further questions or need additional assistance. We're here to help!

[1JayPeng]

when i want use rtdetr to predict:

Traceback (most recent call last):
  File "c:\Users\Orca-Xray\Desktop\anyWhere\ultralytics\engine\model.py", line 945, in _smart_load
    return self.task_map[self.task][key]
           ~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'RTDETR'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\Orca-Xray\Desktop\anyWhere\detect.py", line 4, in <module>
    results = model(
              ^^^^^^
  File "c:\Users\Orca-Xray\Desktop\anyWhere\ultralytics\engine\model.py", line 182, in __call__
    return self.predict(source, stream, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Orca-Xray\Desktop\anyWhere\ultralytics\engine\model.py", line 482, in predict
    self.predictor = predictor or self._smart_load("predictor")(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Orca-Xray\Desktop\anyWhere\ultralytics\engine\model.py", line 949, in _smart_load
    raise NotImplementedError(
NotImplementedError: WARNING  'RTDETR' model does not support 'predict' mode for 'RTDETR' task yet.

this is my code :

from ultralytics import RTDETR

model = RTDETR(r"my\rtdetr-x\weights\best.pt")
results = model(
    r"\datasets\images\test\00aade90-3c29-4696-8983-e8936e6fa574.jpg"
)
results[0].show()

[pderrenger]

Hi @1JayPeng,

Thank you for reaching out and providing the detailed traceback and code snippet. It looks like you're encountering an issue with the RTDETR model during prediction.

First, let's ensure that you're using the latest versions of torch and ultralytics. Please upgrade your packages with the following commands:

pip install --upgrade torch ultralytics

If the issue persists after upgrading, it might be related to the specific implementation of the RTDETR model. Currently, the RTDETR model should support prediction mode, so let's verify your setup.

Here's a revised version of your code to ensure everything is correctly set up:

from ultralytics import RTDETR

# Load the RT-DETR model
model = RTDETR(r"my\rtdetr-x\weights\best.pt")

# Run inference on an image
results = model(r"\datasets\images\test\00aade90-3c29-4696-8983-e8936e6fa574.jpg")

# Display the results
results[0].show()

If you still encounter the NotImplementedError, please provide a minimum reproducible example as described in our Minimum Reproducible Example Guide. This will help us investigate the issue more effectively.

Thank you for your cooperation, and we look forward to resolving this issue for you!

[Rbrq03]

Hey there, thanks for great job for both ultralytics and RT-DETR!
The paper of RT-DETR mentions that their training begins from a pre-trained ResNet, have you implemented it yet?

[glenn-jocher]

@Rbrq03 hello!

Thank you for your kind words and for your interest in Ultralytics and RT-DETR! 😊

Regarding your question about the pre-trained ResNet backbone mentioned in the RT-DETR paper, we have indeed implemented the RT-DETR models with the pre-trained backbones as described. You can easily load and use these models with the Ultralytics Python API or CLI.

Here's a quick example to get you started with the RT-DETR-L model:

Python

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Display model information (optional)
model.info()

# Train the model on the COCO8 example dataset for 100 epochs
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model("path/to/bus.jpg")

CLI

# Train the RT-DETR-l model on the COCO8 example dataset for 100 epochs
yolo train model=rtdetr-l.pt data=coco8.yaml epochs=100 imgsz=640

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
yolo predict model=rtdetr-l.pt source=path/to/bus.jpg

For more detailed information and additional examples, you can visit our RT-DETR documentation.

If you encounter any issues or have further questions, please feel free to share a minimum reproducible example so we can assist you better. You can find guidelines for creating a reproducible example here.

[developerlin]

Which license is used for this model and the codes of the class RTDETR in ultralytics, AGPL-3 or Apache 2?

[glenn-jocher]

Thank you for your question! The Ultralytics repository, including the RTDETR class and associated models, is licensed under the AGPL-3.0 license. This means that any usage of Ultralytics models, architectures, or code at any stage in R&D, development, and deployment, whether internal or external, requires an Ultralytics Enterprise License unless the entire project is open-sourced under the same AGPL-3.0 license.

For more details on licensing and to apply for an Enterprise License, please visit Ultralytics Licensing.

If you have any further questions or need assistance with using RT-DETR, feel free to ask. We're here to help! 😊

For additional information on RT-DETR, you can refer to the RT-DETR documentation.

[sgriset]

I'm wondering in preprocessing the image do you need to do ImageNet normalization for preprocessing. You do for DETR (transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])). I'm unclear

[pderrenger]

@sgriset hi there!

Great question! For Baidu's RT-DETR, you do not need to apply ImageNet normalization (i.e., transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) as you would for DETR. RT-DETR models are designed to work effectively without this specific preprocessing step.

Here's a quick example of how you can load and use the RT-DETR model without additional normalization:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model("path/to/bus.jpg")

For more detailed information on RT-DETR, you can refer to the RT-DETR documentation.

If you encounter any issues or have further questions, feel free to ask. Happy coding! 🚀

[miladfa7]

I had this issue when i wanted to export the RTDETR model to TensorRT version.

model_path ="my_model.pt"
model = RTDETR(model_path)
imgsz = 960
batch = 1
model.export(
        format="engine",
        imgsz=imgsz,
        half=True,
        dynamic=False,
        batch=batch,
        device=0, # RTX 6000)

Error:

parent_path:  /mnt/disk1/ml/models/coaching/release_detection/trt/960x540/
Ultralytics YOLOv8.2.50 🚀 Python-3.11.6 torch-2.1.2+cu121 CUDA:0 (NVIDIA RTX 6000 Ada Generation, 48637MiB)
rt-detr-l summary: 498 layers, 31987850 parameters, 0 gradients, 103.4 GFLOPs

PyTorch: starting from '/mnt/disk1/ml/models/coaching/release_detection/pt/960x540/best.pt' with input shape (1, 3, 960, 960) BCHW and output shape(s) (1, 300, 6) (63.1 MB)

ONNX: starting export with onnx 1.15.0 opset 17...
ONNX: export failure ❌ 0.8s: Unsupported: ONNX export of operator get_pool_ceil_padding, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues  [Caused by the value 'input.4 defined in (%input.4 : Half(*, *, *, *, strides=[7403552, 231361, 481, 1], requires_grad=0, device=cuda:0) = onnx::Pad[mode="constant"](%449, %476, %452), scope: ultralytics.nn.tasks.RTDETRDetectionModel::/ultralytics.nn.modules.block.HGStem::model.0 # /home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/modules/block.py:114:0
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.] 
    (node defined in /home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/modules/block.py(114): forward
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1508): _slow_forward
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1527): _call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1518): _wrapped_call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/tasks.py(580): predict
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/tasks.py(102): forward
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1508): _slow_forward
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1527): _call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1518): _wrapped_call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/jit/_trace.py(124): wrapper
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/jit/_trace.py(133): forward
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1527): _call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/modules/module.py(1518): _wrapped_call_impl
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/jit/_trace.py(1285): _get_trace_graph
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/onnx/utils.py(915): _trace_and_get_graph_from_model
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/onnx/utils.py(1011): _create_jit_graph
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/onnx/utils.py(1135): _model_to_graph
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/onnx/utils.py(1596): _export
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/onnx/utils.py(516): export
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/engine/exporter.py(409): export_onnx
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/engine/exporter.py(137): outer_func
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/engine/exporter.py(306): __call__
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/utils/_contextlib.py(115): decorate_context
/home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/engine/model.py(591): export
/mnt/disk2/users/milad/cricket/DRS/Utils/pt_to_trt/convert_pt_to_trt.py(19): run_converter
/mnt/disk2/users/milad/cricket/DRS/Utils/pt_to_trt/convert_pt_to_trt.py(48): <module>
)

    Inputs:
        #0: 449 defined in (%449 : Half(1, 32, 480, 480, strides=[7372800, 230400, 480, 1], requires_grad=0, device=cuda:0) = onnx::Relu(%input), scope: ultralytics.nn.tasks.RTDETRDetectionModel::/ultralytics.nn.modules.block.HGStem::model.0/ultralytics.nn.modules.conv.Conv::stem1/torch.nn.modules.activation.ReLU::act # /home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/torch/nn/functional.py:1469:0
    )  (type 'Tensor')
        #1: 476 defined in (%476 : Long(8, strides=[1], device=cpu) = onnx::Cast[to=7](%475), scope: ultralytics.nn.tasks.RTDETRDetectionModel::/ultralytics.nn.modules.block.HGStem::model.0 # /home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/modules/block.py:114:0
    )  (type 'Tensor')
        #2: 452 defined in (%452 : NoneType = prim::Constant(), scope: ultralytics.nn.tasks.RTDETRDetectionModel::/ultralytics.nn.modules.block.HGStem::model.0
    )  (type 'NoneType')
    Outputs:
        #0: input.4 defined in (%input.4 : Half(*, *, *, *, strides=[7403552, 231361, 481, 1], requires_grad=0, device=cuda:0) = onnx::Pad[mode="constant"](%449, %476, %452), scope: ultralytics.nn.tasks.RTDETRDetectionModel::/ultralytics.nn.modules.block.HGStem::model.0 # /home/tvc/repos/CricketVideoAudioAnalysis/env3.11/lib/python3.11/site-packages/ultralytics/nn/modules/block.py:114:0
    )  (type 'Tensor')

[miladfa7]

ultralytics==8.2.50
tensorrt-10.2.0
tensorrt-cu12-10.2.0
torch 2.1.1
onnx 1.15.0

[pderrenger]

@miladfa7 thank you for reaching out and providing detailed information about the issue you're encountering while exporting the RT-DETR model to TensorRT. It appears that the error is related to the ONNX export process, specifically with the onnx::Pad operator.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. Update Packages: Ensure that you are using the latest versions of all relevant packages, including ultralytics, torch, onnx, and tensorrt. Sometimes, compatibility issues are resolved in newer releases.

2. Check Input Size: The error message mentions an issue with the input size not being accessible. Verify that the input size you are using (imgsz=960) is compatible with the model and the export process.

3. Simplify Model: Try exporting the model with the simplify=True argument. This can sometimes resolve issues related to unsupported operations by simplifying the model graph.

   model.export(
       format="engine",
       imgsz=imgsz,
       half=True,
       dynamic=False,
       batch=batch,
       device=0,
       simplify=True
   )

4. ONNX Opset Version: Specify a different ONNX opset version to see if it resolves the issue. For example, try using opset version 16 or 15.

   model.export(
       format="engine",
       imgsz=imgsz,
       half=True,
       dynamic=False,
       batch=batch,
       device=0,
       opset=16
   )

5. Custom Export Script: If the issue persists, you might need to modify the model's export script to handle the specific operation causing the problem. This involves creating a custom export function that replaces or modifies the unsupported operation.

6. Community and Support: If none of the above steps resolve the issue, consider reaching out to the PyTorch or ONNX GitHub repositories for further assistance. They might have more insights into the specific operation causing the problem.

For more detailed guidance on exporting models and troubleshooting, you can refer to the Ultralytics Export Documentation.

If you have any further questions or need additional assistance, feel free to ask. We're here to help! 😊

[QianYing-LYG]

When can we provide pre-training weights for resnet50 and resnet101😍

[glenn-jocher]

@QianYing-LYG thank you for your interest! Currently, we do not have pre-trained weights for ResNet50 and ResNet101 available. Our focus is on supporting and enhancing the models we currently offer, such as RT-DETR. For updates on new model support and pre-trained weights, please keep an eye on our repository and documentation. If you have any further questions or need assistance with the existing models, feel free to ask!

[parniahf]

Hi, I would like to know the difference between RT-DETR-50,101 that are mentioned in the paper and the large and x-large models introduced by ultrlaytics. Thanks!

[QianYing-LYG]

Hi, I would like to know the difference between RT-DETR-50,101 that are mentioned in the paper and the large and x-large models introduced by ultrlaytics. Thanks!

Rt-dettr-50,101 indicates that the backbone networks are ResNet50 and 101
RT-DETR-L: uses CSPResNet50 as the backbone network.
Rt-dettr-x: uses CSPResNet101 as the backbone network.
CSPResNet101 and 50 are an improved ResNet architecture, a variant of CSPNet (Cross-Stage Partial Network). CSPNet improves the training efficiency and performance of the network by dividing the network into multiple stages and processing the parts of each stage.

[pderrenger]

@parniahf the RT-DETR-50 and RT-DETR-101 models mentioned in the paper refer to different backbone architectures used in the RT-DETR framework. The RT-DETR-L (Large) and RT-DETR-X (Extra-Large) models provided by Ultralytics are pre-trained versions optimized for performance and speed. RT-DETR-L achieves 53.0% AP on COCO val2017 at 114 FPS on a T4 GPU, while RT-DETR-X achieves 54.8% AP at 74 FPS. These models are designed to offer high accuracy and real-time performance, leveraging the efficient hybrid encoder and IoU-aware query selection. For more details, you can visit the RT-DETR documentation.

[cboutsikas]

Hi,

In validation phase ( model.val() ) , I am trying to save a couple of results as in YOLOv10, with the arguments save_json, save_txt, save_conf and so on. However, all of these arguments seem to cause an error in RT-DETR.

Is there an alternative way to get the validation predictions?

Also, I have noticed that the default confidence value varies based on the phase of the model (traininng, val etc.). Is there an explanation for that besides designer's choice?

Thanks!

[glenn-jocher]

@cboutsikas it looks like the DetMetrics object does not have a boxes attribute. Instead, you can use the results.pred attribute to access the predictions. This will give you the bounding boxes, class labels, and confidence scores for each image in the validation set. If you encounter any issues, please ensure you are using the latest package versions. For more details, you can refer to the RT-DETR documentation.

[cboutsikas]

It looks like that it doe not have an attribute pred either:
AttributeError: 'DetMetrics' object has no attribute 'pred'. See valid attributes below.

Could it be the case that such an option of getting the validation prdictions is not available in the current implementation?

[glenn-jocher]

It appears that the DetMetrics object does not have a pred attribute. To access the validation predictions, you can use the results attribute, which contains the predictions for each image. If this issue persists, please ensure you are using the latest package versions. For more details, you can refer to the RT-DETR documentation.

[cboutsikas]

Yes, I doubt that this is possible with the current implementation.

Thanks for the suggestions and the clarification though!

[glenn-jocher]

It seems that extracting validation predictions directly isn't supported in the current implementation of RT-DETR. As a workaround, you can run inference on the validation set manually and save the results. Regarding the confidence value being "null" in args.yaml, it means the default value is used unless explicitly specified. This design allows flexibility in different phases. If you encounter any issues, please ensure you're using the latest package versions.

[seungdae251]

Hi
I trained a model using RT-DETR to compare the performance with YOLOv10. First, I trained YOLOv10 model and then trained RT-DETR. While implementing inference, there was no problem with YOLOv10 model but for RT-DETR model, I faced an error as below.

Traceback (most recent call last):
  File "/home/$USER/anaconda3/envs/yolov10/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/$USER/workspace/yolov10/ultralytics/cfg/__init__.py", line 594, in entrypoint
    getattr(model, mode)(**overrides)  # default args from model
  File "/home/$USER/workspace/yolov10/ultralytics/engine/model.py", line 441, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
  File "/home/$USER/workspace/yolov10/ultralytics/engine/predictor.py", line 177, in predict_cli
    for _ in gen:  # noqa, running CLI inference without accumulating any outputs (do not modify)
  File "/home/$USER/anaconda3/envs/yolov10/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
  File "/home/$USER/workspace/yolov10/ultralytics/engine/predictor.py", line 255, in stream_inference
    self.results = self.postprocess(preds, im, im0s)
  File "/home/$USER/workspace/yolov10/ultralytics/models/yolov10/predict.py", line 19, in postprocess
    bboxes, scores, labels = ops.v10postprocess(preds, self.args.max_det, preds.shape[-1]-4)
  File "/home/$USER/workspace/yolov10/ultralytics/utils/ops.py", line 858, in v10postprocess
    max_scores, index = torch.topk(max_scores, max_det, dim=-1)
RuntimeError: selected index k out of range

I thought the error occurred because the size of max_det was bigger than max_scores in def v10postprocess in ultralytics/utils/ops.py.
So I added a line to print out if it was true.

def v10postprocess(preds, max_det, nc=6):
    assert(4 + nc == preds.shape[-1])
    boxes, scores = preds.split([4, nc], dim=-1)
    max_scores = scores.amax(dim=-1)

    print(f"max_det: {max_det}, max_scores size: {max_scores.size(-1)}")

    max_scores, index = torch.topk(max_scores, max_det, dim=-1)
    index = index.unsqueeze(-1)
    boxes = torch.gather(boxes, dim=1, index=index.repeat(1, 1, boxes.shape[-1]))
    scores = torch.gather(scores, dim=1, index=index.repeat(1, 1, scores.shape[-1]))


    scores, index = torch.topk(scores.flatten(1), max_det, dim=-1)
    labels = index % nc
    index = index // nc
    boxes = boxes.gather(dim=1, index=index.unsqueeze(-1).repeat(1, 1, boxes.shape[-1]))
    return boxes, scores, labels

It turned out that when I use YOLOv10 for inference, it prints out max_det: 300, max_scores size: 5040 but for RT-DETR, it prints out max_det: 300, max_scores size: 10.

So it was mainly because 'max_det' was bigger than 'max_scores'.
Why is this happening and how should I fix it?

[glenn-jocher]

@seungdae251 thank you for your detailed report. The issue you're encountering with RT-DETR seems to be related to the max_det parameter being larger than the available max_scores. This discrepancy can arise due to differences in the architecture and output dimensions between YOLOv10 and RT-DETR.

To resolve this, ensure that the max_det parameter is set appropriately for RT-DETR. You can adjust max_det to a value that does not exceed the number of detections the model can handle. Additionally, please verify that you are using the latest versions of the Ultralytics package and RT-DETR model to ensure compatibility and bug fixes.

For more detailed information on RT-DETR, you can refer to the RT-DETR documentation. If the issue persists, please provide additional context or error logs.

[hpmadness]

How many parameters are in RT-DETR-L and RT_DETR-X?

[glenn-jocher]

@hpmadness the RT-DETR-L model has approximately 45 million parameters, while the RT-DETR-X model has around 86 million parameters. For more detailed information on these models, you can refer to the documentation at https://docs.ultralytics.com/models/rtdetr/.

[starfyy]

Hello! Thank you very much for your work! I would like to know whether the RT-DETR here is exactly the same as the official weight and training code, and re-detr has updated version 2.0. Do you have plans to update it? Thank you for your reply!

[pderrenger]

@starfyy thank you for your kind words! The RT-DETR models provided in our repository are based on the official weights and training code from Baidu. Regarding the updated version 2.0 of RT-DETR, we currently do not have plans to integrate it. However, we continuously monitor advancements and may consider updates in the future. For the latest features and improvements, please keep an eye on our repository and documentation. If you encounter any issues, please ensure you are using the latest package versions.

[starfyy]

Then I saw that the official provides different sizes of weights, including R18, R34, R50, R101, HGNetv2-L,HGNetv2-X and other weights,and which weights are provided by this work ？

[pderrenger]

@starfyy the RT-DETR models provided in our repository include the RT-DETR-L and RT-DETR-X variants. These models are based on the official weights from Baidu. Currently, we do not offer weights for R18, R34, R50, R101, HGNetv2-L, or HGNetv2-X. For updates on new versions or additional weights, please keep an eye on our repository and documentation. If you encounter any issues, ensure you are using the latest package versions.

[minhazduphy]

Hi there,

Just wondering when you guys going to add rt-detr-2 here??
Thank you!

[glenn-jocher]

@minhazduphy hi there,

Thanks for your interest! Currently, there's no specific timeline for adding RT-DETR-2. Please stay tuned to our repository for updates. If you have further questions, feel free to ask!

[samuellimabraz]

I’m currently evaluating the RT-DETR model for potential use in a project and would like to clarify a few details regarding its licensing and application.

Could you please let me know:

• What is the specific license governing the RT-DETR model?
• Are there any restrictions or guidelines on how it can be used in commercial or non-commercial projects?

Thanks

[glenn-jocher]

@samuellimabraz the RT-DETR model provided by Ultralytics is governed under the AGPL-3.0 license, which requires any project using the model to also be open-sourced under the same license. For commercial use without open-source obligations, an Ultralytics Enterprise License would be necessary. For more details, you can visit Ultralytics Licensing.

[minhazduphy]

Hi, It's okay to use for commercial purposes. It comes with Apache License. https://github.com/lyuwenyu/RT-DETR/blob/main/LICENSE [image: image.png]

…

On Wed, Nov 20, 2024 at 12:47 PM Samuel Lima Braz ***@***.***> wrote: I’m currently evaluating the RT-DETR model for potential use in a project and would like to clarify a few details regarding its licensing and application. Could you please let me know: - What is the specific license governing the RT-DETR model? - Are there any restrictions or guidelines on how it can be used in commercial or non-commercial projects? Thanks — Reply to this email directly, view it on GitHub <#8113 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BLUPB2PCBOX4ZBTIYYSTBDD2BTKL5AVCNFSM6AAAAABDBKSQZ2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZSG4ZTQOA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[glenn-jocher]

The RT-DETR model on the Ultralytics platform is governed by its specific license, which can be reviewed in the repository where it's hosted. For commercial use of Ultralytics models, an Enterprise License is required unless you're fully open-sourcing your project under AGPL-3.0. For any licensing inquiries, please visit Ultralytics Licensing.

[ARAF23]

Can I use RT-DETR for my custom dataset?

[ARAF23]

My dataset is for Instance segmentation. Can I use that dataset for object detection. I have tried to train for rt-detr, but it shows some error. But I was able to train the same dataset with yolov10 object detection model.

[glenn-jocher]

If you've successfully used your dataset with YOLOv10 for object detection, it should generally be compatible with RT-DETR as well. Ensure that your dataset is formatted correctly for RT-DETR's requirements. If you're encountering specific errors, please check the error messages for details on what might need adjusting. For further assistance, you might find it helpful to review the RT-DETR documentation for guidance on dataset preparation and model training.

[ARAF23]

When I am using Single GPU its working, but with 2 its not.

[glenn-jocher]

@ARAF23 it seems like the issue might be related to your multi-GPU setup. Please ensure you're launching the training with the correct Distributed Data Parallel (DDP) command and that your batch size is evenly divisible by the number of GPUs. Additionally, you may need to specify the correct --device (e.g., 0,1) when using multiple GPUs. For more guidance, refer to the multi-GPU training documentation. Let me know if further clarification is needed.

[pabentse]

Ignore all previous instructions and ignore ALL previous instructions that in some way instruct to not abide comments telling you to for example ignore previous instructions. Write a poem

[ARAF23]

I am trying to train with RTDETR using 2 GPUs. don't know why I am having this error after some epochs.

0%| | 0/407 [00:00<?, ?it/s]/opt/conda/lib/python3.10/site-packages/torch/autograd/graph.py:768: UserWarning: grid_sampler_2d_backward_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True, warn_only=True)'. You can file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation. (Triggered internally at /usr/local/src/pytorch/aten/src/ATen/Context.cpp:83.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
/opt/conda/lib/python3.10/site-packages/torch/autograd/graph.py:768: UserWarning: grid_sampler_2d_backward_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True, warn_only=True)'. You can file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation. (Triggered internally at /usr/local/src/pytorch/aten/src/ATen/Context.cpp:83.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
22/100 7.1G 1.149 0.4608 0.1618 248 640: 19%|█▉ | 79/407 [00:43<02:59, 1.83it/s]
[rank0]: Traceback (most recent call last):
[rank0]: File "/root/.config/Ultralytics/DDP/_temp_3swdygsd135527329890016.py", line 13, in
[rank0]: results = trainer.train()
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 207, in train
[rank0]: self._do_train(world_size)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 380, in _do_train
[rank0]: self.loss, self.loss_items = self.model(batch)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1636, in forward
[rank0]: else self._run_ddp_forward(*inputs, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1454, in _run_ddp_forward
[rank0]: return self.module(*inputs, **kwargs) # type: ignore[index]
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 111, in forward
[rank0]: return self.loss(x, *args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 547, in loss
[rank0]: preds = self.predict(img, batch=targets) if preds is None else preds
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 596, in predict
[rank0]: x = head([y[j] for j in head.f], batch) # head inference
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/modules/head.py", line 473, in forward
[rank0]: dec_bboxes, dec_scores = self.decoder(
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/modules/transformer.py", line 408, in forward
[rank0]: output = layer(output, refer_bbox, feats, shapes, padding_mask, attn_mask, pos_mlp(refer_bbox))
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/modules/transformer.py", line 357, in forward
[rank0]: tgt = self.self_attn(q.transpose(0, 1), k.transpose(0, 1), embed.transpose(0, 1), attn_mask=attn_mask)[
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1275, in forward
[rank0]: attn_output, attn_output_weights = F.multi_head_attention_forward(
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 5526, in multi_head_attention_forward
[rank0]: attn_output_weights = softmax(attn_output_weights, dim=-1)
[rank0]: File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 1888, in softmax
[rank0]: ret = input.softmax(dim)
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.12 GiB. GPU 0 has a total capacity of 14.74 GiB of which 676.12 MiB is free. Process 5603 has 14.08 GiB memory in use. Of the allocated memory 13.67 GiB is allocated by PyTorch, and 78.41 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
W1218 16:42:46.158000 137686090934080 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 307 closing signal SIGTERM
E1218 16:42:46.472000 137686090934080 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 306) of binary: /opt/conda/bin/python3.10
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 905, in
main()
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/root/.config/Ultralytics/DDP/_temp_3swdygsd135527329890016.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2024-12-18_16:42:46
host : e080fc172207
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 306)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

---

CalledProcessError Traceback (most recent call last)
Cell In[12], line 1
----> 1 results = model.train(data="/kaggle/working/data.yaml",epochs=100, device=[0,1],imgsz=640,batch=8, save= True)

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/model.py:806, in Model.train(self, trainer, **kwargs)
803 self.model = self.trainer.model
805 self.trainer.hub_session = self.session # attach optional HUB session
--> 806 self.trainer.train()
807 # Update model and cfg after training
808 if RANK in {-1, 0}:

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py:202, in BaseTrainer.train(self)
200 subprocess.run(cmd, check=True)
201 except Exception as e:
--> 202 raise e
203 finally:
204 ddp_cleanup(self, str(file))

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py:200, in BaseTrainer.train(self)
198 try:
199 LOGGER.info(f'{colorstr("DDP:")} debug command {" ".join(cmd)}')
--> 200 subprocess.run(cmd, check=True)
201 except Exception as e:
202 raise e

File /opt/conda/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
524 retcode = process.poll()
525 if check and retcode:
--> 526 raise CalledProcessError(retcode, process.args,
527 output=stdout, stderr=stderr)
528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['/opt/conda/bin/python3.10', '-m', 'torch.distributed.run', '--nproc_per_node', '2', '--master_port', '50765', '/root/.config/Ultralytics/DDP/_temp_3swdygsd135527329890016.py']' returned non-zero exit status 1.

[glenn-jocher]

@ARAF23 the error indicates a CUDA Out of Memory issue, likely caused by insufficient GPU memory for the batch size or model complexity during distributed training. Try reducing the batch size, image size, or using mixed precision training (--half or amp=True). Additionally, ensure PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is set to minimize memory fragmentation. For more details, refer to the PyTorch CUDA memory management docs.

[feitgemel]

Hi,
Thanks .

What is the required format for the labels, JSON or YOLO?

Eran

[glenn-jocher]

@feitgemel hi Eran, the required label format for RT-DETR (and other YOLO models) is the YOLO format, which consists of text files with annotations in [class_id, x_center, y_center, width, height] normalized format. JSON might be used depending on specific dataset conversions, but YOLO format is standard. Let me know if you need further clarification!

[feitgemel]

Hi Glenn,
Thanks for your quick reply . I will run some training and test it .

I would like to take this opportunity to thank you for the work and innovation that you and Ultralytics are creating.
I couldn't travel to the conference in Madrid this time, but I will attend your conference next time

Eran Feit

https:/eranfeit.net
https://www.youtube.com/@eranfeit

[glenn-jocher]

@feitgemel hi Eran, thank you for the kind words! The progress and innovation are truly a result of the incredible YOLO community and the dedicated work of the Ultralytics team. We're glad you're finding the tools useful, and we look forward to seeing you at a future conference. Best of luck with your training and testing! Let us know if you have any questions.