Custom Object Detection Training

[v-nayjack]

@AlexeyAB I am training yolov3 network for 102 classes. My training has crossed 104k which is more than 50% of the training. Whenever I test the weights, I am not getting any predictions on the test image, the avg loss keeps oscillating between 4.5 and 8.0, is this expected? Is it okay to change the anchor values in the cfg file at this stage of the training?(Used default values until and realized that I am training the network to detect small objects.) Any ideas how to make the testing predictions better?

[stephanecharette]

How small is "small"? There are 3 sizes you need to give to us:

1. the dimensions of your network
2. the size of your images
3. the size of the your objects within those images

No, I don't believe you can change the anchors once training has started. This must be done prior to training. Take a look at DarkMark, it has options to easily recalculate anchors, and various other things to speed up training. For example: https://www.ccoderun.ca/programming/darknet_faq/#time_to_train

Loss should be below 1. If it bounces between 4.5 and 8.0, either your annotations are incorrect or the way you've set things up the objects are below the threshold. You may need to look into something like DarkHelp's tiling.

[v-nayjack]

@stephanecharette Thank you.

1. Here are some of the parameters I am using for training in cfg file, I hope this provides the dimension of the network

Training

batch=64
subdivisions=32
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 204000
policy=steps
steps=163200,183600
scales=.1,.1
2. I thought the size of the input images didn't matter, since all of them will be resized to 416x416, right? Input image sizes are of different sizes (e.g., 6205x4015, 5687x4394, 5933x4193 etc)
3. Some of the smallest object sizes are 21x35, 27x34, 53x57 etc., There could be even smaller objects in my input images.

Okay, I may have to retrain the network from the beginning and use the anchor values generated using darknet.

I went through my training data set again and I didn't find any wrong annotations.

If I have to retrain the network for my data set, what would be the optimal parameters to use in the cfg file?

Thank you again.

[stephanecharette]

Let's take one of your images, say 5933x4193. And the object is 21x35. (Or even smaller you say?)

So your image is resized to 416x416 to match the network dimensions. That is a horizontal factor of 14.26 and vertical factor of 10.08.
So your object of 21x25 now becomes 1x3 pixels in size...

Darknet cannot find a 1-pixel object in your images.

What you'd have to do is enable tiling for example. Darknet doesn't support this natively, but I have some information on the DarkHelp page: https://www.ccoderun.ca/darkhelp/api/Tiling.html

If you need more help in understanding network/image/object sizes, I have some videos on youtube that explains the relationship. For example: https://www.youtube.com/watch?v=Oz-49MpO2rQ

[v-nayjack]

Awesome, I'll take a look at both the links and let you know if I have any questions.

If I have to start training from scratch, I am thinking of using yolov4. Will I still face the same issues with yolov4 if I were to keep all the default values?

Basically I am trying to detect numbers letters and some symbols from the images, that is why the object sizes are so small. Do you recommend using Full-model or Tiny-model of yolo for this purpose?

[stephanecharette]

Doesn't matter if you use full or tiny, you'll still have the exact same issue. Darknet cannot find an object that is 1 pixel in size.

Sounds like a what you are doing is similar to this project: https://www.youtube.com/watch?v=u6SRR9KrHjk

[v-nayjack]

Oh okay, I am going through DarkHelp and image tiling, hopefully I'll have better results this time. I'll keep you posted.

[v-nayjack]

@stephanecharette If I am using image tiling as mentioned in DarkMark and DarkHelp, should I still consider the recommendations provided in " How to improve object detection" section of Readme file at https://github.com/AlexeyAB/darknet ?

Do I need to use image tiling only while training or I must use it for both training and testing?

Have you tried running any of your projects of Google Colab?

[stephanecharette]

should I still consider the recommendations provided in " How to improve object detection" section of Readme file at https://github.com/AlexeyAB/darknet ?

Which recommendation are you referring to? There are many. Most are good recommendations to always follow.

Do I need to use image tiling only while training or I must use it for both training and testing?

Obviously both. The whole idea is to work around you resizing your objects to impossibly small 1-pixel objects.

Have you tried running any of your projects of Google Colab?

I've never run on Google Colab. DarkHelp requires Windows or Linux, while DarkMark requires Linux with X since it is a GUI application.

[v-nayjack]

1. All recommendations provided in that section. Most of them apply to my case. e.g., If I am using image tiling, then the resized objects won't be less than 16x16 (maybe they will be, I don't know yet), would changing the layers = 23 instead of layers = 54 for yolov4 help with the training and testing? If I use image tiling method, would these recommendations still apply to my case?
2. Okay I'll check it out and let you know.
3. I don't have access to a CUDA drives on my local machine that I can use to train the network, that is why I am using Google Colab to train the network on custom data set.

[stephanecharette]

If you using tiling, then there is zero resizing. That is the point to tiling. See the links and videos I posted above where tiling is explained.

[v-nayjack]

Okay, I need to figure out a way to implement it on Google Colab. Even with image tiling, I have restart the entire training process from scratch correct?

[v-nayjack]

@AlexeyAB, @stephanecharette, In the section "How to improve object detection", to make the bounding boxes more accurate, it is suggested to change ignore_thresh = .9 iou_normalizer=0.5 iou_loss=giou to each [yolo] layer for YOLOv4. However, the default value of iou_normalizer=0.07 in yolov4-custom.cfg, should I change it to 0.5 or 0.05? I am wondering whether this is a typo in the documentation.