How to convert Yolov4x-mish/csp model into onnx &do inferencing?

[akashAD98]

I followed this tutorial https://github.com/linghu8812/tensorrt_inference
I was able to do the conversion, but how can i do inferencing or testing of this converted yolov4x-mish.onnx model?

[stevenlix]

You may use OnnxRuntime for onnx mode inferencing. https://github.com/microsoft/onnxruntime. It provides various execution providers such as CPU, CUDA and TensorRT.

[akashAD98]

@stevenlix i was able to do inference using this script, but how should i draw bbox & visualize results?
this is code

import onnxruntime
import cv2
import numpy as np

model = onnxruntime.InferenceSession(r"K:\company_work\MODEL_WEIGHTS\trt_onnx\yolov4x-mish-custom-electronics-416x416.onnx",None)
print("model Loaded")
img = np.zeros([3,416,416],dtype = np.float32)

#def onnx_inf(model,img):
img = img[np.newaxis,...].astype(np.float32)
input_name = model.get_inputs()[0].name
onnx_output_name = [out.name for out in model.get_outputs()]
result = model.run(onnx_output_name, {input_name: img})
print(results)

reference:
https://github.com/jkjung-avt/tensorrt_demos/blob/master/modnet/test_onnx.py

[akashAD98]

using this scipt i was able to solve inferense , this is yolov4.onxx which is provided by repo
https://github.com/onnx/models/blob/master/vision/object_detection_segmentation/yolov4/model/yolov4.onnx

But when im trying to do same with my custom data im not able to get proper output shapes

i tried this repo for converting yolo models to onnx & tensorrt
& i im getting results on this converted model using tensorrt
https://github.com/jkjung-avt/tensorrt_demos

[stevenlix]

the error message indicates your input shape doesn't match model's input. Please check your input data on 000_net

[akashAD98]

@stevenlix i solved this problem using np.rollaxis , my model is reciving NHWC as input,so i converted it into NCHW ,but again im getting this issue in postprocessing.

i solved this issue by conv_raw_dxdy = pred[:, :, :, :, 0:2] to this commenting one conv_raw_dxdy = pred[:, :, :, 0:2]

again im getting this issue

[jkjung-avt]

@akashAD98 Could you provide the full python script you're working with?

Otherwise, what are the output tensor shapes when you inference your custom model with onnxruntime?

......
result = model.run(onnx_output_name, {input_name: img})
print(results)  # What are the shapes?

[akashAD98]

@jkjung-avt im using this scipt for onnx inference

https://github.com/onnx/models/blob/master/vision/object_detection_segmentation/yolov4/dependencies/inference.ipynb

import onnxruntime
import cv2
import numpy as np

model = onnxruntime.InferenceSession(r"K:\company_work\MODEL_WEIGHTS\trt_onnx\yolov4x-mish-custom-electronics-416x416.onnx",None)
print("model Loaded")
img = np.zeros([3,416,416],dtype = np.float32)

#def onnx_inf(model,img):
img = img[np.newaxis,...].astype(np.float32)
input_name = model.get_inputs()[0].name
onnx_output_name = [out.name for out in model.get_outputs()]
result = model.run(onnx_output_name, {input_name: img})

#print(len(output))
print(result)

--->
[array([[[[5.43156087e-01, 5.38730502e-01, 4.62164581e-01, ...,
5.12988985e-01, 5.09564400e-01, 5.12043834e-01],
[5.16158760e-01, 4.61559117e-01, 4.31563795e-01, ...,
5.43736160e-01, 5.67586184e-01, 5.66980243e-01],
[5.12543499e-01, 4.32360113e-01, 4.53552306e-01, ...,
5.67302525e-01, 5.82414269e-01, 5.57022810e-01],
...,
[5.04360497e-01, 4.82946813e-01, 4.90960002e-01, ...,
5.76641083e-01, 6.24482334e-01, 5.19942403e-01],
[4.92966831e-01, 5.32791436e-01, 4.49620664e-01, ...,
6.05896771e-01, 5.61023772e-01, 5.09344757e-01],
[4.99246687e-01, 5.43988764e-01, 4.45850015e-01, ...,
5.49508035e-01, 5.43327272e-01, 5.04986584e-01]],

    [[5.36974788e-01, 5.24553537e-01, 4.67143923e-01, ...,
      3.71252656e-01, 3.98430407e-01, 4.51538593e-01],
     [5.16066372e-01, 4.74151522e-01, 4.49761689e-01, ...,
      4.63975430e-01, 5.23927808e-01, 5.52703500e-01],
     [4.67342854e-01, 4.12634462e-01, 4.03690666e-01, ...,
      4.13400263e-01, 4.38304871e-01, 4.64697480e-01],
     ...,
     [5.75694323e-01, 6.34198070e-01, 6.52417362e-01, ...,
      6.21224701e-01, 6.01580858e-01, 5.75538635e-01],
     [4.92898732e-01, 5.56907117e-01, 6.32649481e-01, ...,
      5.97918928e-01, 5.69094121e-01, 5.40199101e-01],
     [4.87957597e-01, 4.86043215e-01, 4.91812527e-01, ...,
      4.99950886e-01, 5.12410343e-01, 4.86961126e-01]],

    [[5.23443937e-01, 7.79292107e-01, 9.07118022e-01, ...,
      8.78608942e-01, 7.61271536e-01, 4.38795388e-01],
     [4.55329448e-01, 7.08072305e-01, 8.29850137e-01, ...,
      8.56963336e-01, 7.07964420e-01, 3.79904032e-01],
     [4.47295010e-01, 6.74613118e-01, 7.91363716e-01, ...,
      8.19179177e-01, 6.79931641e-01, 4.00378555e-01],
     ...,
     [4.41179067e-01, 7.08656669e-01, 8.15967798e-01, ...,
      8.01772654e-01, 6.60377979e-01, 4.46031868e-01],
     [4.39138174e-01, 7.59865761e-01, 8.72310162e-01, ...,
      8.63716364e-01, 7.28884459e-01, 4.47639048e-01],
     [4.14665610e-01, 7.16006160e-01, 8.40928793e-01, ...,
      8.36795390e-01, 6.88124239e-01, 4.22244400e-01]],

    ...,

    [[1.85467005e-02, 8.53380561e-03, 3.49757075e-03, ...,
      8.82121921e-03, 1.54880881e-02, 3.84887159e-02],
     [1.15795434e-02, 6.30369782e-03, 4.77072597e-03, ...,
      2.43663788e-03, 6.25890493e-03, 3.95016968e-02],
     [1.07738376e-02, 7.96377659e-03, 8.93741846e-03, ...,
      5.26088476e-03, 7.91543722e-03, 3.22411358e-02],
     ...,
     [6.91694021e-03, 8.35761428e-03, 9.16910172e-03, ...,
      1.30720437e-02, 1.53514445e-02, 2.85124183e-02],
     [7.45928288e-03, 1.36717856e-02, 1.55269504e-02, ...,
      2.31153369e-02, 1.57741308e-02, 2.37288475e-02],
     [8.14956427e-03, 1.13956034e-02, 1.10798478e-02, ...,
      1.75575018e-02, 1.23745799e-02, 1.25564337e-02]],

    [[4.33595479e-02, 2.30600834e-02, 1.73615813e-02, ...,
      1.64348781e-02, 2.26296484e-02, 5.99259436e-02],
     [4.48815823e-02, 1.34840310e-02, 1.30189359e-02, ...,
      5.93450665e-03, 1.35666728e-02, 6.22644126e-02],
     [5.61770797e-02, 2.33932436e-02, 7.28278458e-02, ...,
      1.11199915e-02, 1.72501802e-02, 4.85971570e-02],
     ...,
     [1.01473629e-02, 4.24060225e-03, 4.53561544e-03, ...,
      7.10424781e-03, 1.17792785e-02, 2.84530818e-02],
     [1.72387362e-02, 1.10016763e-02, 1.15490556e-02, ...,
      1.37041807e-02, 1.38654709e-02, 2.51607597e-02],
     [3.68901789e-02, 2.81743705e-02, 2.78824568e-02, ...,
      1.65835321e-02, 1.88879073e-02, 2.82895267e-02]],

    [[1.32935941e-02, 2.34824419e-03, 1.26335025e-03, ...,
      5.09813428e-03, 5.77861071e-03, 2.42372155e-02],
     [4.84403968e-03, 1.01804733e-03, 6.72370195e-04, ...,
      9.83953476e-03, 6.44868612e-03, 1.99303627e-02],
     [8.70105624e-03, 2.13134289e-03, 1.93604827e-03, ...,
      1.75871849e-02, 1.35977864e-02, 2.57759392e-02],
     ...,
     [6.23479486e-03, 1.55609846e-03, 1.16497278e-03, ...,
      8.16076994e-03, 8.66988301e-03, 1.53889656e-02],
     [6.87518716e-03, 1.66732073e-03, 1.13087893e-03, ...,
      7.97024369e-03, 5.68249822e-03, 1.54123902e-02],
     [2.71277130e-02, 1.03957355e-02, 7.73647428e-03, ...,
      1.70073509e-02, 1.32489800e-02, 1.97783113e-02]]]],
  dtype=float32), array([[[[6.82633579e-01, 5.92198133e-01, 5.36219060e-01, ...,
      5.22313654e-01, 5.33153296e-01, 4.08586204e-01],
     [5.63020945e-01, 4.97934908e-01, 4.85931903e-01, ...,
      5.14598966e-01, 5.69518089e-01, 4.36063617e-01],
     [5.41509748e-01, 4.83195364e-01, 5.04000783e-01, ...,
      4.88779098e-01, 5.51217556e-01, 4.52640295e-01],
     ...,
     [5.38222015e-01, 4.33686465e-01, 4.58312184e-01, ...,
      5.19559264e-01, 5.18851757e-01, 5.14279842e-01],
     [5.81782997e-01, 4.60810632e-01, 5.05498111e-01, ...,
      4.84434098e-01, 4.95694876e-01, 4.41713005e-01],
     [6.11413419e-01, 4.79556262e-01, 5.27434230e-01, ...,
      4.62450296e-01, 4.83112365e-01, 4.03794616e-01]],

    [[6.37833953e-01, 6.42928839e-01, 6.27475202e-01, ...,
      6.16223693e-01, 6.30499005e-01, 6.48454309e-01],
     [4.97826725e-01, 4.56331700e-01, 4.40702945e-01, ...,
      4.61066753e-01, 4.56165671e-01, 4.99953568e-01],
     [4.74054486e-01, 4.78003621e-01, 4.57944244e-01, ...,
      5.32258451e-01, 5.19274056e-01, 5.04699171e-01],
     ...,
     [4.30407256e-01, 4.30192322e-01, 4.67941225e-01, ...,
      5.03148675e-01, 4.92921591e-01, 4.81659621e-01],
     [4.17766780e-01, 4.18412983e-01, 4.43399787e-01, ...,
      4.79176104e-01, 4.43045139e-01, 4.04151917e-01],
     [3.07517707e-01, 3.47622275e-01, 3.89939308e-01, ...,
      3.81535292e-01, 3.78883064e-01, 3.79344821e-01]],

    [[4.57153320e-01, 6.56591892e-01, 7.65714765e-01, ...,
      7.75973320e-01, 5.85696876e-01, 4.44520652e-01],
     [3.88055593e-01, 5.95386744e-01, 7.94610620e-01, ...,
      7.05632269e-01, 5.53930521e-01, 3.66048396e-01],
     [3.76973093e-01, 6.13675117e-01, 7.69203782e-01, ...,
      7.30606318e-01, 5.62068701e-01, 3.43667805e-01],
     ...,
     [3.83884132e-01, 5.85814714e-01, 7.69175887e-01, ...,
      7.41853058e-01, 5.92259824e-01, 3.28055650e-01],
     [3.84677619e-01, 5.80844343e-01, 7.98523188e-01, ...,
      7.78843999e-01, 6.17938697e-01, 3.80232394e-01],
     [4.02339399e-01, 5.67262769e-01, 7.13167489e-01, ...,
      7.63966739e-01, 6.26933932e-01, 4.51630592e-01]],

    ...,

    [[1.53054297e-02, 2.84454226e-03, 7.10904598e-04, ...,
      2.52485275e-03, 3.63647938e-03, 1.90643668e-02],
     [3.91662121e-03, 1.00281835e-03, 1.96963549e-04, ...,
      1.93920732e-03, 3.61639261e-03, 2.13303268e-02],
     [5.59258461e-03, 1.60396099e-03, 1.05446577e-03, ...,
      4.41458821e-03, 5.30707836e-03, 1.41301751e-02],
     ...,
     [3.07147801e-02, 2.93711424e-02, 1.66025162e-02, ...,
      5.99618852e-02, 2.56006122e-02, 1.48141384e-02],
     [2.08319128e-02, 1.41707957e-02, 5.74657321e-03, ...,
      1.72817111e-02, 6.36804104e-03, 7.03477859e-03],
     [2.44514346e-02, 4.44053113e-02, 4.16858792e-02, ...,
      2.06050873e-02, 1.64674819e-02, 1.65671110e-02]],

    [[4.03031707e-03, 8.62270594e-04, 1.64061785e-04, ...,
      1.55246258e-03, 2.49183178e-03, 1.24293268e-02],
     [3.61201167e-03, 4.31030989e-04, 5.04553318e-05, ...,
      1.04942918e-03, 2.28783488e-03, 1.18854046e-02],
     [1.50660276e-02, 1.65867805e-03, 3.09884548e-04, ...,
      3.61818075e-03, 1.11929178e-02, 2.72503793e-02],
     ...,
     [4.83229756e-03, 2.75895000e-03, 6.92218542e-04, ...,
      2.49090791e-03, 7.38501549e-03, 1.05685294e-02],
     [2.18084455e-03, 7.71880150e-04, 1.03861094e-04, ...,
      4.57286835e-04, 2.17649341e-03, 6.27920032e-03],
     [7.27033615e-03, 3.02991271e-03, 1.21340156e-03, ...,
      1.92365050e-03, 5.83842397e-03, 1.63541436e-02]],

    [[1.51494443e-02, 2.53537297e-03, 7.02977180e-04, ...,
      7.22974539e-03, 1.03940070e-02, 2.41469443e-02],
     [2.86045671e-03, 4.72068787e-04, 1.08987093e-04, ...,
      2.15992332e-03, 4.55725193e-03, 1.23973191e-02],
     [3.32146883e-03, 7.33464956e-04, 4.30107117e-04, ...,
      1.79910660e-03, 3.71208787e-03, 1.16176009e-02],
     ...,
     [2.11039782e-02, 3.10261548e-02, 7.01978505e-02, ...,
      8.55357349e-02, 3.97977829e-02, 2.49013603e-02],
     [8.98000598e-03, 7.03552365e-03, 8.68326426e-03, ...,
      6.59251213e-03, 5.32424450e-03, 8.27720761e-03],
     [1.41623318e-02, 1.30357146e-02, 1.09080374e-02, ...,
      7.37300515e-03, 7.38766789e-03, 1.39478445e-02]]]],
  dtype=float32), array([[[[0.6213708 , 0.5661254 , 0.4481128 , ..., 0.47417   ,
      0.39457536, 0.31794304],
     [0.60866475, 0.5381434 , 0.4955645 , ..., 0.5112848 ,
      0.4531557 , 0.32479328],
     [0.5456804 , 0.49108282, 0.43284062, ..., 0.46667838,
      0.4916706 , 0.34456927],
     ...,
     [0.5514385 , 0.49755314, 0.46719247, ..., 0.53557044,
      0.48335603, 0.39584845],
     [0.5848158 , 0.5570413 , 0.4873504 , ..., 0.5354574 ,
      0.4694582 , 0.40961075],
     [0.56729645, 0.4964865 , 0.49604815, ..., 0.49212763,
      0.46994185, 0.37628034]],

    [[0.5797265 , 0.55703634, 0.57070756, ..., 0.53550434,
      0.5818699 , 0.6344744 ],
     [0.48733407, 0.43768787, 0.43407708, ..., 0.41947973,
      0.470441  , 0.5378491 ],
     [0.47731167, 0.47195023, 0.5722215 , ..., 0.5269695 ,
      0.50486314, 0.5404542 ],
     ...,
     [0.47561967, 0.5077128 , 0.48406363, ..., 0.43139488,
      0.43169814, 0.4254319 ],
     [0.41377634, 0.4478855 , 0.48052   , ..., 0.503476  ,
      0.49483192, 0.45689604],
     [0.32941583, 0.3828954 , 0.42593485, ..., 0.44739684,
      0.39706135, 0.35836905]],

    [[0.37236294, 0.46066505, 0.54077566, ..., 0.5734195 ,
      0.4744218 , 0.36087704],
     [0.27793106, 0.38961384, 0.49518955, ..., 0.49869967,
      0.4144217 , 0.29593486],
     [0.2490434 , 0.37769425, 0.47748053, ..., 0.4834631 ,
      0.38290572, 0.28281468],
     ...,
     [0.26597092, 0.39243084, 0.49143824, ..., 0.51652086,
      0.43144616, 0.28304714],
     [0.28111193, 0.4223315 , 0.5530058 , ..., 0.55130154,
      0.43678355, 0.28008476],
     [0.33966106, 0.45716453, 0.5357013 , ..., 0.6242875 ,
      0.4995057 , 0.34788686]],

    ...,

    [[0.09541142, 0.0106169 , 0.00353444, ..., 0.01782465,
      0.08237103, 0.26649067],
     [0.06177682, 0.00348103, 0.00145179, ..., 0.02080318,
      0.09804979, 0.1838795 ],
     [0.0975647 , 0.00991598, 0.0057207 , ..., 0.04245368,
      0.11341035, 0.1739361 ],
     ...,
     [0.06827524, 0.03735083, 0.02400085, ..., 0.02770317,
      0.05273968, 0.07968196],
     [0.04161656, 0.00982705, 0.00560099, ..., 0.02726749,
      0.0604848 , 0.10021621],
     [0.06214532, 0.01442632, 0.0111495 , ..., 0.03796089,
      0.09984532, 0.17455974]],

    [[0.08270353, 0.01272744, 0.00861493, ..., 0.01821861,
      0.05675682, 0.16674715],
     [0.095061  , 0.01328394, 0.00812387, ..., 0.02328688,
      0.06117198, 0.17921776],
     [0.12598729, 0.02685288, 0.01897079, ..., 0.04337043,
      0.1049211 , 0.1742737 ],
     ...,
     [0.07125282, 0.01545295, 0.01271072, ..., 0.03880051,
      0.08900312, 0.11251709],
     [0.03476515, 0.00483862, 0.00450209, ..., 0.02490637,
      0.04450244, 0.08635229],
     [0.06149161, 0.01381046, 0.01042584, ..., 0.01981547,
      0.05496967, 0.15815589]],

    [[0.04172212, 0.00397569, 0.00199655, ..., 0.00866416,
      0.02224022, 0.10098869],
     [0.04964879, 0.00369138, 0.00213677, ..., 0.01223791,
      0.02142304, 0.08171579],
     [0.07312593, 0.01034167, 0.00652793, ..., 0.02509719,
      0.04538339, 0.09740078],
     ...,
     [0.06944206, 0.02742767, 0.016478  , ..., 0.05350739,
      0.08730069, 0.08299339],
     [0.0652068 , 0.00961456, 0.00706685, ..., 0.03660542,
      0.04542011, 0.054407  ],
     [0.12979382, 0.02721074, 0.017088  , ..., 0.07418898,
      0.12004897, 0.09969553]]]], dtype=float32)]

[jkjung-avt]

What are the "shapes" of the output tensors? Are they (1, 52, 52, 3, 8), (1, 26, 26, 3, 8) and (1, 13, 13, 3, 8)?

[akashAD98]

yes its Output shape: [(1, 51, 52, 52), (1, 51, 26, 26), (1, 51, 13, 13)]

[jkjung-avt]

It looks like the postprocess_boxes() code in inference.ipynb handles channel_last tensors. But the outputs of "your yolov4-mish-custom-electronics-416x416.onnx" are channel_first.

You need to either rewrite/modify the postprocessing code or shuffle the outputs (converting channel_first to channel last)...

[akashAD98]

so i changed NHWC to NCHW using
image_data=np.rollaxis(image_data, 3,0) at the time of loading image

then i got output shape which i have mentioned here #253 (comment)

now i ran last script & got this issue

now im confused ,where & which line should i need to do conversion of Channnel first to channel last ? in
postprocess_boxes() can you please help me here

[akashAD98]

def postprocess_boxes(pred_bbox, org_img_shape, input_size, score_threshold):
'''remove boundary boxs with a low detection probability'''

valid_scale=[0, np.inf]
pred_bbox = np.array(pred_bbox)
pred_xywh = pred_bbox[:, 0:4]
pred_conf = pred_bbox[:, 4]
pred_prob = pred_bbox[:, 5:]

# # (1) (x, y, w, h) --> (xmin, ymin, xmax, ymax)
pred_coor = np.concatenate([pred_xywh[:, :2] - pred_xywh[:, 2:] * 0.5,
                            pred_xywh[:, :2] + pred_xywh[:, 2:] * 0.5], axis=-1)
# # (2) (xmin, ymin, xmax, ymax) -> (xmin_org, ymin_org, xmax_org, ymax_org)
org_h, org_w = org_img_shape
resize_ratio = min(input_size / org_w, input_size / org_h)

dw = (input_size - resize_ratio * org_w) / 2
dh = (input_size - resize_ratio * org_h) / 2

pred_coor[:, 0::2] = 1.0 * (pred_coor[:, 0::2] - dw) / resize_ratio
pred_coor[:, 1::2] = 1.0 * (pred_coor[:, 1::2] - dh) / resize_ratio

# # (3) clip some boxes that are out of range
pred_coor = np.concatenate([np.maximum(pred_coor[:, :2], [0, 0]),
                            np.minimum(pred_coor[:, 2:], [org_w - 1, org_h - 1])], axis=-1)
invalid_mask = np.logical_or((pred_coor[:, 0] > pred_coor[:, 2]), (pred_coor[:, 1] > pred_coor[:, 3]))
pred_coor[invalid_mask] = 0

# # (4) discard some invalid boxes
bboxes_scale = np.sqrt(np.multiply.reduce(pred_coor[:, 2:4] - pred_coor[:, 0:2], axis=-1))
scale_mask = np.logical_and((valid_scale[0] < bboxes_scale), (bboxes_scale < valid_scale[1]))

# # (5) discard some boxes with low scores
classes = np.argmax(pred_prob, axis=-1)
scores = pred_conf * pred_prob[np.arange(len(pred_coor)), classes]
score_mask = scores > score_threshold
mask = np.logical_and(scale_mask, score_mask)
coors, scores, classes = pred_coor[mask], scores[mask], classes[mask]

return np.concatenate([coors, scores[:, np.newaxis], classes[:, np.newaxis]], axis=-1)

[jkjung-avt]

@akashAD98 This post might be helpful to you: jkjung-avt/tensorrt_demos#510 (comment)