output.Render problem using openCV

[motoki-tmu]

Hi, I would like to import openCV into detectnet.py to do real-time object detection based on grayscale video. I added code to detectnet.py based on jetson-utils/cuda-to-cv.py and cuda-from-cv.py, but output does not appear on my display. Thank you!

import sys
import argparse
import cv2
import numpy as np

from jetson_inference import detectNet 
from jetson_utils import videoSource, videoOutput, Log, cudaAllocMapped, cudaConvertColor, cudaDeviceSynchronize, cudaToNumpy, cudaFromNumpy

# parse the command line
parser = argparse.ArgumentParser(description="Locate objects in a live camera stream using an object detection DNN.", 
                             formatter_class=argparse.RawTextHelpFormatter, 
                             epilog=detectNet.Usage() + videoSource.Usage() + videoOutput.Usage() + Log.Usage())

parser.add_argument("input", type=str, default="", nargs='?', help="URI of the input stream")
parser.add_argument("output", type=str, default="", nargs='?', help="URI of the output stream")
parser.add_argument("--network", type=str, default="ssd-mobilenet-v2", help="pre-trained model to load (see below for options)")
parser.add_argument("--overlay", type=str, default="box,labels,conf", help="detection overlay flags (e.g. --overlay=box,labels,conf)\nvalid combinations are:  'box', 'labels', 'conf', 'none'")
parser.add_argument("--threshold", type=float, default=0.5, help="minimum detection threshold to use") 

try:
args = parser.parse_known_args()[0]
except:
print("")
parser.print_help()
sys.exit(0)

# create video sources and outputs
input = videoSource(args.input, argv=sys.argv)
output = videoOutput(args.output, argv=sys.argv)

# load the object detection network
net = detectNet(args.network, sys.argv, args.threshold)

# note: to hard-code the paths to load a model, the following API can be used:
#
# net = detectNet(model="model/ssd-mobilenet.onnx", labels="model/labels.txt", 
#                 input_blob="input_0", output_cvg="scores", output_bbox="boxes", 
#                 threshold=args.threshold)


# process frames until EOS or the user exits
while True:

# capture the next image
rgb_img = input.Capture()

if rgb_img is None: # timeout
    continue

# openCV process
bgr_img = cudaAllocMapped(width=rgb_img.width, height=rgb_img.height, format='bgr8')

cudaConvertColor(rgb_img, bgr_img)

cudaDeviceSynchronize()

cv_img = cudaToNumpy(bgr_img)

cv_gray_img = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY)

cv_bgr_img = cv2.cvtColor(cv_gray_img, cv2.COLOR_GRAY2BGR)

cuda_bgr_img = cudaFromNumpy(cv_bgr_img, isBGR=True)

cuda_rgb_img = cudaAllocMapped(width=cuda_bgr_img.width, height=cuda_bgr_img.height, format='rgb8')

cudaConvertColor(cuda_bgr_img, cuda_rgb_img)

# detect objects in the image (with overlay)
detections = net.Detect(cuda_rgb_img, overlay=args.overlay)

# print the detections
print("detected {:d} objects in image".format(len(detections)))

for detection in detections:
    print(detection)

# render the image
output.Render(cuda_rgb_img)

# update the title bar
output.SetStatus("{:s} | Network {:.0f} FPS".format(args.network, net.GetNetworkFPS()))

# print out performance info
net.PrintProfilerTimes()

# exit on input/output EOS
if not input.IsStreaming() or not output.IsStreaming():
    break

[dusty-nv]

Hi @motoki-tmu, do you see it print out any detections that would indicate the image is still valid? Does any window display (if you have display connected) or is the output not working entirely?

[motoki-tmu]

Hi, thank you for your response. The detection program runs successfully, the number of objects and so on are output. However, the image through the camera does not appear on the display. When compared to the original, my code outputs error ''[Open GL] failed to create x11 window. [video] no valid output streams, creating fake null output". I use jetson nano, by the way. Thank you!

[dusty-nv]

OK gotcha - try doing the import cv2 call after this line:

output = videoOutput(args.output, argv=sys.argv)

Its strange, but python-opencv module sometimes will change other libraries OpenGL bindings when it gets loaded.

[motoki-tmu]

Hi, when I changed the line to import cv2, "ImportError: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block" was displayed, and after "export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1", a grayscale detection image appeared on the display! Thank you for the helpful advice!