In this project, I used Python and OpenCV to detect lane lines on the road. I developed a processing pipeline that works on a series of individual images, and applied the result to a video stream.
- Load test images.
- Apply Color Selection
- Apply Canny edge detection.
- Apply gray scaling to the images.
- Apply Gaussian smoothing.
- Perform Canny edge detection.
- Determine the region of interest.
- Apply Hough transform.
- Average and extrapolating the lane lines.
- Apply on video streams.
We have 6 test images. I wrote a function called list_images() that shows all the test images we're working on using matplotlib.
def list_images(images, cols = 2, rows = 5, cmap=None):
"""
Display a list of images in a single figure with matplotlib.
Parameters:
images: List of np.arrays compatible with plt.imshow.
cols (Default = 2): Number of columns in the figure.
rows (Default = 5): Number of rows in the figure.
cmap (Default = None): Used to display gray images.
"""
plt.figure(figsize=(10, 11))
for i, image in enumerate(images):
plt.subplot(rows, cols, i+1)
#Use gray scale color map if there is only one channel
cmap = 'gray' if len(image.shape) == 2 else cmap
plt.imshow(image, cmap = cmap)
plt.xticks([])
plt.yticks([])
plt.tight_layout(pad=0, h_pad=0, w_pad=0)
plt.show()Lane lines in the test images are in white and yellow. We need to choose the most suitable color space, that clearly highlights the lane lines. I applied color selection to the original RGB images, HSV images, and HSL images, and found out that using HSL will be the best color space to use.
def HSL_color_selection(image):
"""
Apply color selection to the HSL images to blackout everything except for white and yellow lane lines.
Parameters:
image: An np.array compatible with plt.imshow.
"""
#Convert the input image to HSL
converted_image = convert_hsl(image)
#White color mask
lower_threshold = np.uint8([0, 200, 0])
upper_threshold = np.uint8([255, 255, 255])
white_mask = cv2.inRange(converted_image, lower_threshold, upper_threshold)
#Yellow color mask
lower_threshold = np.uint8([10, 0, 100])
upper_threshold = np.uint8([40, 255, 255])
yellow_mask = cv2.inRange(converted_image, lower_threshold, upper_threshold)
#Combine white and yellow masks
mask = cv2.bitwise_or(white_mask, yellow_mask)
masked_image = cv2.bitwise_and(image, image, mask = mask)
return masked_imageWe need to detect edges in the images to be able to correctly detect lane lines. The Canny edge detector is an edge detection operator that uses a multi-stage algorithm to detect a wide range of edges in images. The Canny algorithm involves the following steps:
- Gray scaling the images: The Canny edge detection algorithm measures the intensity gradients of each pixel. So, we need to convert the images into gray scale in order to detect edges.
- Gaussian smoothing: Since all edge detection results are easily affected by image noise, it is essential to filter out the noise to prevent false detection caused by noise. To smooth the image, a Gaussian filter is applied to convolve with the image. This step will slightly smooth the image to reduce the effects of obvious noise on the edge detector.
- Find the intensity gradients of the image.
- Apply non-maximum suppression to get rid of spurious response to edge detection.
- Apply double threshold to determine potential edges.
- Track edge by hysteresis: Finalize the detection of edges by suppressing all the other edges that are weak and not connected to strong edges. If an edge pixel’s gradient value is higher than the high threshold value, it is marked as a strong edge pixel. If an edge pixel’s gradient value is smaller than the high threshold value and larger than the low threshold value, it is marked as a weak edge pixel. If an edge pixel's value is smaller than the low threshold value, it will be suppressed. The two threshold values are empirically determined and their definition will depend on the content of a given input image.
We're interested in the area facing the camera, where the lane lines are found. So, we'll apply region masking to cut out everything else.
def region_selection(image):
"""
Determine and cut the region of interest in the input image.
Parameters:
image: An np.array compatible with plt.imshow.
"""
mask = np.zeros_like(image)
#Defining a 3 channel or 1 channel color to fill the mask with depending on the input image
if len(image.shape) > 2:
channel_count = image.shape[2]
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
#We could have used fixed numbers as the vertices of the polygon,
#but they will not be applicable to images with different dimesnions.
rows, cols = image.shape[:2]
bottom_left = [cols * 0.1, rows * 0.95]
top_left = [cols * 0.4, rows * 0.6]
bottom_right = [cols * 0.9, rows * 0.95]
top_right = [cols * 0.6, rows * 0.6]
vertices = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32)
cv2.fillPoly(mask, vertices, ignore_mask_color)
masked_image = cv2.bitwise_and(image, mask)
return masked_imageThe Hough transform is a technique which can be used to isolate features of a particular shape within an image. I'll use it to detected the lane lines in selected_region_images.
def hough_transform(image):
"""
Determine and cut the region of interest in the input image.
Parameters:
image: The output of a Canny transform.
"""
rho = 1 #Distance resolution of the accumulator in pixels.
theta = np.pi/180 #Angle resolution of the accumulator in radians.
threshold = 20 #Only lines that are greater than threshold will be returned.
minLineLength = 20 #Line segments shorter than that are rejected.
maxLineGap = 300 #Maximum allowed gap between points on the same line to link them
return cv2.HoughLinesP(image, rho = rho, theta = theta, threshold = threshold,
minLineLength = minLineLength, maxLineGap = maxLineGap)We have multiple lines detected for each lane line. We need to average all these lines and draw a single line for each lane line. We also need to extrapolate the lane lines to cover the full lane line length.
The left lane should have a negative slope (given that y coordinate is reversed in the images), and the right lane should have a negative slope. Therefore, we'll collect positive slope lines and negative slope lines separately and take averages.
After calculating the average slope and intercept of each lane line, we'll convert these values to pixel points, and then we'll be able to draw the full length lane line.
def average_slope_intercept(lines):
"""
Find the slope and intercept of the left and right lanes of each image.
Parameters:
lines: The output lines from Hough Transform.
"""
left_lines = [] #(slope, intercept)
left_weights = [] #(length,)
right_lines = [] #(slope, intercept)
right_weights = [] #(length,)
for line in lines:
for x1, y1, x2, y2 in line:
if x1 == x2:
continue
slope = (y2 - y1) / (x2 - x1)
intercept = y1 - (slope * x1)
length = np.sqrt(((y2 - y1) ** 2) + ((x2 - x1) ** 2))
if slope < 0:
left_lines.append((slope, intercept))
left_weights.append((length))
else:
right_lines.append((slope, intercept))
right_weights.append((length))
left_lane = np.dot(left_weights, left_lines) / np.sum(left_weights) if len(left_weights) > 0 else None
right_lane = np.dot(right_weights, right_lines) / np.sum(right_weights) if len(right_weights) > 0 else None
return left_lane, right_lane
def pixel_points(y1, y2, line):
"""
Converts the slope and intercept of each line into pixel points.
Parameters:
y1: y-value of the line's starting point.
y2: y-value of the line's end point.
line: The slope and intercept of the line.
"""
if line is None:
return None
slope, intercept = line
x1 = int((y1 - intercept)/slope)
x2 = int((y2 - intercept)/slope)
y1 = int(y1)
y2 = int(y2)
return ((x1, y1), (x2, y2))
def lane_lines(image, lines):
"""
Create full lenght lines from pixel points.
Parameters:
image: The input test image.
lines: The output lines from Hough Transform.
"""
left_lane, right_lane = average_slope_intercept(lines)
y1 = image.shape[0]
y2 = y1 * 0.6
left_line = pixel_points(y1, y2, left_lane)
right_line = pixel_points(y1, y2, right_lane)
return left_line, right_line
def draw_lane_lines(image, lines, color=[255, 0, 0], thickness=12):
"""
Draw lines onto the input image.
Parameters:
image: The input test image.
lines: The output lines from Hough Transform.
color (Default = red): Line color.
thickness (Default = 12): Line thickness.
"""
line_image = np.zeros_like(image)
for line in lines:
if line is not None:
cv2.line(line_image, *line, color, thickness)
return cv2.addWeighted(image, 1.0, line_image, 1.0, 0.0)Now, we'll use the above functions to detect lane lines from a video stream. The video inputs are in test_videos folder. The video outputs are generated in output_videos folder.
def frame_processor(image):
"""
Process the input frame to detect lane lines.
Parameters:
image: Single video frame.
"""
color_select = HSL_color_selection(image)
gray = gray_scale(color_select)
smooth = gaussian_smoothing(gray)
edges = canny_detector(smooth)
region = region_selection(edges)
hough = hough_transform(region)
result = draw_lane_lines(image, lane_lines(image, hough))
return result
def process_video(test_video, output_video):
"""
Read input video stream and produce a video file with detected lane lines.
Parameters:
test_video: Input video.
output_video: A video file with detected lane lines.
"""
input_video = VideoFileClip(os.path.join('test_videos', test_video), audio=False)
processed = input_video.fl_image(frame_processor)
processed.write_videofile(os.path.join('output_videos', output_video), audio=False)The project succeeded in detecting the lane lines clearly in the video streams. This project is intended to only detect (mostly) straight lines. A possible improvement would be to modify the above pipeline to detect curved lane line. Another potential improvement could be to identify the lane boundaries and detect the full lane, not just the lane lines.