Advanced Lane Finding using OpenCV

In this 4’th project from the Self-Driving Car engineer program designed by Udacity, our goal is to write a software pipeline to identify the lane boundaries in a video from a front-facing camera on a car. The camera calibration images, test road images, and project videos are available here repository.

The goals / steps of this project are the following:

  • Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
  • Apply a distortion correction to raw images.
  • Use color transforms, gradients, etc., to create a thresholded binary image.
  • Apply a perspective transform to rectify binary image (“birds-eye view”).
  • Detect lane pixels and fit to find the lane boundary.
  • Determine the curvature of the lane and vehicle position with respect to center.
  • Warp the detected lane boundaries back onto the original image.
  • Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

Camera calibration matrix and distortion coefficients

Image distortion occurs when a camera looks at 3D objects in the real world and transforms them into a 2D image; this transformation isn’t perfect. Distortion actually changes what the shape and size of these 3D objects appear to be. So, the first step in analyzing camera images, is to undo this distortion so that you can get correct and useful information out of them.

camera_distorsion

Real cameras use curved lenses to form an image, and light rays often bend a little too much or too little at the edges of these lenses. This creates an effect that distorts the edges of images, so that lines or objects appear more or less curved than they actually are. This is called radial distortion, and it’s the most common type of distortion.

Another type of distortion, is tangential distortion. This occurs when a camera’s lens is not aligned perfectly parallel to the imaging plane, where the camera film or sensor is. This makes an image look tilted so that some objects appear farther away or closer than they actually are.

There are three coefficients needed to correct for radial distortion: k1, k2, and k3, and 2 for tangential distortion: p1, p2. In this project the camera calibration is implemented using OpenCV and a chessboard panel with 9×6 corners.

I start by preparing “object points”, which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp is just a replicated array of coordinates, and objpoints will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. imgpoints will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.

I then used the output objpoints and imgpoints to compute the camera calibration and distortion coefficients using the OpenCV cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:

The code for this step is contained in the file “./camera_calibration.py”. Applying this step to a sample image, you’ll get a result like this:

undistort_output

Use color transforms, gradients, etc., to create a thresholded binary image.

I used a combination of color and gradient thresholds to generate a binary image as implemented in the method compute_binary_image() which can be found in lane_detection.py. There are various combinations of color and gradient thresholds to generate a binary image where the lane lines are clearly visible.

Here’s an example of my output for this step.

test1_binary

Apply a perspective transform to rectify binary image (“birds-eye view”).

Next, we want to identify four source points for the perspective transform. In this case, we can assume the road is a flat plane. This isn’t strictly true, but it can serve as an approximation for this project. We would like to pick four points in a trapezoidal shape (similar to region masking) that would represent a rectangle when looking down on the road from above.

The easiest way to do this is to investigate an image where the lane lines are straight, and find four points lying along the lines that, after perspective transform, make the lines look straight and vertical from a bird’s eye view perspective.

The code for my perspective transform includes 2 functions called compute_perspective_transform() and apply_perspective_transform(), which appear in the file lane_detection.py. The compute_perspective_transform() builds a transformation matrix M used by apply_perspective_transform() to transform a binary image.

I chose the hardcode the source and destination points in the following manner:

The output of these 2 methods looks like this:

test1_warped

Detect lane pixels and fit to find the lane boundary.

We now have a thresholded warped image and we’re ready to map out the lane lines! There are many ways we could go about this, but using peaks in the histogram works well.

After applying calibration, thresholding, and a perspective transform to a road image, we should have a binary image where the lane lines stand out clearly. However, we still need to decide explicitly which pixels are part of the lines and which belong to the left line and which belong to the right line.

I first take a histogram along all the columns in the lower half of the image like this:

test1_histogram

With this histogram I am adding up the pixel values along each column in the image. In my thresholded binary image, pixels are either 0 or 1, so the two most prominent peaks in this histogram will be good indicators of the x-position of the base of the lane lines. I can use that as a starting point for where to search for the lines. From that point, I can use a sliding window, placed around the line centers, to find and follow the lines up to the top of the frame.

The output should look something like this:

test1_color_fit_lines

Determine the curvature of the lane and vehicle position with respect to center.

Self-driving cars need to be told the correct steering angle to turn, left or right. We can calculate this angle if we know a few things about the speed and dynamics of the car and how much the lane is curving. One way to calculate the curvature of a lane line, is to fit a 2nd degree polynomial to that line, and from this we can easily extract useful information.

For a lane line that is close to vertical, we can fit a line using this formula: f(y) = Ay^2 + By + C, where A, B, and C are coefficients. A gives us the curvature of the lane line, B gives us the heading or direction that the line is pointing, and C gives us the position of the line based on how far away it is from the very left of an image (y = 0).

curvature

Implementation:

Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

Function render_curvature_and_offset in lane_detection.py is used to warp the detected lane lines back onto the original image and plot the detected lane using a filled polygon. It also plots the curvature and position in the top left corner and at bottom of the image or video frame.

test1_output

Results for all six test images:

test_all_output

Video pipeline

The same pipeline is applied to videos. The pipeline is implemented in p4.py in the ‘process_video’ method.

Here is the video result.

Problems / issues faced in the implementation of this project

The main issue associated with this project was the allocated time, the estimated 10 hours was far to little. As a result a number of improvements have not been implemented:

  • The class Line.py to encapsulates a single lane line to get a more robust detection by averaging the results computed in each image.
  • The challenge videos. The pipeline has not been applied to the other two challenge videos.

Resources:

https://www.udacity.com/drive

https://medium.com/@ajsmilutin/advanced-lane-finding-5d0be4072514#.cemxisl35

https://github.com/ajsmilutin/CarND-Advanced-Lane-Lines

https://medium.com/towards-data-science/robust-lane-finding-using-advanced-computer-vision-techniques-mid-project-update-540387e95ed3#.s0rzay4b1

https://github.com/GeoffBreemer/SDC-Term1-P4-Advanced-Lane-Finding

https://github.com/Nallo/CarND-P4-Advanced-Lane-Lines

https://github.com/Deborah-Digges/SDC-ND-term-1/tree/master/P4-Advanced-Lane-Lines

Leave a Reply

Your email address will not be published.