This project is part of Udacity’s self driving car program and the goal is to find lanes on the road and determine car’s position in the lane. For this project I’ll only analyze data coming from one camera centered on the dashboard.

First I’ll import the necessary libraries for exploring these pictures

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import cv2
%matplotlib inline

img = mpimg.imread('test_images/test2.jpg')
plt.imshow(img)

I’ll be plotting a lot of images so here is a helper function I created to plot multiple images.

def pltMultImg(images,titles=None):
    f, imagesPlts = plt.subplots(1, len(images), figsize=(20,10))
    for indx, im in enumerate(imagesPlts):
        if titles != None:
            assert len(images) == len(titles)
            im.title.set_text(titles[indx])
        im.imshow(images[indx])

img = mpimg.imread('test_images/test1.jpg')
img2 = mpimg.imread('test_images/test2.jpg')
img3 = mpimg.imread('test_images/test3.jpg')
pltMultImg([img,img2,img3],['1','2','3'])

Step 1 Calibrate Camera When Using a New Camera

Whenever working with images it is important to understand that every camera causes different kinds and amounts of distortion. Below are shown how objects look distorted when their picture is captured from different angles.

img = mpimg.imread('camera_cal/calibration1.jpg')
img2 = mpimg.imread('camera_cal/calibration2.jpg')
img3 = mpimg.imread('camera_cal/calibration3.jpg')

pltMultImg([img,img2,img3])

In order to account for distortion we need to take pictures of objects with known dimensions. When we know what the image should look like then we can take many pictures from different angles to later measure the distortion effects and correct for them. Opencv includes some helpful functions such as the cv2.findChessboardCorners, cv2.drawChessboardCorners, and cv2.calibrateCamera to accomplish this task.

# first read in the images, find their corners and object points,
# so that they can be used in the cv2.calibrateCamera function.

import glob
images = glob.glob('camera_cal/calibration*.jpg')

#number of x and y inside corners there are in the chessboards
xNumPts = 9
yNumPts = 6

objpoints = []
imgpoints = []

objp = np.zeros((xNumPts*yNumPts,3),np.float32)
objp[:,:2] = np.mgrid[0:xNumPts,0:yNumPts].T.reshape(-1,2)

for fname in images:
    img = mpimg.imread(fname)
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    ret, corners = cv2.findChessboardCorners(gray, (xNumPts,yNumPts),None)
    if ret:
        imgpoints.append(corners)
        objpoints.append(objp)
        img = cv2.drawChessboardCorners(img, (xNumPts,yNumPts), corners, ret)
        plt.imshow(img)

Now that we’ve collected the necessary pieces we can correct for the distortion effects. I’ve included some examples below.

ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)

img = mpimg.imread('camera_cal/calibration1.jpg')
img2 = mpimg.imread('camera_cal/calibration2.jpg')

dst = cv2.undistort(img,mtx,dist,None,mtx)
dst2 = cv2.undistort(img2,mtx,dist,None,mtx)

pltMultImg([img,dst,img2,dst2],["Distorted 1","Undistorted 1","Distorted 2","Undistorted 2"])

img = mpimg.imread('test_images/test1.jpg')
dst = cv2.undistort(img,mtx,dist,None,mtx)
pltMultImg([img,dst],['Distorted','Corrected'])

Step 2 Feature Extraction and Exploration

Images are rich with color, shapes, and lines which we can use to understand whats in an image. Much of this we take for granted with our eyes and brain, but a computer needs to be told what to look for and how to look for it. Below I’ll walk through some techniques that I’ll be using to detect lines on the road under various road conditions.

# this function uses the absolute value of returned gradients using sobel filters to detect edges
def abs_sobel_thresh(img, orient='x', thresh_min=0, thresh_max=255):
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    if orient == 'x':
        abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 1, 0))
    if orient == 'y':
        abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 0, 1))
    scaled_sobel = np.uint8(255*abs_sobel/np.max(abs_sobel))
    binary_output = np.zeros_like(scaled_sobel)
    binary_output[(scaled_sobel >= thresh_min) & (scaled_sobel <= thresh_max)] = 1
    return binary_output

# this function finds gradients with certain magnitudes
def mag_threshold(img, sobel_kernel=3, thresh=(0,255)):
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
    sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
    magGradient = np.sqrt(sobely**2 + sobelx**2)
    magGradient = (magGradient/ (np.max(magGradient)/255)).astype(np.uint8)
    b_output = np.zeros_like(magGradient)
    b_output[(magGradient >= thresh[0]) & (magGradient <= thresh[1])] = 1
    return b_output

# this function finds gradients with certain directions using arctangent
def dir_threshold(img, sobel_kernel=3, thresh=(0,np.pi/2)):
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
    sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
    absgraddir = np.arctan2(np.absolute(sobely), np.absolute(sobelx))
    b_output = np.zeros_like(absgraddir)
    b_output[(absgraddir >= thresh[0]) & (absgraddir <= thresh[1])] = 1
    return b_output
    
# this function uses color channels to find and filter colors
def color_threshold(img, colorTransform=cv2.COLOR_RGB2HLS, channelNum=2, thresh=(0,255)):
    if colorTransform != None:
        tImg = cv2.cvtColor(img, colorTransform)
    else:
        tImg = img
    cChannel = tImg[:,:,channelNum]
    b_output = np.zeros_like(cChannel)
    b_output[(cChannel >= thresh[0]) & (cChannel <= thresh[1])] = 1
    return b_output
    

I created a helper function that combines the output of the filtered images when techniques like those from above are applied.

def combineThresholds(img,binaryResultsList):
    b_combined = np.zeros_like(img[:,:,0])
    for br in binaryResultsList:
        b_combined[(br == 1) | (b_combined == 1)] = 1
    return b_combined

I will show case these functions later. For now I’m going to show color channels.

Color Channels

Colors are a powerful way to detect lane lines. Yellow and white lines are easily contrasted against the dark pavement and below I’ll show different colorspaces and we’ll see which will be helpful for finding lanes.

Below I’ve show the red, green, and blue channels that we traditionally think of, but it turns out there are many other color spaces used in computer vision. I’ve included several below and I encourage you to read about them.

#color space filtering
img = mpimg.imread('test_images/test2.jpg')
pltMultImg([img[:,:,0],img[:,:,1],img[:,:,2]],['Red Channel','Green Channel','Blue Channel'])

HSV = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
pltMultImg([HSV[:,:,0],HSV[:,:,1],HSV[:,:,2]],['H','S','V'])

HLS = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
pltMultImg([HLS[:,:,0],HLS[:,:,1],HLS[:,:,2]],['H','L','S'])

HLS = cv2.cvtColor(img, cv2.COLOR_RGB2LUV)
pltMultImg([HLS[:,:,0],HLS[:,:,1],HLS[:,:,2]],['L','U','V'])

HLS = cv2.cvtColor(img, cv2.COLOR_RGB2LAB)
pltMultImg([HLS[:,:,0],HLS[:,:,1],HLS[:,:,2]],['L','A','B'])

img = mpimg.imread('test_images/test3.jpg')
dst = cv2.undistort(img,mtx,dist,None,mtx)

pltMultImg([dst, 
            color_threshold(dst,cv2.COLOR_RGB2HSV,channelNum=2,thresh=(220,255)), #V Filter
            color_threshold(dst,cv2.COLOR_RGB2HLS,channelNum=2,thresh=(180,255)), #S Filter
            color_threshold(dst,None,channelNum=0,thresh=(220,255)) #Red Filter
           ],['Original','V Filter','S Filter','Red Filter'])

Edge detection

img = mpimg.imread('test_images/test3.jpg')
dst = cv2.undistort(img,mtx,dist,None,mtx)

pltMultImg([abs_sobel_thresh(dst,'x',12,100),
            abs_sobel_thresh(dst,'y',12,100),
            mag_threshold(dst, sobel_kernel=15, thresh=(50,255)),
            dir_threshold(dst,thresh=(np.pi/2.5,(2*np.pi)/2.5))
           ],['Sobel X','Sobel Y','Magnitude','Direction'])

From the above it is easy to see that the color filters and the sobel x and magnitude filters are the most promising. Later in the more difficult situations we’ll see when each of these fail and where they can complement each other to provide a more robust line detector.

Perspective Transform: Bird’s Eye View

An amazing thing you can do with a little math, is the transformation of an image to make it look like you are looking at an object from another perspective.

Opencv provides the tools for you to select 4 points whether manually or automatically and then transform the perspective so that we can look at the lanes as if we are looking at them from above. With these transformed images we can determine if the lane is curving and by how much.

img = mpimg.imread('test_images/straight_lines2.jpg')
dst = cv2.undistort(img,mtx,dist,None,mtx)

def perspectiveTransformDimensions(img):
    img_size = (img.shape[1],img.shape[0])
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    ybottom_src = img.shape[0]
    ytop_src = ybottom_src - 250
    xleft_top_src = int(img.shape[1]/3)+100
    xright_top_src = int(img.shape[1]-(img.shape[1]/3))-100
    xleft_bot_src = 0
    xright_bot_src = img.shape[1]
    src = np.float32([[xleft_bot_src, ybottom_src], #bottom left
         [xleft_top_src, ytop_src], #top left
         [xright_top_src, ytop_src], #top right
         [xright_bot_src, ybottom_src]]) #bottom right
    sqSize = 300
    ybottom = 200
    ytop = xleft = ybottom + sqSize
    xright = xleft + sqSize
    
    dst = np.float32(
        [[xleft, ybottom], #bottom left
         [xleft, ytop], #top left
         [xright, ytop], #top right
         [xright, ybottom]] #bottom right
    )
    return src, dst, ybottom,ytop,xleft,xright

def my_perspective_transform(img):
    src, dst, ybottom, ytop, xleft, xright = perspectiveTransformDimensions(img)
    M = cv2.getPerspectiveTransform(src, dst)
    warped = cv2.warpPerspective(img, M, (img.shape[1],img.shape[0]), flags=cv2.INTER_LINEAR)
    return cv2.flip(warped[ybottom:ytop,xleft:xright],0)

pltMultImg([img,my_perspective_transform(img),my_perspective_transform(dst)])

When dealing with shadows sometimes it is useful to use a method called contrastive limiting. Below is a function that applies the technique to colored images.

def contrastiveLimitingColorImage(img):
    lab = cv2.cvtColor(img, cv2.COLOR_RGB2LAB)
    lab_planes = cv2.split(lab)
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
    lab_planes[0] = clahe.apply(lab_planes[0])
    lab = cv2.merge(lab_planes)
    result = cv2.cvtColor(lab, cv2.COLOR_LAB2RGB)
    return result
#     pltMultImg([img,result])

image = mpimg.imread('treeShadow.jpg')
orig = image
image = cv2.undistort(image,mtx,dist,None,mtx)
image = contrastiveLimitingColorImage(image)
image = my_perspective_transform(image)
pltMultImg([my_perspective_transform(orig),image],['Original','Contrastive Limiting'])

Below is the code for an interactive widget that I used to tune my filters. I spent a fair amount of time filtering and playing with this. One tip: if you want to “turn off” a filter set the low above the high value.

Blog Note: The widget doesn’t work with the blog page. I’ve included the code for your reference, but I’ve removed the output.

from IPython.html import widgets 
from IPython.html.widgets import interact
from IPython.display import display

image = mpimg.imread('treeShadow.jpg')

image = cv2.undistort(image,mtx,dist,None,mtx)
image = my_perspective_transform(image)
image = contrastiveLimitingColorImage(image)
plt.imshow(image)

def combined_binary_mask_interactive(ksize, mag_low, mag_high, dir_low, dir_high, hls_low, hls_high, bright_low, bright_high, red_low, red_high):
    binary_warped = combineThresholds(image,binaryResultsList=[
        mag_threshold(image, sobel_kernel=ksize, thresh=(mag_low,mag_high)),
#         dir_threshold(image, sobel_kernel=ksize, thresh=(dir_low,dir_high)),
        color_threshold(image,cv2.COLOR_RGB2HSV,channelNum=2,thresh=(bright_low,bright_high)),#v1
#         color_threshold(image,cv2.COLOR_RGB2HLS,channelNum=2,thresh=(hls_low,hls_high)),#v1
        color_threshold(image,cv2.COLOR_RGB2LAB,channelNum=2,thresh=(hls_low,hls_high)),#v1
        color_threshold(image,None,channelNum=0,thresh=(red_low,red_high))
    ])
    plt.figure(figsize=(10,10)) 
    plt.imshow(binary_warped,cmap='gray')

interact(combined_binary_mask_interactive, ksize=(1,31,2), mag_low=(0,255), mag_high=(0,255), dir_low=(0, np.pi/2), dir_high=(0, np.pi/2), hls_low=(0,255), hls_high=(0,255), bright_low=(0,255), bright_high=(0,255), red_low=(0,255), red_high=(0,255))

from IPython.html import widgets 
from IPython.html.widgets import interact
from IPython.display import display

#image = mpimg.imread('test_images/test3.jpg')
image = mpimg.imread('bridge1.jpg')

image = cv2.undistort(image,mtx,dist,None,mtx)
image = my_perspective_transform(image)
image = contrastiveLimitingColorImage(image)
plt.imshow(image)

def combined_binary_mask_interactive(ksize, lab_low, lab_high, luv_low, luv_high, bright_low, bright_high, red_low, red_high):
    binary_warped = combineThresholds(image,binaryResultsList=[
        color_threshold(image,cv2.COLOR_RGB2HSV,channelNum=2,thresh=(bright_low,bright_high)),#v1
        color_threshold(image,cv2.COLOR_RGB2LUV,channelNum=0,thresh=(luv_low,luv_high)),#v1
        color_threshold(image,cv2.COLOR_RGB2LAB,channelNum=2,thresh=(lab_low,lab_high)),#v1
        color_threshold(image,None,channelNum=0,thresh=(red_low,red_high))
    ])
    plt.figure(figsize=(10,10)) 
    plt.imshow(binary_warped,cmap='gray')

interact(combined_binary_mask_interactive, ksize=(1,31,2), lab_low=(0,255), lab_high=(0,255), luv_low=(0,255), luv_high=(0,255), bright_low=(0,255), bright_high=(0,255), red_low=(0,255), red_high=(0,255))

Detect lanes from above and identify lanes

Now we’ll combine the above pieces to detect the lanes and find their curvature. First we’ll correct for distortion, find the lanes, and then plot the histogram of discovered points in the image. Then we’ll run a windowing algorithm that will search segment by segment for where the lane is going from one segment to the next. Then we’ll fit a line to the points and measure its curvature. With the curvature we can determine which direction we are headed and we’ll use that along with points immediately below the camera to determine if we are centered in the lane.

#pipeline for returning lanes from top down view
def warpedBinary(img): #v1
    dst = cv2.undistort(img,mtx,dist,None,mtx)
    warped = my_perspective_transform(dst)
    binary_warped = combineThresholds(warped,binaryResultsList=[
            abs_sobel_thresh(warped,'x',20,100),
            color_threshold(warped,cv2.COLOR_RGB2HSV,channelNum=2,thresh=(220,255)),#v1
            color_threshold(warped,cv2.COLOR_RGB2HLS,channelNum=2,thresh=(180,255)),#v1
            color_threshold(warped,None,channelNum=0,thresh=(220,255))]) #v1
    return binary_warped

def hist(img, half=True):
    if half:
        bottom_half = img[img.shape[0]//2:,:]
    else:
        bottom_half = img
    histogram = np.sum(bottom_half, axis=0)
    return histogram

plt.plot(hist(warpedBinary(img)))

nwindows = 9
margin = 100
minpix = 50

# Load our image
def find_lane_pixels(binary_warped):
    # Take a histogram of the bottom half of the image
    histogram = hist(binary_warped)
    # Create an output image to draw on and visualize the result
    out_img = np.dstack((binary_warped, binary_warped, binary_warped))
    # Find the peak of the left and right halves of the histogram
    # These will be the starting point for the left and right lines
    midpoint = np.int(histogram.shape[0]//2)
    leftx_base = np.argmax(histogram[:midpoint])
    rightx_base = np.argmax(histogram[midpoint:]) + midpoint

    # HYPERPARAMETERS
    # Choose the number of sliding windows
    nwindows = 9
    # Set the width of the windows +/- margin
    margin = 50
    # Set minimum number of pixels found to recenter window
    minpix = 50

    # Set height of windows - based on nwindows above and image shape
    window_height = np.int(binary_warped.shape[0]//nwindows)
    # Identify the x and y positions of all nonzero pixels in the image
    nonzero = binary_warped.nonzero()
    nonzeroy = np.array(nonzero[0])
    nonzerox = np.array(nonzero[1])
    # Current positions to be updated later for each window in nwindows
    leftx_current = leftx_base
    rightx_current = rightx_base

    # Create empty lists to receive left and right lane pixel indices
    left_lane_inds = []
    right_lane_inds = []

    # Step through the windows one by one
    for window in range(nwindows):
        # Identify window boundaries in x and y (and right and left)
        win_y_low = binary_warped.shape[0] - (window+1)*window_height
        win_y_high = binary_warped.shape[0] - window*window_height
        win_xleft_low = leftx_current - margin
        win_xleft_high = leftx_current + margin
        win_xright_low = rightx_current - margin
        win_xright_high = rightx_current + margin
        
        # Draw the windows on the visualization image
        cv2.rectangle(out_img,(win_xleft_low,win_y_low),
        (win_xleft_high,win_y_high),(0,255,0), 2) 
        cv2.rectangle(out_img,(win_xright_low,win_y_low),
        (win_xright_high,win_y_high),(0,255,0), 2) 
        
        # Identify the nonzero pixels in x and y within the window #
        good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & 
        (nonzerox >= win_xleft_low) &  (nonzerox < win_xleft_high)).nonzero()[0]
        good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & 
        (nonzerox >= win_xright_low) &  (nonzerox < win_xright_high)).nonzero()[0]
        
        # Append these indices to the lists
        left_lane_inds.append(good_left_inds)
        right_lane_inds.append(good_right_inds)
        
        # If you found > minpix pixels, recenter next window on their mean position
        if len(good_left_inds) > minpix:
            leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
        if len(good_right_inds) > minpix:        
            rightx_current = np.int(np.mean(nonzerox[good_right_inds]))

    # Concatenate the arrays of indices (previously was a list of lists of pixels)
    try:
        left_lane_inds = np.concatenate(left_lane_inds)
        right_lane_inds = np.concatenate(right_lane_inds)
    except ValueError:
        # Avoids an error if the above is not implemented fully
        pass

    # Extract left and right line pixel positions
    leftx = nonzerox[left_lane_inds]
    lefty = nonzeroy[left_lane_inds] 
    rightx = nonzerox[right_lane_inds]
    righty = nonzeroy[right_lane_inds]

    return leftx, lefty, rightx, righty, out_img

def fit_polynomialDemo(binary_warped):
    # Find our lane pixels first
    leftx, lefty, rightx, righty, out_img = find_lane_pixels(binary_warped)

    # Fit a second order polynomial to each using `np.polyfit`
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)

    # Generate x and y values for plotting
    ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
    try:
        left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
        right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
    except TypeError:
        # Avoids an error if `left` and `right_fit` are still none or incorrect
        print('The function failed to fit a line!')
        left_fitx = 1*ploty**2 + 1*ploty
        right_fitx = 1*ploty**2 + 1*ploty

    ## Visualization ##
    # Colors in the left and right lane regions
    out_img[lefty, leftx] = [255, 0, 0]
    out_img[righty, rightx] = [0, 0, 255]
    return out_img

img= mpimg.imread('test_images/test3.jpg')
out_img = fit_polynomialDemo(warpedBinary(img))
pltMultImg([img,out_img],['Original','Lane Finding'])

def fit_poly(img_shape, leftx, lefty, rightx, righty):
     ### TO-DO: Fit a second order polynomial to each with np.polyfit() ###
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)
    # Generate x and y values for plotting
    ploty = np.linspace(0, img_shape[0]-1, img_shape[0])
    left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
    right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
    return left_fit, right_fit, left_fitx, right_fitx, ploty

def measure_curvature_real(left_fit,right_fit,y_eval,xm_per_pix=3.7/720, ym_per_pix=30/720):
    '''
    Calculates the curvature of polynomial functions in meters.
    '''
    # Define conversions in x and y from pixels space to meters
    left_curverad = ((1 + (2*left_fit[0]*ym_per_pix*y_eval*ym_per_pix + left_fit[1]*xm_per_pix)**2)**1.5) / np.absolute(2*left_fit[0]*ym_per_pix) 
    right_curverad = ((1 + (2*right_fit[0]*ym_per_pix*y_eval*ym_per_pix + right_fit[1]*xm_per_pix)**2)**1.5) / np.absolute(2*right_fit[0]*ym_per_pix)
    return left_curverad, right_curverad

One of the challenges I came across while working on this is that different filters do well under different situations. In order to explore the many situations I created a debuging video that sped things up significantly.

Below is an example of when the gradient methods don’t do as well. When there are dark contrasting lines for example. Many of the color spaces struggle under this dark bridge as well.

half = mpimg.imread('halfRoad.png')
underpass = mpimg.imread('underpass.jpg')
pltMultImg([half, underpass])

pltMultImg([abs_sobel_thresh(underpass,'x',5,100),
            abs_sobel_thresh(underpass,'y',12,100),
            mag_threshold(underpass, sobel_kernel=15, thresh=(50,255))
           ],['Sobel X','Sobel Y','Magnitude'])

In the end I found that the LAB and LUV color spaces do a good job of finding the yellow and white lines under varying lighting conditions.

def warpedBinaryColorOnly(img): #half dark
    dst = cv2.undistort(img,mtx,dist,None,mtx)
    warped = my_perspective_transform(dst)
    img_stack = [
          color_threshold(warped,cv2.COLOR_RGB2LAB,channelNum=2,thresh=(150,255)),#v1,
          color_threshold(warped,cv2.COLOR_RGB2LUV,channelNum=0,thresh=(215,255))#,#v1
    ] # half dark
    binary_warped = combineThresholds(warped,binaryResultsList=img_stack) # half dark
    return binary_warped, img_stack

def inverseTransformDimensions(source_image, destination_image):
    newTarget, dst_, ybottom,ytop,xleft,xright = perspectiveTransformDimensions(destination_image)
    ybottom_src = source_image.shape[0]
    ytop_src = 0
    xleft = 0
    xright = source_image.shape[1]
#     xright = -800
    newSource = np.float32([[xleft, ybottom_src], #bottom left
         [xleft, ytop_src], #top left
         [xright, ytop_src], #top right
         [xright, ybottom_src]]) #bottom right
    return newSource, newTarget

def create_debug_video(img):
    binary_warped, imgStack = warpedBinaryColorOnly(img)
    leftx, lefty, rightx, righty, out_img = find_lane_pixels(binary_warped)
    xm_per_pix = 3.7/700
    ym_per_pix = 30/720
    left_fit, right_fit, left_fitx, right_fitx, ploty = fit_poly(img.shape, leftx, lefty, rightx, righty)
    left_curverad, right_curverad = measure_curvature_real(left_fit,right_fit,img.shape[0],xm_per_pix,ym_per_pix)

    warp_zero = np.zeros_like(binary_warped).astype(np.uint8)
    color_warp = np.dstack((warp_zero, warp_zero, warp_zero))

    # Recast the x and y points into usable format for cv2.fillPoly()
    pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
    pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
    pts = np.hstack((pts_left, pts_right))

    justBelowLeft = np.vstack([left_fitx, ploty])[0][0:20]
    justBelowRight = np.vstack([right_fitx, ploty])[0][0:20]

    # Draw the lane onto the warped blank image
    cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))
    cv2.polylines(color_warp, np.int_([pts_left]), False, (255, 0, 0), thickness=20)
    cv2.polylines(color_warp, np.int_([pts_right]), False, (0, 0, 255), thickness=20)
    # Warp the blank back to original image space using inverse perspective matrix (Minv)

    newSource, newTarget = inverseTransformDimensions(color_warp, img)
    Minv = cv2.getPerspectiveTransform(newSource,newTarget)
    newwarp = cv2.warpPerspective(color_warp, Minv, (img.shape[1], img.shape[0]), flags=cv2.INTER_LINEAR) 
    result = cv2.addWeighted(img, 1, newwarp, 0.9, 0)
    radCurveTxt = "Radius of Curvature = " + str(int((left_curverad + right_curverad)/2)//10)+" (meters)"
    
    centerPosVal = ((justBelowLeft.mean() + justBelowRight.mean())/2)
    width_midpoint = img.shape[1]//2
    centerPosVal = round(((width_midpoint - centerPosVal) * xm_per_pix)/10,2)
    
    centerPosTxt = "Car is " + str(centerPosVal) + " (meters) off center."
    font = cv2.FONT_HERSHEY_SIMPLEX
    cv2.putText(result,radCurveTxt,(100,100), font, 1, (255, 255, 255), 2, cv2.LINE_AA)
    cv2.putText(result,centerPosTxt,(100,140), font, 1, (255, 255, 255), 2, cv2.LINE_AA)
    vis = result
    
    count = 0
    out_img = np.dstack((imgStack[0], imgStack[0], imgStack[0]))
    output = np.zeros_like(out_img)
    for im in imgStack:
        if count == 0:
            color = 1
        elif count == 1:
            color = 2
        else:
            color = 0
        im = np.dstack((im, im, im))
        im[:,:,color] = im[:,:,color] * 255
        if count == 0:
            output = im            
        else:
            output = np.hstack([output,im])
        count += 1
    
    
    output = cv2.resize(output, (vis.shape[1],vis.shape[0]//2), cv2.INTER_AREA)
    outvis = np.zeros((vis.shape[0]+output.shape[0], vis.shape[1],3))
    outvis[:output.shape[0], :output.shape[1],:] = output
    outvis[output.shape[0]:output.shape[0]+vis.shape[0], :vis.shape[1],:] = vis
    return outvis
    

from moviepy.editor import VideoFileClip
from IPython.display import HTML

white_output = 'project_video_onlyDebug.mp4'
clip1 = VideoFileClip("project_video.mp4")
white_clip = clip1.fl_image(create_debug_video) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)
clip1.reader.close()
clip1.audio.reader.close_proc()

[MoviePy] >>>> Building video project_video_onlyDebug.mp4
[MoviePy] Writing video project_video_onlyDebug.mp4


100%|█████████████████████████████████████████████████████████████████████████████▉| 1260/1261 [01:32<00:00, 13.99it/s]


[MoviePy] Done.
[MoviePy] >>>> Video ready: project_video_onlyDebug.mp4 

Wall time: 1min 32s

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format('project_video_onlyDebug.mp4'))

Conclusion

The pipeline above does a good job of following the road when the lines are well marked and the road is darker. I would need to tune the values more and potentially use other feature detectors to improve the pipeline for more challenging sections of road. Also in harder videos I’ve tested my current code focuses too far out and looks only straight ahead for lane lines. When the road curves sharply it doesn’t work well at all. In order to address this I could change the perspective transform to shift as the road turns, but I’m hesitant. I think shortening the length in which it looks ahead could help. I’ll leave that for a later date or for the reader to try.