Traffic Light Classifier

---

1. Loading and visualizing the data. The first step in any classification task is to be familiar with our data; we'll need to load in the images of traffic lights and visualize them!

2. Pre-processing. The input images and output labels need to be standardized. This way, we can analyze all the input images using the same classification pipeline, and you know what output to expect when we eventually classify a new image.

3. Feature extraction. Next, we'll extract some features from each image that will help distinguish and eventually classify these images.

4. Classification and visualizing error. Finally, we'll write one function that uses our features to classify any traffic light image. This function will take in an image and output a label. we'll also be given code to determine the accuracy of our classification model.

5. Evaluate our model. Our classifier must be >90% accurate and never classify any red lights as green; it's likely that we'll need to improve the accuracy of our classifier by changing existing features or adding new features.

Some sample images from the dataset (from left to right: red, green, and yellow traffic lights):

---

1. Greater than 90% accuracy
2. Never classify red lights as green

1. Loading and Visualizing the Traffic Light Dataset

This traffic light dataset consists of 1484 number of color images in 3 categories - red, yellow, and green. As with most human-sourced data, the data is not evenly distributed among the types. There are:

• 904 red traffic light images
• 536 green traffic light images
• 44 yellow traffic light images

Note: All images come from this MIT self-driving car course and are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Import resources

Before we get started on the project code, import the libraries and resources that we'll need.

import cv2 # computer vision library
import helpers # helper functions

import random
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg # for loading in images

%matplotlib inline

Training and Testing Data

All 1484 of the traffic light images are separated into training and testing datasets.

• 80% of these images are training images, for us to use as we create a classifier.
• 20% are test images, which will be used to test the accuracy of our classifier.
• All images are pictures of 3-light traffic lights with one light illuminated.

Define the image directories

First, we set some variables to keep track of some where our images are stored:

IMAGE_DIR_TRAINING: the directory where our training image data is stored
IMAGE_DIR_TEST: the directory where our test image data is stored

# Image data directories
IMAGE_DIR_TRAINING = "traffic_light_images/training/"
IMAGE_DIR_TEST = "traffic_light_images/test/"

Load the datasets

These first few lines of code will load the training traffic light images and store all of them in a variable, IMAGE_LIST. This list contains the images and their associated label ("red", "yellow", "green").

For example, the first image-label pair in IMAGE_LIST can be accessed by index: IMAGE_LIST[0][:].

# Using the load_dataset function in helpers.py
# Load training data
IMAGE_LIST = helpers.load_dataset(IMAGE_DIR_TRAINING)

Visualize the Data

The first steps in analyzing any dataset are to 1. load the data and 2. look at the data. Seeing what it looks like will give us an idea of what to look for in the images, what kind of noise or inconsistencies we have to deal with, and so on. This will help us understand the image dataset, and understanding a dataset is part of making predictions about the data.

---

Visualize the input images

Visualize and explore the image data! Write code to display an image in IMAGE_LIST:

• Display the image
• Print out the shape of the image
• Print out its corresponding label

## TODO: Write code to display an image in IMAGE_LIST (try finding a yellow traffic light!)
## TODO: Print out 1. The shape of the image and 2. The image's label

# The first image in IMAGE_LIST is displayed below (without information about shape or label)
selected_image = IMAGE_LIST[752][0]
image = IMAGE_LIST[752][0].shape
label = IMAGE_LIST[752][1]
plt.imshow(selected_image)
print(label, image)

yellow (37, 17, 3)

2. Pre-process the Data

After loading in each image, we have to standardize the input and output!

Input

This means that every input image should be in the same format, of the same size, and so on. Next, we create features by performing the same analysis on every picture, and for a classification task like this, it's important that similar images create similar features!

Output

Also need the output to be a label that is easy to read and easy to compare with other labels. It is good practice to convert categorical data like "red" and "green" to numerical data.

A very common classification output is a 1D list that is the length of the number of classes - three in the case of red, yellow, and green lights - with the values 0 or 1 indicating which class a certain image is. For example, since we have three classes (red, yellow, and green), we can make a list with the order: [red value, yellow value, green value]. In general, order does not matter, we choose the order [red value, yellow value, green value] in this case to reflect the position of each light in descending vertical order.

A red light should have the label: [1, 0, 0]. Yellow should be: [0, 1, 0]. Green should be: [0, 0, 1]. These labels are called one-hot encoded labels.

(Note: one-hot encoding will be especially important when we work with machine learning algorithms).

---

(IMPLEMENTATION): Standardize the input images

• Resize each image to the desired input size: 32x32px.
• (Optional) We may choose to crop, shift, or rotate the images in this step as well.

It's very common to have square input sizes that can be rotated (and remain the same size), and analyzed in smaller, square patches. It's also important to make all images the same size so that they can be sent through the same pipeline of classification steps!

# This function should take in an RGB image and return a new, standardized version
def standardize_input(image):
    
    ## TODO: Resize image and pre-process so that all "standard" images are the same size  
    standard_im = np.copy(image)
    standard_im = cv2.resize(standard_im,(32,32))
    return standard_im
    

Standardize the output

With each loaded image, we also specify the expected output. For this, we use one-hot encoding.

• One-hot encode the labels. To do this, create an array of zeros representing each class of traffic light (red, yellow, green), and set the index of the expected class number to 1.

Since we have three classes (red, yellow, and green), we have imposed an order of: [red value, yellow value, green value]. To one-hot encode, say, a yellow light, we would first initialize an array to [0, 0, 0] and change the middle value (the yellow value) to 1: [0, 1, 0].

---

(IMPLEMENTATION): Implement one-hot encoding

## TODO: One hot encode an image label
## Given a label - "red", "green", or "yellow" - return a one-hot encoded label

# Examples: 
# one_hot_encode("red") should return: [1, 0, 0]
# one_hot_encode("yellow") should return: [0, 1, 0]
# one_hot_encode("green") should return: [0, 0, 1]

def one_hot_encode(label):
    
    ## TODO: Create a one-hot encoded label that works for all classes of traffic lights
    one_hot_encoded = [] 
    if label == 'red':
        one_hot_encoded = [1,0,0]
    elif label == 'yellow':
        one_hot_encoded = [0,1,0]
    elif label == 'green':
        one_hot_encoded = [0,0,1]
    return one_hot_encoded

Testing as we Code

All test code can be found in the file test_functions.py. One test function we'll find is: test_one_hot(self, one_hot_function) which takes in one argument, a one_hot_encode function, and tests its functionality. If our one_hot_label code does not work as expected, this test will print ot an error message that will tell you a bit about why your code failed. Once your code works, this should print out TEST PASSED.

# Importing the tests
import test_functions
tests = test_functions.Tests()

# Test for one_hot_encode function
tests.test_one_hot(one_hot_encode)

TEST PASSED

Construct a STANDARDIZED_LIST of input images and output labels.

This function takes in a list of image-label pairs and outputs a standardized list of resized images and one-hot encoded labels.

This uses the functions we defined above to standardize the input and output, so those functions must be complete for this standardization to work!

def standardize(image_list):
    
    # Empty image data array
    standard_list = []

    # Iterate through all the image-label pairs
    for item in image_list:
        image = item[0]
        label = item[1]

        # Standardize the image
        standardized_im = standardize_input(image)

        # One-hot encode the label
        one_hot_label = one_hot_encode(label)    

        # Append the image, and it's one hot encoded label to the full, processed list of image data 
        standard_list.append((standardized_im, one_hot_label))
        
    return standard_list

# Standardize all training images
STANDARDIZED_LIST = standardize(IMAGE_LIST)

Visualize the standardized data

Display a standardized image from STANDARDIZED_LIST and compare it with a non-standardized image from IMAGE_LIST. Note that their sizes and appearance are different!

## TODO: Display a standardized image and its label
f, (pl1, pl2) = plt.subplots(1,2)
pl1.imshow(STANDARDIZED_LIST[5][0])
pl2.imshow(IMAGE_LIST[5][0])

<matplotlib.image.AxesImage at 0x7f8678a766a0>

3. Feature Extraction

we'll be using what we now about color spaces, shape analysis, and feature construction to create features that help distinguish and classify the three types of traffic light images.

1. A brightness feature.
   • Using HSV color space, create a feature that helps us identify the 3 different classes of traffic light.
   • We'll be asked some questions about what methods we tried to locate this traffic light, so, as we progress through this notebook, always be thinking about our approach: what works and what doesn't?

Creating a brightness feature

Pictured below is a sample pipeline for creating a brightness feature (from left to right: standardized image, HSV color-masked image, cropped image, brightness feature):

RGB to HSV conversion

Below, a test image is converted from RGB to HSV colorspace and each component is displayed in an image.

# Convert and image to HSV colorspace
# Visualize the individual color channels

image_num = 0
test_im = STANDARDIZED_LIST[image_num][0]
test_label = STANDARDIZED_LIST[image_num][1]

# Convert to HSV
hsv = cv2.cvtColor(test_im, cv2.COLOR_RGB2HSV)

# Print image label
print('Label [red, yellow, green]: ' + str(test_label))

# HSV channels
h = hsv[:,:,0]
s = hsv[:,:,1]
v = hsv[:,:,2]

# Plot the original image and the three channels
f, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4, figsize=(20,10))
ax1.set_title('Standardized image')
ax1.imshow(test_im)
ax2.set_title('H channel')
ax2.imshow(h, cmap='gray')
ax3.set_title('S channel')
ax3.imshow(s, cmap='gray')
ax4.set_title('V channel')
ax4.imshow(v, cmap='gray')

Label [red, yellow, green]: [1, 0, 0]





<matplotlib.image.AxesImage at 0x7f8678944ac8>

---

(IMPLEMENTATION): Create a brightness feature that uses HSV color space

A function that takes in an RGB image and returns a 1D feature vector and/or single value that will help classify an image of a traffic light. The only requirement is that this function should apply an HSV colorspace transformation, the rest is up to us.

From this feature, we should be able to estimate an image's label and classify it as either a red, green, or yellow traffic light. We may also define helper functions if they simplify our code.

## TODO: Create a brightness feature that takes in an RGB image and outputs a feature vector and/or value
## This feature should use HSV colorspace values
def create_feature(rgb_image):
    
    ## TODO: Convert image to HSV color space
    hsv = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2HSV)
    
    ## TODO: Create and return a feature value and/or vector
    feature = []
    feature = mostColor(hsv)
    
    return feature

(Optional) Create more features to help accurately label the traffic light images

# (Optional) Add more image analysis and create more features

def mostColor(hsv):
    
    color = []   
    
    lR = np.array([0, 0, 180])
    uR = np.array([255, 255, 255])
    mask = cv2.inRange(hsv, lR, uR)
    masked = np.copy(hsv)
    masked[mask == 0] = [0, 0, 0]
    y_axis = 6
    x_axis = 11
    cropped = masked[y_axis:32-y_axis,x_axis:32-x_axis]
    red = cropped[:5,:,2]
    yellow = cropped[7:13,:,2]
    green = cropped[15:23,:,2]
    color.append(np.sum(red))
    color.append(np.sum(yellow))
    
    color.append(np.sum(green))
    
    return color

4. Classification and Visualizing Error

Using all of our features, we write a function that takes in an RGB image and, using our extracted features, outputs whether a light is red, green or yellow as a one-hot encoded label. This classification function should be able to classify any image of a traffic light!

---

(IMPLEMENTATION): Build a complete classifier

# This function should take in RGB image input
# Analyze that image using our feature creation code and output a one-hot encoded label
def estimate_label(rgb_image):
    
    ## TODO: Extract feature(s) from the RGB image and use those features to
    ## classify the image and output a one-hot encoded label
    predicted_label = []
    
    Vect = create_feature(rgb_image)
    dominantColor = max(Vect)
  
    if Vect[0] == dominantColor:
        predicted_label = one_hot_encode('red')
    elif Vect[1] == dominantColor:
        predicted_label = one_hot_encode('yellow')
    elif Vect[2] == dominantColor:
        predicted_label = one_hot_encode('green')
    else:
        print("err")
        
        
    return predicted_label   
    

Testing the classifier

Here is where we test our classification algorithm using our test set of data that we set aside at the beginning of the notebook! This project will be complete once we've pogrammed a "good" classifier.

A "good" classifier in this case should meet the following criteria

1. Get above 90% classification accuracy.
2. Never classify a red light as a green light.

Test dataset

Below, we load in the test dataset, standardize it using the standardize function we defined above, and then shuffle it; this ensures that order will not play a role in testing accuracy.

# Using the load_dataset function in helpers.py
# Load test data
TEST_IMAGE_LIST = helpers.load_dataset(IMAGE_DIR_TEST)

# Standardize the test data
STANDARDIZED_TEST_LIST = standardize(TEST_IMAGE_LIST)

# Shuffle the standardized test data
random.shuffle(STANDARDIZED_TEST_LIST)

Determine the Accuracy

Compare the output of our classification algorithm (a.k.a. our "model") with the true labels and determine the accuracy.

This code stores all the misclassified images, their predicted labels, and their true labels, in a list called MISCLASSIFIED. This code is used for testing and should not be changed.

# Constructs a list of misclassified images given a list of test images and their labels
# This will throw an AssertionError if labels are not standardized (one-hot encoded)

def get_misclassified_images(test_images):
    # Track misclassified images by placing them into a list
    misclassified_images_labels = []

    # Iterate through all the test images
    # Classify each image and compare to the true label
    for image in test_images:

        # Get true data
        im = image[0]
        true_label = image[1]
        assert(len(true_label) == 3), "The true_label is not the expected length (3)."

        # Get predicted label from our classifier
        predicted_label = estimate_label(im)
        assert(len(predicted_label) == 3), "The predicted_label is not the expected length (3)."

        # Compare true and predicted labels 
        if(predicted_label != true_label):
            # If these labels are not equal, the image has been misclassified
            misclassified_images_labels.append((im, predicted_label, true_label))
            
    # Return the list of misclassified [image, predicted_label, true_label] values
    return misclassified_images_labels


# Find all misclassified images in a given test set
MISCLASSIFIED = get_misclassified_images(STANDARDIZED_TEST_LIST)

# Accuracy calculations
total = len(STANDARDIZED_TEST_LIST)
num_correct = total - len(MISCLASSIFIED)
accuracy = num_correct/total

print('Accuracy: ' + str(accuracy))
print("Number of misclassified images = " + str(len(MISCLASSIFIED)) +' out of '+ str(total))

Accuracy: 0.936026936026936
Number of misclassified images = 19 out of 297

---

Visualize the misclassified images

Visualize some of the images we classified wrong (in the MISCLASSIFIED list) and note any qualities that make them difficult to classify. This will help us identify any weaknesses in our classification algorithm.

# Visualize misclassified example(s)
## TODO: Display an image in the `MISCLASSIFIED` list 
## TODO: Print out its predicted label - to see what the image *was* incorrectly classified as
f, (pl1) = plt.subplots(1)
pl1.imshow(MISCLASSIFIED[1][0])
print(str(MISCLASSIFIED[1][1]))

[0, 1, 0]

---

Test if we classify any red lights as green

The code below lets us test to see if we've misclassified any red lights as green in the test set. This test assumes that MISCLASSIFIED is a list of tuples with the order: [misclassified_image, predicted_label, true_label].

Note: this is not an all encompassing test, but its a good indicator that, if we pass, we are on the right track! This iterates through our list of misclassified examples and checks to see if any red traffic lights have been mistakenly labelled [0, 1, 0] (green).

# Importing the tests
import test_functions
tests = test_functions.Tests()

if(len(MISCLASSIFIED) > 0):
    # Test code for one_hot_encode function
    tests.test_red_as_green(MISCLASSIFIED)
else:
    print("MISCLASSIFIED may not have been populated with images.")

TEST PASSED

---