CNN Part 2: Downloading and Preprocessing the car dataset.

Welcome to part 2 of this series on CNN. In the previous lesson we learned about the working of CNN, Now in this lesson and upcoming lessons, we will build a fully functional model to determine the probability of a cat or a dog in an image

For this first download the data set from the official microsoft site by clicking here

You will get two data set, one contains cat image and another contains dog images, download both, this dataset contains 12501 images of both cats and dogs.

Preprocessing the Data

We have the dataset downloaded, now, we start preprocessing them but before we move forward lets learn why this preprocessing is required,

First of all Colour in not a deciding factor here, Red Audi, Black Audi, Audi is Audi, 😀 there are datasets where we should consider colours like determining RBC cells or like Rubix cube pattern but here we don’t need colour. If we still let the colour in RGB pattern then it required 3x computational cost which is just a waste.

Second is we don’t need these huge images we can resize it till the point image is recognizable, also we have to normalize all the image in a same shape and size otherwise some images will add more weight and it will lead to the inaccuracy in the model. 

Here lets see the code to determine the size of an image

import cv2
import numpy as np
import os
import matplotlib.pyplot as plt

Dog_folder_path="/Users/pushkarsingh/Desktop/catVSdog/Datasets/categories/dog"
img_size=80
for img in os.listdir(Dog_folder_path):
    img= cv2.imread(os.path.join(Dog_folder_path,img))
    img_gray= cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_gray_resize= cv2.resize(img_gray,(img_size,img_size))
    plt.imshow(img_gray_resize, cmap="gray")
    plt.show()
    break

In the 16th line we are iterating to all the image in the folder and saving it into a variable im_array next we convert the image into a grayscale image, to convert from 3 channel to 1 channel, then we resize the image using cv2.resize() function of openCV. here we can toggle various image sizes from line 5 to judge the image. Let’s look at it.

Here in 20×20 image, it’s unclear what is in the image, in 40X40 it’s something looking like a dog because of legs or say body structure. but still unclear and some details can be missed from here, from 60X60 it can be easily understood about the content of an image, but for the safe side, we are using 80X80 because here everything is looking clearly.

Once the image size is detrmined we can proceed to preproccess the data or say compress the data to reduce the computational cost.

Lets look at the code to preprocess the data

import cv2
import numpy as np
import os
import pickle

path="/Users/pushkarsingh/Desktop/catVSdog/Datasets/categories"
img_size=80
training_dataset=[]
categories=["Dog","Cat"]

for category in categories:
    def preprocessing_data(folder_path):
        image_path=os.path.join(folder_path,category)
        try:
            for Image in os.listdir(folder_path):
                img= cv2.imread(os.path.join(folder_path,Image))
                img_gray= cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                img_gray_resize= cv2.resize(img_gray,(img_size,img_size))
                training_dataset.append([im_gray_array_resize,categories.index(category)])
        except:
            pass

preprocessing_data(path)

X=[]
y=[]
training_dataset=np.random.shuffle(training_dataset)
for features, label in training_dataset:
    X.append(features)
    y.append(label)
    
X = np.array(X).reshape(-1, img_size, img_size, 1) 


pickle_X=open("/Users/pushkarsingh/Desktop/catVSdog/Pickle/X.pickle","wb")
pickle.dump(X,pickle_X)
pickle_X.close()

pickle_y=open("/Users/pushkarsingh/Desktop/catVSdog/Pickle/y.pickle","wb")
pickle.dump(y,pickle_y)
pickle_y.close()

First of all we are importing the necessary packages and defining the path to the folder where we stored our images then we define the size of the image which determined above.we are also initializing a list “training_dataset” in which we will store the processed data.

import cv2
import numpy as np
import os
import pickle

path="/Users/pushkarsingh/Desktop/catVSdog/Datasets/categories"
img_size=80
training_dataset=[]

This is the main loop. First we are defining our categories, we will use it to serve two purpose. First is to defining the path to the destination folder and second is to define the labels.

Here in the 19th line. we are assigning a label to an individual image, it’s like giving a name to every image but we learn earlier that Neural Network learns in the form of numbers.

So we give them the name in the form of numbers, we use list.index() method which will pass the index value of current category, here 0 for a Cat, 1 for a Dog. Now when we fed the data into Neural network it analyzes the image and attaches the conclusion with a particular label, in this way a conclusion and label go hand in hand.

Note: We are using try and except block otherwise it will throw an path error like this.

Traceback (most recent call last)
<timed exec> in <module>
<timed exec> in preprocessing_data(folder_path)
error: OpenCV(4.0.0) /Users/travis/build/skvark/opencv-python/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
for category in categories:
    def preprocessing_data(folder_path):
        image_path=os.path.join(folder_path,category)
        try:
            for Image in os.listdir(folder_path):
                img= cv2.imread(os.path.join(folder_path,Image))
                img_gray= cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                img_gray_resize= cv2.resize(img_gray,(img_size,img_size))
                training_dataset.append([im_gray_array_resize,categories.index(category)])
        except:
            pass

preprocessing_data(path)
CPU times: user 2min 53s, sys: 26.4 s, total: 3min 20s
Wall time: 4min 58s

Here in this step, we are initializing the two list X is for Features and y is for Labels. Next, we are extracting the features and labels and saving it to X and y respectivelly.

Next we are reshaping the array in order to make it suitable for keras,

X=[]
y=[]
training_dataset=np.random.shuffle(training_dataset)
for features, label in training_dataset:
    X.append(features)
    y.append(label)
    
X = np.array(X).reshape(-1, img_size, img_size, 1)

One thing to note that, we are dealing with a dataset in which we have two features i.e Cat and Dog. thus we should randomize the dataset because we don’t want to show similar images to a neural network at one time, this can lead to the unbiased result.

Next we are pickling the features and labels of a dataset, because as we saw above it took around 5 min to process it and we don’t want to process the data again and again

so after pickling, we can load this processed data quickly. even now we can delete the dataset and as we can only work with this pickle data.

This is also very useful in training the model online because we can’t upload the whole dataset of around 1 GB online and process from here, but we can upload the pickle dataset as it will become far less than the original datasets.

For this we first open the path and then dump our feature set i.e X into it. afterthat we close it. Similary we can save label dataset.

pickle_X=open("/Users/pushkarsingh/Desktop/catVSdog/Pickle/X.pickle","wb")
pickle.dump(X,pickle_X)
pickle_X.close()

pickle_y=open("/Users/pushkarsingh/Desktop/catVSdog/Pickle/y.pickle","wb")
pickle.dump(y,pickle_y)
pickle_y.close()

That’s all for this lesson.

In the next lesson we will upload this pickle to google drive and use Google Colab to train our Convolutional Neural Network, We will also understand the basics of TensorFlow and Keras Layers.

Thanks for reading and if you have any doubt or suggestion then please comment below.

Here is the link for CNN part 3: Setting up Google Colab and training Model using TensorFlow and Keras

Leave a Reply

Your email address will not be published. Required fields are marked *