CNN Part 4: How to use TensorBoard to avoid Overfitting

Welcome to part 4 of this series on CNN. In the previous lesson, we trained our model with the high accuracy but the question is if this accuracy is considered to be the best or not. because the model can become overfit in no time.

Actually, the biggest problem in training a neural network is avoiding the overfitting itself, we struggle to fit the model but at the same time we have to struggle to avoid overfitting, In this lesson, we will learn how to visualize a model and how to select the best model using TensorBoard.

Thus, we need TensorBoard, if you already have Tensorflow installed in your system then don’t worry you already have tensorboard too. if not then install TensorFlow using pip command

pip install tensorflow.

Note that Tensorflow is not available for Python 3.7 you should have Python 3.6 or 3.5 to run Tensorflow.

Now once we have installed Tensorflow correctly, its time to create a folder in which we can download our logs and models. Logs and models should be in separate folders. I created another folder ‘run’ and saved the log into it.

Now open the terminal or command prompt and type tensorboard —logdir=full-path-to-the-folder where logs are stored. in my case it stored in the run folder.

tensorboard --logdir=/Users/pushkarsingh/Desktop/TB/logs/run

then you well get an output something like this.

TensorBoard 1.12.2 at http://Pushkars-MacBook-Air.local:6006 (Press CTRL+C to quit)

Now Just copy the link and paste it into the browser and tensorboard will start. you will get a widnow like this. 

TensorBoard

 In the search bar just type epoch and you will get these four charts,

First of all let’s understand these charts.

Accuracy and loss is the in sample accuracy and loss while

Validation Accuracy and Validation loss is out sample accuracy and out sample loss

Accuracy should be rising and loss should be falling. As you can see the validation loss was falling till 4-5th epoch but then it start increasing, 

Note if the graph is looking good in accuracy and loss but worse in validation accuracy and validation loss then that’s the worst model. This means the overfitting has occurred.

Our main deciding factor is the Validation Loss. its really easy for the system to achieve great accuracy in the in sample data, so we should always go for the Val loss.

Now the question arise how can we train a model without overfitting, and the answer is just keep toggeling the values till you get a decent graph or you can train a model with many possibilities in one go

Here is a code for this, 

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras.callbacks import TensorBoard
import pickle
import time
from tensorflow.keras.utils import to_categorical



pickle_in = open("/content/drive/My Drive/Projects/cat vs dog/Xv.pickle","rb")
X = pickle.load(pickle_in)

pickle_in = open("/content/drive/My Drive/Projects/cat vs dog/Y.pickle","rb")
y = pickle.load(pickle_in)
y = to_categorical(y)

X = X/255.0

dense_layers = [0, 1, 2]
layer_sizes = [32, 64, 128]
conv_layers = [1, 2, 3]

for dense_layer in dense_layers:
    for layer_size in layer_sizes:
        for conv_layer in conv_layers:
            NAME = f"{conv_layer}-conv-{layer_size}-nodes-{dense_layer}-dense-{int(time.time()}"
            print(NAME)

            model = Sequential()

            model.add(Conv2D(layer_size, (3, 3), input_shape=X.shape[1:]))
            model.add(Activation('relu'))
            model.add(MaxPooling2D(pool_size=(2, 2)))
            model.add(Dropout(0.3))

            for l in range(conv_layer-1):
                model.add(Conv2D(layer_size, (3, 3)))
                model.add(Activation('relu'))
                model.add(MaxPooling2D(pool_size=(2, 2)))
                model.add(Dropout(0.3))

            model.add(Flatten())

            for _ in range(dense_layer):
                model.add(Dense(layer_size))
                model.add(Activation('relu'))
                

            model.add(Dense(2))
            model.add(Activation('softmax'))

            tensorboard = TensorBoard(log_dir=f"/content/drive/My Drive/Projects/cat vs dog/Logs2/{NAME}")

            model.compile(loss='categorical_crossentropy',
                          optimizer='adam',
                          metrics=['accuracy'],
                          )

            model.fit(X, y,
                      batch_size=32,
                      epochs=10,
                      validation_split=0.3,
                      callbacks=[tensorboard])
            
            model.save(f"/content/drive/My Drive/Projects/cat vs dog/Models/{NAME}.model")

The code is same as in the part 3 but here we are using the for loops to cover as many possibility as we can.

In this way we can cover whole lot of different set of models and then we can analyse them in tensorboard and select for the best one. 

Just train it and it will take around 20 min or more to complete.

After the completion, you will get 27 different set of logs and models in their respective folders. Just download the logs and open the tensorboard, its time to do some analysis, 

Here is the screen looks like when 27 models are uploaded. it looks very complicated but as we learnt earlier, the deciding factor is Val_loss. So we move to the Val_loss graph and select three or four curves from the bottom.

as we can see 3 conv 128 layer and 0 dense is the lowest one, after that 3 conv 128 layers 1 dense. so lets select 3 conv 128 layers and 0 dense and see how the graphs look like.

As we can see the validation graph is decreasing as we wanted and we are getting the accuracy of something around 84-85% which is quite good. so we will go with this model.

Now you understand how to train a model, how to use TensorBoard and how to judge a model, its time to hunt for a dataset online and go for the training.

If you have any doubts or suggestion then please comment below. I will be more then happy to resolve them.

Thanks for reading 

Leave a Reply

Your email address will not be published. Required fields are marked *