This course is a deep dive into the details of deep learning architectures with a focus on learning end-to-end models for these tasks, particularly image classification
This page contains my solutions and approaches for the assignment All source codes of my solutions are available on GitHub
Introduction to PyTorch
You’ve written a lot of code in this assignment to provide a whole host of neural network functionality. Dropout, Batch Norm, and 2D convolutions are some of the workhorses of deep learning in computer vision. You’ve also worked hard to make your code efficient and vectorized.
For the last part of this assignment, though, we’re going to leave behind your beautiful codebase and instead migrate to one of two popular deep learning frameworks: in this instance, PyTorch.
Why do we use deep learning frameworks?
Our code will now run on GPUs! This will allow our models to train much faster. When using a framework like PyTorch you can harness the power of the GPU for your own custom neural network architectures without having to write CUDA code directly (which is beyond the scope of this class).
In this class, we want you to be ready to use one of these frameworks for your project so you can experiment more efficiently than if you were writing every feature you want to use by hand.
We want you to stand on the shoulders of giants! PyTorch is an excellent frameworks that will make your lives a lot easier, and now that you understand their guts, you are free to use them :)
Finally, we want you to be exposed to the sort of deep learning code you might run into in academia or industry.
What is PyTorch?
PyTorch is a system for executing dynamic computational graphs over Tensor objects that behave similarly as numpy ndarray. It comes with a powerful automatic differentiation engine that removes the need for manual back-propagation.
How do I learn PyTorch?
One of our former instructors, Justin Johnson, made an excellent tutorial for PyTorch.
You can also find the detailed API doc here. If you have other questions that are not addressed by the API docs, the PyTorch forum is a much better place to ask than StackOverflow.
Table of Contents
This assignment has 5 parts. You will learn PyTorch on three different levels of abstraction, which will help you understand it better and prepare you for the final project.
Part I, Preparation: we will use CIFAR-10 dataset.
Part II, Barebones PyTorch: Abstraction level 1, we will work directly with the lowest-level PyTorch Tensors.
Part III, PyTorch Module API: Abstraction level 2, we will use nn.Module to define arbitrary neural network architecture.
Part IV, PyTorch Sequential API: Abstraction level 3, we will use nn.Sequential to define a linear feed-forward network very conveniently.
Part V, CIFAR-10 open-ended challenge: please implement your own network to get as high accuracy as possible on CIFAR-10. You can experiment with any layer, optimizer, hyperparameters or other advanced features.
Here is a table of comparison:
API
Flexibility
Convenience
Barebone
High
Low
nn.Module
High
Medium
nn.Sequential
Low
High
GPU
You can manually switch to a GPU device on Colab by clicking Runtime -> Change runtime type and selecting GPU under Hardware Accelerator. You should do this before running the following cells to import packages, since the kernel gets restarted upon switching runtimes.
import torchimport torch.nn as nnimport torch.optim as optimfrom torch.utils.data import DataLoaderfrom torch.utils.data import samplerimport torchvision.datasets as dsetimport torchvision.transforms as Timport numpy as npUSE_GPU =Truedtype = torch.float32 # We will be using float throughout this tutorial.if USE_GPU and torch.cuda.is_available(): device = torch.device('cuda')else: device = torch.device('cpu')# Constant to control how frequently we print train loss.print_every =100print('using device:', device)
using device: cuda
Part I. Preparation
Now, let’s load the CIFAR-10 dataset. This might take a couple minutes the first time you do it, but the files should stay cached after that.
In previous parts of the assignment we had to write our own code to download the CIFAR-10 dataset, preprocess it, and iterate through it in minibatches; PyTorch provides convenient tools to automate this process for us.
NUM_TRAIN =49000# The torchvision.transforms package provides tools for preprocessing data# and for performing data augmentation; here we set up a transform to# preprocess the data by subtracting the mean RGB value and dividing by the# standard deviation of each RGB value; we've hardcoded the mean and std.transform = T.Compose([ T.ToTensor(), T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)) ])# We set up a Dataset object for each split (train / val / test); Datasets load# training examples one at a time, so we wrap each Dataset in a DataLoader which# iterates through the Dataset and forms minibatches. We divide the CIFAR-10# training set into train and val sets by passing a Sampler object to the# DataLoader telling how it should sample from the underlying Dataset.cifar10_train = dset.CIFAR10('./cs231n/datasets', train=True, download=True, transform=transform)loader_train = DataLoader(cifar10_train, batch_size=64, sampler=sampler.SubsetRandomSampler(range(NUM_TRAIN)))cifar10_val = dset.CIFAR10('./cs231n/datasets', train=True, download=True, transform=transform)loader_val = DataLoader(cifar10_val, batch_size=64, sampler=sampler.SubsetRandomSampler(range(NUM_TRAIN, 50000)))cifar10_test = dset.CIFAR10('./cs231n/datasets', train=False, download=True, transform=transform)loader_test = DataLoader(cifar10_test, batch_size=64)
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Part II. Barebones PyTorch
PyTorch ships with high-level APIs to help us define model architectures conveniently, which we will cover in Part II of this tutorial. In this section, we will start with the barebone PyTorch elements to understand the autograd engine better. After this exercise, you will come to appreciate the high-level model API more.
We will start with a simple fully-connected ReLU network with two hidden layers and no biases for CIFAR classification. This implementation computes the forward pass using operations on PyTorch Tensors, and uses PyTorch autograd to compute gradients. It is important that you understand every line, because you will write a harder version after the example.
When we create a PyTorch Tensor with requires_grad=True, then operations involving that Tensor will not just compute values; they will also build up a computational graph in the background, allowing us to easily backpropagate through the graph to compute gradients of some Tensors with respect to a downstream loss. Concretely if x is a Tensor with x.requires_grad == True then after backpropagation x.grad will be another Tensor holding the gradient of x with respect to the scalar loss at the end.
PyTorch Tensors: Flatten Function
A PyTorch Tensor is conceptionally similar to a numpy array: it is an n-dimensional grid of numbers, and like numpy PyTorch provides many functions to efficiently operate on Tensors. As a simple example, we provide a flatten function below which reshapes image data for use in a fully-connected neural network.
Recall that image data is typically stored in a Tensor of shape N x C x H x W, where:
N is the number of datapoints
C is the number of channels
H is the height of the intermediate feature map in pixels
W is the height of the intermediate feature map in pixels
This is the right way to represent the data when we are doing something like a 2D convolution, that needs spatial understanding of where the intermediate features are relative to each other. When we use fully connected affine layers to process the image, however, we want each datapoint to be represented by a single vector – it’s no longer useful to segregate the different channels, rows, and columns of the data. So, we use a “flatten” operation to collapse the C x H x W values per representation into a single long vector. The flatten function below first reads in the N, C, H, and W values from a given batch of data, and then returns a “view” of that data. “View” is analogous to numpy’s “reshape” method: it reshapes x’s dimensions to be N x ??, where ?? is allowed to be anything (in this case, it will be C x H x W, but we don’t need to specify that explicitly).
def flatten(x): N = x.shape[0] # read in N, C, H, Wreturn x.view(N, -1) # "flatten" the C * H * W values into a single vector per imagedef test_flatten(): x = torch.arange(12).view(2, 1, 3, 2)print('Before flattening: ', x)print('After flattening: ', flatten(x))test_flatten()
Here we define a function two_layer_fc which performs the forward pass of a two-layer fully-connected ReLU network on a batch of image data. After defining the forward pass we check that it doesn’t crash and that it produces outputs of the right shape by running zeros through the network.
You don’t have to write any code here, but it’s important that you read and understand the implementation.
import torch.nn.functional as F # useful stateless functionsdef two_layer_fc(x, params):""" A fully-connected neural networks; the architecture is: NN is fully connected -> ReLU -> fully connected layer. Note that this function only defines the forward pass; PyTorch will take care of the backward pass for us. The input to the network will be a minibatch of data, of shape (N, d1, ..., dM) where d1 * ... * dM = D. The hidden layer will have H units, and the output layer will produce scores for C classes. Inputs: - x: A PyTorch Tensor of shape (N, d1, ..., dM) giving a minibatch of input data. - params: A list [w1, w2] of PyTorch Tensors giving weights for the network; w1 has shape (D, H) and w2 has shape (H, C). Returns: - scores: A PyTorch Tensor of shape (N, C) giving classification scores for the input data x. """# first we flatten the image x = flatten(x) # shape: [batch_size, C x H x W] w1, w2 = params# Forward pass: compute predicted y using operations on Tensors. Since w1 and# w2 have requires_grad=True, operations involving these Tensors will cause# PyTorch to build a computational graph, allowing automatic computation of# gradients. Since we are no longer implementing the backward pass by hand we# don't need to keep references to intermediate values.# you can also use `.clamp(min=0)`, equivalent to F.relu() x = F.relu(x.mm(w1)) x = x.mm(w2)return xdef two_layer_fc_test(): hidden_layer_size =42 x = torch.zeros((64, 50), dtype=dtype) # minibatch size 64, feature dimension 50 w1 = torch.zeros((50, hidden_layer_size), dtype=dtype) w2 = torch.zeros((hidden_layer_size, 10), dtype=dtype) scores = two_layer_fc(x, [w1, w2])print(scores.size()) # you should see [64, 10]two_layer_fc_test()
torch.Size([64, 10])
Barebones PyTorch: Three-Layer ConvNet
Here you will complete the implementation of the function three_layer_convnet, which will perform the forward pass of a three-layer convolutional network. Like above, we can immediately test our implementation by passing zeros through the network. The network should have the following architecture:
A convolutional layer (with bias) with channel_1 filters, each with shape KW1 x KH1, and zero-padding of two
ReLU nonlinearity
A convolutional layer (with bias) with channel_2 filters, each with shape KW2 x KH2, and zero-padding of one
ReLU nonlinearity
Fully-connected layer with bias, producing scores for C classes.
Note that we have no softmax activation here after our fully-connected layer: this is because PyTorch’s cross entropy loss performs a softmax activation for you, and by bundling that step in makes computation more efficient.
HINT: For convolutions: http://pytorch.org/docs/stable/nn.html#torch.nn.functional.conv2d; pay attention to the shapes of convolutional filters!
def three_layer_convnet(x, params):""" Performs the forward pass of a three-layer convolutional network with the architecture defined above. Inputs: - x: A PyTorch Tensor of shape (N, 3, H, W) giving a minibatch of images - params: A list of PyTorch Tensors giving the weights and biases for the network; should contain the following: - conv_w1: PyTorch Tensor of shape (channel_1, 3, KH1, KW1) giving weights for the first convolutional layer - conv_b1: PyTorch Tensor of shape (channel_1,) giving biases for the first convolutional layer - conv_w2: PyTorch Tensor of shape (channel_2, channel_1, KH2, KW2) giving weights for the second convolutional layer - conv_b2: PyTorch Tensor of shape (channel_2,) giving biases for the second convolutional layer - fc_w: PyTorch Tensor giving weights for the fully-connected layer. Can you figure out what the shape should be? - fc_b: PyTorch Tensor giving biases for the fully-connected layer. Can you figure out what the shape should be? Returns: - scores: PyTorch Tensor of shape (N, C) giving classification scores for x """ conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b = params scores =None################################################################################# TODO: Implement the forward pass for the three-layer ConvNet. ################################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)***** N, Ch, H, W = x.size() C = fc_b.size() channel_1, Ch, KH1, KW1 = conv_w1.size() channel_2, channel_1, KH2, KW2 = conv_w2.size() H1, W1 = (((H+4-KH1)/1) +1, ((W+4-KW1)/1) +1) H2, W2 = (((H1+2-KH2)/1) +1, ((W1+2-KW2)/1) +1) fc_w_in, fc_w_out =(H2*W2, C) x = F.conv2d(x, conv_w1, conv_b1, padding=2) x = F.relu(x) x = F.conv2d(x, conv_w2, conv_b2, padding=1) x = F.relu(x) x = flatten(x).mm(fc_w) + fc_b scores = x# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****################################################################################# END OF YOUR CODE #################################################################################return scores
After defining the forward pass of the ConvNet above, run the following cell to test your implementation.
When you run this function, scores should have shape (64, 10).
def three_layer_convnet_test(): x = torch.zeros((64, 3, 32, 32), dtype=dtype) # minibatch size 64, image size [3, 32, 32] conv_w1 = torch.zeros((6, 3, 5, 5), dtype=dtype) # [out_channel, in_channel, kernel_H, kernel_W] conv_b1 = torch.zeros((6,)) # out_channel conv_w2 = torch.zeros((9, 6, 3, 3), dtype=dtype) # [out_channel, in_channel, kernel_H, kernel_W] conv_b2 = torch.zeros((9,)) # out_channel# you must calculate the shape of the tensor after two conv layers, before the fully-connected layer fc_w = torch.zeros((9*32*32, 10)) fc_b = torch.zeros(10) scores = three_layer_convnet(x, [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b])print(scores.size()) # you should see [64, 10]three_layer_convnet_test()
torch.Size([64, 10])
Barebones PyTorch: Initialization
Let’s write a couple utility methods to initialize the weight matrices for our models.
random_weight(shape) initializes a weight tensor with the Kaiming normalization method.
zero_weight(shape) initializes a weight tensor with all zeros. Useful for instantiating bias parameters.
The random_weight function uses the Kaiming normal initialization method, described in:
He et al, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, ICCV 2015, https://arxiv.org/abs/1502.01852
def random_weight(shape):""" Create random Tensors for weights; setting requires_grad=True means that we want to compute gradients for these Tensors during the backward pass. We use Kaiming normalization: sqrt(2 / fan_in) """iflen(shape) ==2: # FC weight fan_in = shape[0]else: fan_in = np.prod(shape[1:]) # conv weight [out_channel, in_channel, kH, kW]# randn is standard normal distribution generator. w = torch.randn(shape, device=device, dtype=dtype) * np.sqrt(2./ fan_in) w.requires_grad =Truereturn wdef zero_weight(shape):return torch.zeros(shape, device=device, dtype=dtype, requires_grad=True)# create a weight of shape [3 x 5]# you should see the type `torch.cuda.FloatTensor` if you use GPU. # Otherwise it should be `torch.FloatTensor`random_weight((3, 5))
When training the model we will use the following function to check the accuracy of our model on the training or validation sets.
When checking accuracy we don’t need to compute any gradients; as a result we don’t need PyTorch to build a computational graph for us when we compute scores. To prevent a graph from being built we scope our computation under a torch.no_grad() context manager.
def check_accuracy_part2(loader, model_fn, params):""" Check the accuracy of a classification model. Inputs: - loader: A DataLoader for the data split we want to check - model_fn: A function that performs the forward pass of the model, with the signature scores = model_fn(x, params) - params: List of PyTorch Tensors giving parameters of the model Returns: Nothing, but prints the accuracy of the model """ split ='val'if loader.dataset.train else'test'print('Checking accuracy on the %s set'% split) num_correct, num_samples =0, 0with torch.no_grad():for x, y in loader: x = x.to(device=device, dtype=dtype) # move to device, e.g. GPU y = y.to(device=device, dtype=torch.int64) scores = model_fn(x, params) _, preds = scores.max(1) num_correct += (preds == y).sum() num_samples += preds.size(0) acc =float(num_correct) / num_samplesprint('Got %d / %d correct (%.2f%%)'% (num_correct, num_samples, 100* acc))
BareBones PyTorch: Training Loop
We can now set up a basic training loop to train our network. We will train the model using stochastic gradient descent without momentum. We will use torch.functional.cross_entropy to compute the loss; you can read about it here.
The training loop takes as input the neural network function, a list of initialized parameters ([w1, w2] in our example), and learning rate.
def train_part2(model_fn, params, learning_rate):""" Train a model on CIFAR-10. Inputs: - model_fn: A Python function that performs the forward pass of the model. It should have the signature scores = model_fn(x, params) where x is a PyTorch Tensor of image data, params is a list of PyTorch Tensors giving model weights, and scores is a PyTorch Tensor of shape (N, C) giving scores for the elements in x. - params: List of PyTorch Tensors giving weights for the model - learning_rate: Python scalar giving the learning rate to use for SGD Returns: Nothing """for t, (x, y) inenumerate(loader_train):# Move the data to the proper device (GPU or CPU) x = x.to(device=device, dtype=dtype) y = y.to(device=device, dtype=torch.long)# Forward pass: compute scores and loss scores = model_fn(x, params) loss = F.cross_entropy(scores, y)# Backward pass: PyTorch figures out which Tensors in the computational# graph has requires_grad=True and uses backpropagation to compute the# gradient of the loss with respect to these Tensors, and stores the# gradients in the .grad attribute of each Tensor. loss.backward()# Update parameters. We don't want to backpropagate through the# parameter updates, so we scope the updates under a torch.no_grad()# context manager to prevent a computational graph from being built.with torch.no_grad():for w in params: w -= learning_rate * w.grad# Manually zero the gradients after running the backward pass w.grad.zero_()if t % print_every ==0:print('Iteration %d, loss = %.4f'% (t, loss.item())) check_accuracy_part2(loader_val, model_fn, params)print()
BareBones PyTorch: Train a Two-Layer Network
Now we are ready to run the training loop. We need to explicitly allocate tensors for the fully connected weights, w1 and w2.
Each minibatch of CIFAR has 64 examples, so the tensor shape is [64, 3, 32, 32].
After flattening, x shape should be [64, 3 * 32 * 32]. This will be the size of the first dimension of w1. The second dimension of w1 is the hidden layer size, which will also be the first dimension of w2.
Finally, the output of the network is a 10-dimensional vector that represents the probability distribution over 10 classes.
You don’t need to tune any hyperparameters but you should see accuracies above 40% after training for one epoch.
Iteration 0, loss = 3.4425
Checking accuracy on the val set
Got 124 / 1000 correct (12.40%)
Iteration 100, loss = 1.7402
Checking accuracy on the val set
Got 343 / 1000 correct (34.30%)
Iteration 200, loss = 2.1003
Checking accuracy on the val set
Got 380 / 1000 correct (38.00%)
Iteration 300, loss = 2.0187
Checking accuracy on the val set
Got 387 / 1000 correct (38.70%)
Iteration 400, loss = 1.9708
Checking accuracy on the val set
Got 359 / 1000 correct (35.90%)
Iteration 500, loss = 1.6966
Checking accuracy on the val set
Got 414 / 1000 correct (41.40%)
Iteration 600, loss = 1.8955
Checking accuracy on the val set
Got 407 / 1000 correct (40.70%)
Iteration 700, loss = 1.8057
Checking accuracy on the val set
Got 456 / 1000 correct (45.60%)
BareBones PyTorch: Training a ConvNet
In the below you should use the functions defined above to train a three-layer convolutional network on CIFAR. The network should have the following architecture:
Convolutional layer (with bias) with 32 5x5 filters, with zero-padding of 2
ReLU
Convolutional layer (with bias) with 16 3x3 filters, with zero-padding of 1
ReLU
Fully-connected layer (with bias) to compute scores for 10 classes
You should initialize your weight matrices using the random_weight function defined above, and you should initialize your bias vectors using the zero_weight function above.
You don’t need to tune any hyperparameters, but if everything works correctly you should achieve an accuracy above 42% after one epoch.
learning_rate =3e-3channel_1 =32channel_2 =16conv_w1 =Noneconv_b1 =Noneconv_w2 =Noneconv_b2 =Nonefc_w =Nonefc_b =None################################################################################# TODO: Initialize the parameters of a three-layer ConvNet. ################################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****KW1 = KH1 =5KW2 = KH2 =3conv_w1 = random_weight((channel_1, 3, KH1, KW1))conv_b1 = zero_weight((channel_1))conv_w2 = random_weight((channel_2, channel_1, KH2, KW2))conv_b2 = zero_weight((channel_2))fc_w = random_weight((channel_2 *32*32, 10))fc_b = zero_weight((10))# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****################################################################################# END OF YOUR CODE #################################################################################params = [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b]train_part2(three_layer_convnet, params, learning_rate)
Iteration 0, loss = 3.9675
Checking accuracy on the val set
Got 103 / 1000 correct (10.30%)
Iteration 100, loss = 1.7996
Checking accuracy on the val set
Got 386 / 1000 correct (38.60%)
Iteration 200, loss = 1.6441
Checking accuracy on the val set
Got 412 / 1000 correct (41.20%)
Iteration 300, loss = 1.4472
Checking accuracy on the val set
Got 439 / 1000 correct (43.90%)
Iteration 400, loss = 1.2470
Checking accuracy on the val set
Got 448 / 1000 correct (44.80%)
Iteration 500, loss = 1.1712
Checking accuracy on the val set
Got 483 / 1000 correct (48.30%)
Iteration 600, loss = 1.3478
Checking accuracy on the val set
Got 481 / 1000 correct (48.10%)
Iteration 700, loss = 1.4701
Checking accuracy on the val set
Got 500 / 1000 correct (50.00%)
Part III. PyTorch Module API
Barebone PyTorch requires that we track all the parameter tensors by hand. This is fine for small networks with a few tensors, but it would be extremely inconvenient and error-prone to track tens or hundreds of tensors in larger networks.
PyTorch provides the nn.Module API for you to define arbitrary network architectures, while tracking every learnable parameters for you. In Part II, we implemented SGD ourselves. PyTorch also provides the torch.optim package that implements all the common optimizers, such as RMSProp, Adagrad, and Adam. It even supports approximate second-order methods like L-BFGS! You can refer to the doc for the exact specifications of each optimizer.
To use the Module API, follow the steps below:
Subclass nn.Module. Give your network class an intuitive name like TwoLayerFC.
In the constructor __init__(), define all the layers you need as class attributes. Layer objects like nn.Linear and nn.Conv2d are themselves nn.Module subclasses and contain learnable parameters, so that you don’t have to instantiate the raw tensors yourself. nn.Module will track these internal parameters for you. Refer to the doc to learn more about the dozens of builtin layers. Warning: don’t forget to call the super().__init__() first!
In the forward() method, define the connectivity of your network. You should use the attributes defined in __init__ as function calls that take tensor as input and output the “transformed” tensor. Do not create any new layers with learnable parameters in forward()! All of them must be declared upfront in __init__.
After you define your Module subclass, you can instantiate it as an object and call it just like the NN forward function in part II.
Module API: Two-Layer Network
Here is a concrete example of a 2-layer fully connected network:
class TwoLayerFC(nn.Module):def__init__(self, input_size, hidden_size, num_classes):super().__init__()# assign layer objects to class attributesself.fc1 = nn.Linear(input_size, hidden_size)# nn.init package contains convenient initialization methods# http://pytorch.org/docs/master/nn.html#torch-nn-init nn.init.kaiming_normal_(self.fc1.weight)self.fc2 = nn.Linear(hidden_size, num_classes) nn.init.kaiming_normal_(self.fc2.weight)def forward(self, x):# forward always defines connectivity x = flatten(x) scores =self.fc2(F.relu(self.fc1(x)))return scoresdef test_TwoLayerFC(): input_size =50 x = torch.zeros((64, input_size), dtype=dtype) # minibatch size 64, feature dimension 50 model = TwoLayerFC(input_size, 42, 10) scores = model(x)print(scores.size()) # you should see [64, 10]test_TwoLayerFC()
torch.Size([64, 10])
Module API: Three-Layer ConvNet
It’s your turn to implement a 3-layer ConvNet followed by a fully connected layer. The network architecture should be the same as in Part II:
Convolutional layer with channel_1 5x5 filters with zero-padding of 2
ReLU
Convolutional layer with channel_2 3x3 filters with zero-padding of 1
ReLU
Fully-connected layer to num_classes classes
You should initialize the weight matrices of the model using the Kaiming normal initialization method.
After you implement the three-layer ConvNet, the test_ThreeLayerConvNet function will run your implementation; it should print (64, 10) for the shape of the output scores.
class ThreeLayerConvNet(nn.Module):def__init__(self, in_channel, channel_1, channel_2, num_classes):super().__init__()######################################################################### TODO: Set up the layers you need for a three-layer ConvNet with the ## architecture defined above. ########################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****self.conv1 = nn.Conv2d(in_channel, channel_1, 5, padding=2) nn.init.kaiming_normal_(self.conv1.weight)self.conv2 = nn.Conv2d(channel_1, channel_2, 3, padding=1) nn.init.kaiming_normal_(self.conv1.weight)self.linear = nn.Linear(channel_2 *32*32, num_classes) nn.init.kaiming_normal_(self.linear.weight)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****######################################################################### END OF YOUR CODE # ########################################################################def forward(self, x): scores =None######################################################################### TODO: Implement the forward function for a 3-layer ConvNet. you ## should use the layers you defined in __init__ and specify the ## connectivity of those layers in forward() ########################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)***** x = F.relu(self.conv1(x)) x = F.relu(self.conv2(x)) scores =self.linear(flatten(x))# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****######################################################################### END OF YOUR CODE #########################################################################return scoresdef test_ThreeLayerConvNet(): x = torch.zeros((64, 3, 32, 32), dtype=dtype) # minibatch size 64, image size [3, 32, 32] model = ThreeLayerConvNet(in_channel=3, channel_1=12, channel_2=8, num_classes=10) scores = model(x)print(scores.size()) # you should see [64, 10]test_ThreeLayerConvNet()
torch.Size([64, 10])
Module API: Check Accuracy
Given the validation or test set, we can check the classification accuracy of a neural network.
This version is slightly different from the one in part II. You don’t manually pass in the parameters anymore.
def check_accuracy_part34(loader, model):if loader.dataset.train:print('Checking accuracy on validation set')else:print('Checking accuracy on test set') num_correct =0 num_samples =0 model.eval() # set model to evaluation modewith torch.no_grad():for x, y in loader: x = x.to(device=device, dtype=dtype) # move to device, e.g. GPU y = y.to(device=device, dtype=torch.long) scores = model(x) _, preds = scores.max(1) num_correct += (preds == y).sum() num_samples += preds.size(0) acc =float(num_correct) / num_samplesprint('Got %d / %d correct (%.2f)'% (num_correct, num_samples, 100* acc))
Module API: Training Loop
We also use a slightly different training loop. Rather than updating the values of the weights ourselves, we use an Optimizer object from the torch.optim package, which abstract the notion of an optimization algorithm and provides implementations of most of the algorithms commonly used to optimize neural networks.
def train_part34(model, optimizer, epochs=1):""" Train a model on CIFAR-10 using the PyTorch Module API. Inputs: - model: A PyTorch Module giving the model to train. - optimizer: An Optimizer object we will use to train the model - epochs: (Optional) A Python integer giving the number of epochs to train for Returns: Nothing, but prints model accuracies during training. """ model = model.to(device=device) # move the model parameters to CPU/GPUfor e inrange(epochs):for t, (x, y) inenumerate(loader_train): model.train() # put model to training mode x = x.to(device=device, dtype=dtype) # move to device, e.g. GPU y = y.to(device=device, dtype=torch.long) scores = model(x) loss = F.cross_entropy(scores, y)# Zero out all of the gradients for the variables which the optimizer# will update. optimizer.zero_grad()# This is the backwards pass: compute the gradient of the loss with# respect to each parameter of the model. loss.backward()# Actually update the parameters of the model using the gradients# computed by the backwards pass. optimizer.step()if t % print_every ==0:print('Iteration %d, loss = %.4f'% (t, loss.item())) check_accuracy_part34(loader_val, model)print()
Module API: Train a Two-Layer Network
Now we are ready to run the training loop. In contrast to part II, we don’t explicitly allocate parameter tensors anymore.
Simply pass the input size, hidden layer size, and number of classes (i.e. output size) to the constructor of TwoLayerFC.
You also need to define an optimizer that tracks all the learnable parameters inside TwoLayerFC.
You don’t need to tune any hyperparameters, but you should see model accuracies above 40% after training for one epoch.
Iteration 0, loss = 3.9317
Checking accuracy on validation set
Got 147 / 1000 correct (14.70)
Iteration 100, loss = 2.0209
Checking accuracy on validation set
Got 313 / 1000 correct (31.30)
Iteration 200, loss = 2.0517
Checking accuracy on validation set
Got 362 / 1000 correct (36.20)
Iteration 300, loss = 2.3523
Checking accuracy on validation set
Got 351 / 1000 correct (35.10)
Iteration 400, loss = 2.1379
Checking accuracy on validation set
Got 427 / 1000 correct (42.70)
Iteration 500, loss = 1.7526
Checking accuracy on validation set
Got 396 / 1000 correct (39.60)
Iteration 600, loss = 1.7753
Checking accuracy on validation set
Got 422 / 1000 correct (42.20)
Iteration 700, loss = 1.3974
Checking accuracy on validation set
Got 440 / 1000 correct (44.00)
Module API: Train a Three-Layer ConvNet
You should now use the Module API to train a three-layer ConvNet on CIFAR. This should look very similar to training the two-layer network! You don’t need to tune any hyperparameters, but you should achieve above above 45% after training for one epoch.
You should train the model using stochastic gradient descent without momentum.
learning_rate =3e-3channel_1 =32channel_2 =16model =Noneoptimizer =None################################################################################# TODO: Instantiate your ThreeLayerConvNet model and a corresponding optimizer ################################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****in_channel =3num_classes =10model = ThreeLayerConvNet(3, channel_1, channel_2, num_classes)optimizer = optim.SGD(model.parameters(), learning_rate)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****################################################################################# END OF YOUR CODE #################################################################################train_part34(model, optimizer)
Iteration 0, loss = 2.6068
Checking accuracy on validation set
Got 109 / 1000 correct (10.90)
Iteration 100, loss = 2.0093
Checking accuracy on validation set
Got 325 / 1000 correct (32.50)
Iteration 200, loss = 1.7616
Checking accuracy on validation set
Got 369 / 1000 correct (36.90)
Iteration 300, loss = 1.6231
Checking accuracy on validation set
Got 402 / 1000 correct (40.20)
Iteration 400, loss = 1.7143
Checking accuracy on validation set
Got 421 / 1000 correct (42.10)
Iteration 500, loss = 1.5316
Checking accuracy on validation set
Got 415 / 1000 correct (41.50)
Iteration 600, loss = 1.4928
Checking accuracy on validation set
Got 462 / 1000 correct (46.20)
Iteration 700, loss = 1.5596
Checking accuracy on validation set
Got 458 / 1000 correct (45.80)
Part IV. PyTorch Sequential API
Part III introduced the PyTorch Module API, which allows you to define arbitrary learnable layers and their connectivity.
For simple models like a stack of feed forward layers, you still need to go through 3 steps: subclass nn.Module, assign layers to class attributes in __init__, and call each layer one by one in forward(). Is there a more convenient way?
Fortunately, PyTorch provides a container Module called nn.Sequential, which merges the above steps into one. It is not as flexible as nn.Module, because you cannot specify more complex topology than a feed-forward stack, but it’s good enough for many use cases.
Sequential API: Two-Layer Network
Let’s see how to rewrite our two-layer fully connected network example with nn.Sequential, and train it using the training loop defined above.
Again, you don’t need to tune any hyperparameters here, but you shoud achieve above 40% accuracy after one epoch of training.
# We need to wrap `flatten` function in a module in order to stack it# in nn.Sequentialclass Flatten(nn.Module):def forward(self, x):return flatten(x)hidden_layer_size =4000learning_rate =1e-2model = nn.Sequential( Flatten(), nn.Linear(3*32*32, hidden_layer_size), nn.ReLU(), nn.Linear(hidden_layer_size, 10),)# you can use Nesterov momentum in optim.SGDoptimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, nesterov=True)train_part34(model, optimizer)
Iteration 0, loss = 2.3635
Checking accuracy on validation set
Got 141 / 1000 correct (14.10)
Iteration 100, loss = 1.5867
Checking accuracy on validation set
Got 373 / 1000 correct (37.30)
Iteration 200, loss = 1.7681
Checking accuracy on validation set
Got 414 / 1000 correct (41.40)
Iteration 300, loss = 1.8605
Checking accuracy on validation set
Got 407 / 1000 correct (40.70)
Iteration 400, loss = 1.5008
Checking accuracy on validation set
Got 417 / 1000 correct (41.70)
Iteration 500, loss = 1.5089
Checking accuracy on validation set
Got 433 / 1000 correct (43.30)
Iteration 600, loss = 2.1685
Checking accuracy on validation set
Got 439 / 1000 correct (43.90)
Iteration 700, loss = 1.8974
Checking accuracy on validation set
Got 438 / 1000 correct (43.80)
Sequential API: Three-Layer ConvNet
Here you should use nn.Sequential to define and train a three-layer ConvNet with the same architecture we used in Part III:
Convolutional layer (with bias) with 32 5x5 filters, with zero-padding of 2
ReLU
Convolutional layer (with bias) with 16 3x3 filters, with zero-padding of 1
ReLU
Fully-connected layer (with bias) to compute scores for 10 classes
You can use the default PyTorch weight initialization.
You should optimize your model using stochastic gradient descent with Nesterov momentum 0.9.
Again, you don’t need to tune any hyperparameters but you should see accuracy above 55% after one epoch of training.
from torch.nn.modules.activation import ReLUchannel_1 =32channel_2 =16learning_rate =1e-2model =Noneoptimizer =None################################################################################# TODO: Rewrite the 2-layer ConvNet with bias from Part III with the ## Sequential API. ################################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****in_channel =3num_classes =10model = nn.Sequential( nn.Conv2d(in_channel, channel_1, 5, padding=2), nn.ReLU(), nn.Conv2d(channel_1, channel_2, 3, padding=1), nn.ReLU(), Flatten(), nn.Linear(channel_2 *32*32, num_classes))optimizer = optim.SGD(model.parameters(), learning_rate, momentum=0.9, nesterov=True)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****################################################################################# END OF YOUR CODE #################################################################################train_part34(model, optimizer)
Iteration 0, loss = 2.3079
Checking accuracy on validation set
Got 141 / 1000 correct (14.10)
Iteration 100, loss = 1.4638
Checking accuracy on validation set
Got 450 / 1000 correct (45.00)
Iteration 200, loss = 1.3307
Checking accuracy on validation set
Got 523 / 1000 correct (52.30)
Iteration 300, loss = 1.3672
Checking accuracy on validation set
Got 546 / 1000 correct (54.60)
Iteration 400, loss = 1.2416
Checking accuracy on validation set
Got 520 / 1000 correct (52.00)
Iteration 500, loss = 1.4224
Checking accuracy on validation set
Got 554 / 1000 correct (55.40)
Iteration 600, loss = 1.2014
Checking accuracy on validation set
Got 569 / 1000 correct (56.90)
Iteration 700, loss = 1.0010
Checking accuracy on validation set
Got 578 / 1000 correct (57.80)
Part V. CIFAR-10 open-ended challenge
In this section, you can experiment with whatever ConvNet architecture you’d like on CIFAR-10.
Now it’s your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves at least 70% accuracy on the CIFAR-10 validation set within 10 epochs. You can use the check_accuracy and train functions from above. You can use either nn.Module or nn.Sequential API.
Describe what you did at the end of this notebook.
Here are the official API documentation for each component. One note: what we call in the class “spatial batch norm” is called “BatchNorm2D” in PyTorch.
Layers in torch.nn package: http://pytorch.org/docs/stable/nn.html
Filter size: Above we used 5x5; would smaller filters be more efficient?
Number of filters: Above we used 32 filters. Do more or fewer do better?
Pooling vs Strided Convolution: Do you use max pooling or just stride convolutions?
Batch normalization: Try adding spatial batch normalization after convolution layers and vanilla batch normalization after affine layers. Do your networks train faster?
Network architecture: The network above has two layers of trainable parameters. Can you do better with a deep network? Good architectures to try include:
[conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]
[conv-relu-conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]
[batchnorm-relu-conv]xN -> [affine]xM -> [softmax or SVM]
Global Average Pooling: Instead of flattening and then having multiple affine layers, perform convolutions until your image gets small (7x7 or so) and then perform an average pooling operation to get to a 1x1 image picture (1, 1 , Filter#), which is then reshaped into a (Filter#) vector. This is used in Google’s Inception Network (See Table 1 for their architecture).
Regularization: Add l2 weight regularization, or perhaps use Dropout.
Tips for training
For each network architecture that you try, you should tune the learning rate and other hyperparameters. When doing this there are a couple important things to keep in mind:
If the parameters are working well, you should see improvement within a few hundred iterations
Remember the coarse-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.
Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.
You should use the validation set for hyperparameter search, and save your test set for evaluating your architecture on the best parameters as selected by the validation set.
Going above and beyond
If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are not required to implement any of these, but don’t miss the fun if you have time!
Alternative optimizers: you can try Adam, Adagrad, RMSprop, etc.
Alternative activation functions such as leaky ReLU, parametric ReLU, ELU, or MaxOut.
Model ensembles
Data augmentation
New Architectures
ResNets where the input from the previous layer is added to the output.
DenseNets where inputs into previous layers are concatenated together.
################################################################################# TODO: # # Experiment with any architectures, optimizers, and hyperparameters. ## Achieve AT LEAST 70% accuracy on the *validation set* within 10 epochs. ## ## Note that you can use the check_accuracy function to evaluate on either ## the test set or the validation set, by passing either loader_test or ## loader_val as the second argument to check_accuracy. You should not touch ## the test set until you have finished your architecture and hyperparameter ## tuning, and only run the test set once at the end to report a final value. #################################################################################model =Noneoptimizer =None# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****num_classes =10in_channel =3model = nn.Sequential(# Features nn.Conv2d(in_channel, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(True), nn.Conv2d(64, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(True), nn.Conv2d(64, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(True), nn.MaxPool2d(2, 2), nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(True), nn.Conv2d(128, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(True), nn.Conv2d(128, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(True), nn.MaxPool2d(2, 2),# AvgPool nn.AdaptiveAvgPool2d((8, 8)),# Flatten Flatten(),# Classifier nn.Linear(128*8*8, 1024), nn.ReLU(True), nn.Dropout(), nn.Linear(1024, 1024), nn.ReLU(True), nn.Dropout(), nn.Linear(1024, num_classes))# optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9, nesterov=True)optimizer = optim.Adam(model.parameters())# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****################################################################################# END OF YOUR CODE ################################################################################## You should get at least 70% accuracy.# You may modify the number of epochs to any number below 15.train_part34(model, optimizer, epochs=10)
Iteration 0, loss = 2.3433
Checking accuracy on validation set
Got 79 / 1000 correct (7.90)
Iteration 100, loss = 1.8039
Checking accuracy on validation set
Got 361 / 1000 correct (36.10)
Iteration 200, loss = 1.7020
Checking accuracy on validation set
Got 385 / 1000 correct (38.50)
Iteration 300, loss = 1.6002
Checking accuracy on validation set
Got 414 / 1000 correct (41.40)
Iteration 400, loss = 1.6643
Checking accuracy on validation set
Got 474 / 1000 correct (47.40)
Iteration 500, loss = 1.6907
Checking accuracy on validation set
Got 493 / 1000 correct (49.30)
Iteration 600, loss = 1.5363
Checking accuracy on validation set
Got 551 / 1000 correct (55.10)
Iteration 700, loss = 1.3180
Checking accuracy on validation set
Got 515 / 1000 correct (51.50)
Iteration 0, loss = 1.5074
Checking accuracy on validation set
Got 542 / 1000 correct (54.20)
Iteration 100, loss = 1.3768
Checking accuracy on validation set
Got 539 / 1000 correct (53.90)
Iteration 200, loss = 1.2128
Checking accuracy on validation set
Got 574 / 1000 correct (57.40)
Iteration 300, loss = 1.2710
Checking accuracy on validation set
Got 553 / 1000 correct (55.30)
Iteration 400, loss = 0.8962
Checking accuracy on validation set
Got 599 / 1000 correct (59.90)
Iteration 500, loss = 0.9972
Checking accuracy on validation set
Got 621 / 1000 correct (62.10)
Iteration 600, loss = 1.4752
Checking accuracy on validation set
Got 578 / 1000 correct (57.80)
Iteration 700, loss = 1.2342
Checking accuracy on validation set
Got 633 / 1000 correct (63.30)
Iteration 0, loss = 1.0089
Checking accuracy on validation set
Got 628 / 1000 correct (62.80)
Iteration 100, loss = 1.2676
Checking accuracy on validation set
Got 638 / 1000 correct (63.80)
Iteration 200, loss = 1.0300
Checking accuracy on validation set
Got 657 / 1000 correct (65.70)
Iteration 300, loss = 1.2805
Checking accuracy on validation set
Got 672 / 1000 correct (67.20)
Iteration 400, loss = 1.2017
Checking accuracy on validation set
Got 664 / 1000 correct (66.40)
Iteration 500, loss = 1.0106
Checking accuracy on validation set
Got 656 / 1000 correct (65.60)
Iteration 600, loss = 1.0961
Checking accuracy on validation set
Got 661 / 1000 correct (66.10)
Iteration 700, loss = 0.9384
Checking accuracy on validation set
Got 709 / 1000 correct (70.90)
Iteration 0, loss = 0.9305
Checking accuracy on validation set
Got 686 / 1000 correct (68.60)
Iteration 100, loss = 0.7547
Checking accuracy on validation set
Got 698 / 1000 correct (69.80)
Iteration 200, loss = 0.8938
Checking accuracy on validation set
Got 727 / 1000 correct (72.70)
Iteration 300, loss = 0.9808
Checking accuracy on validation set
Got 720 / 1000 correct (72.00)
Iteration 400, loss = 1.1267
Checking accuracy on validation set
Got 725 / 1000 correct (72.50)
Iteration 500, loss = 1.0420
Checking accuracy on validation set
Got 716 / 1000 correct (71.60)
Iteration 600, loss = 0.8533
Checking accuracy on validation set
Got 745 / 1000 correct (74.50)
Iteration 700, loss = 0.7182
Checking accuracy on validation set
Got 721 / 1000 correct (72.10)
Iteration 0, loss = 0.7148
Checking accuracy on validation set
Got 746 / 1000 correct (74.60)
Iteration 100, loss = 0.6595
Checking accuracy on validation set
Got 725 / 1000 correct (72.50)
Iteration 200, loss = 0.8453
Checking accuracy on validation set
Got 725 / 1000 correct (72.50)
Iteration 300, loss = 0.9387
Checking accuracy on validation set
Got 728 / 1000 correct (72.80)
Iteration 400, loss = 0.7530
Checking accuracy on validation set
Got 736 / 1000 correct (73.60)
Iteration 500, loss = 0.5019
Checking accuracy on validation set
Got 740 / 1000 correct (74.00)
Iteration 600, loss = 0.6756
Checking accuracy on validation set
Got 759 / 1000 correct (75.90)
Iteration 700, loss = 0.7967
Checking accuracy on validation set
Got 759 / 1000 correct (75.90)
Iteration 0, loss = 0.6872
Checking accuracy on validation set
Got 740 / 1000 correct (74.00)
Iteration 100, loss = 0.6063
Checking accuracy on validation set
Got 761 / 1000 correct (76.10)
Iteration 200, loss = 0.7173
Checking accuracy on validation set
Got 776 / 1000 correct (77.60)
Iteration 300, loss = 0.5167
Checking accuracy on validation set
Got 758 / 1000 correct (75.80)
Iteration 400, loss = 0.9691
Checking accuracy on validation set
Got 746 / 1000 correct (74.60)
Iteration 500, loss = 0.5447
Checking accuracy on validation set
Got 791 / 1000 correct (79.10)
Iteration 600, loss = 0.7218
Checking accuracy on validation set
Got 787 / 1000 correct (78.70)
Iteration 700, loss = 0.6918
Checking accuracy on validation set
Got 798 / 1000 correct (79.80)
Iteration 0, loss = 0.6830
Checking accuracy on validation set
Got 738 / 1000 correct (73.80)
Iteration 100, loss = 0.6006
Checking accuracy on validation set
Got 789 / 1000 correct (78.90)
Iteration 200, loss = 0.8499
Checking accuracy on validation set
Got 786 / 1000 correct (78.60)
Iteration 300, loss = 0.7119
Checking accuracy on validation set
Got 784 / 1000 correct (78.40)
Iteration 400, loss = 0.4287
Checking accuracy on validation set
Got 800 / 1000 correct (80.00)
Iteration 500, loss = 0.6572
Checking accuracy on validation set
Got 819 / 1000 correct (81.90)
Iteration 600, loss = 0.5079
Checking accuracy on validation set
Got 803 / 1000 correct (80.30)
Iteration 700, loss = 0.4734
Checking accuracy on validation set
Got 794 / 1000 correct (79.40)
Iteration 0, loss = 0.4609
Checking accuracy on validation set
Got 807 / 1000 correct (80.70)
Iteration 100, loss = 0.4870
Checking accuracy on validation set
Got 795 / 1000 correct (79.50)
Iteration 200, loss = 0.6740
Checking accuracy on validation set
Got 812 / 1000 correct (81.20)
Iteration 300, loss = 0.6449
Checking accuracy on validation set
Got 808 / 1000 correct (80.80)
Iteration 400, loss = 0.4098
Checking accuracy on validation set
Got 787 / 1000 correct (78.70)
Iteration 500, loss = 0.4778
Checking accuracy on validation set
Got 801 / 1000 correct (80.10)
Iteration 600, loss = 0.2906
Checking accuracy on validation set
Got 804 / 1000 correct (80.40)
Iteration 700, loss = 0.4786
Checking accuracy on validation set
Got 801 / 1000 correct (80.10)
Iteration 0, loss = 0.3460
Checking accuracy on validation set
Got 806 / 1000 correct (80.60)
Iteration 100, loss = 0.4821
Checking accuracy on validation set
Got 805 / 1000 correct (80.50)
Iteration 200, loss = 0.5182
Checking accuracy on validation set
Got 822 / 1000 correct (82.20)
Iteration 300, loss = 0.6620
Checking accuracy on validation set
Got 821 / 1000 correct (82.10)
Iteration 400, loss = 0.5648
Checking accuracy on validation set
Got 816 / 1000 correct (81.60)
Iteration 500, loss = 0.6143
Checking accuracy on validation set
Got 818 / 1000 correct (81.80)
Iteration 600, loss = 0.5816
Checking accuracy on validation set
Got 824 / 1000 correct (82.40)
Iteration 700, loss = 0.6218
Checking accuracy on validation set
Got 804 / 1000 correct (80.40)
Iteration 0, loss = 0.5109
Checking accuracy on validation set
Got 819 / 1000 correct (81.90)
Iteration 100, loss = 0.3318
Checking accuracy on validation set
Got 822 / 1000 correct (82.20)
Iteration 200, loss = 0.4391
Checking accuracy on validation set
Got 825 / 1000 correct (82.50)
Iteration 300, loss = 0.4257
Checking accuracy on validation set
Got 830 / 1000 correct (83.00)
Iteration 400, loss = 0.5367
Checking accuracy on validation set
Got 827 / 1000 correct (82.70)
Iteration 500, loss = 0.4858
Checking accuracy on validation set
Got 825 / 1000 correct (82.50)
Iteration 600, loss = 0.3824
Checking accuracy on validation set
Got 828 / 1000 correct (82.80)
Iteration 700, loss = 0.4988
Checking accuracy on validation set
Got 833 / 1000 correct (83.30)
Describe what you did
In the cell below you should write an explanation of what you did, any additional features that you implemented, and/or any graphs that you made in the process of training and evaluating your network.
Answer: I was inspired by VGG16, but VGG16 is designed for ImageNet which is way bigger dataset than Cifar-10. So, I just took the idea of staking 3x3 convolutions followed by MaxPooling layer.
I used 2 of [conv]xN-MaxPool layers instead of 5 which is used in VGG16.
I also added spatial batch norm layers between conv and relu layers
Using AdaptivePooling is also not my idea, I took it from the VGG16 paper
I also simplified the classifier FC net from 4096-4096-1000 which is used in VGG16 to 1024-1024-10
Here is the architecture:
INPUT: [32 X 32 X 3]
CONV3-64: [32 X 32 X 64]
CONV3-64: [32 X 32 X 64]
CONV3-64: [32 X 32 X 64]
POOL-2: [16 X 16 X 64]
CONV3-128: [16 X 16 X 128]
CONV3-128: [16 X 16 X 128]
CONV3-128: [16 X 16 X 128]
POOL-2: [8 X 8 X 128]
FC: [1 X 1 X 1024]
FC: [1 X 1 X 1024]
FC: [1 X 1 X 10]
Test set – run this only once
Now that we’ve gotten a result we’re happy with, we test our final model on the test set (which you should store in best_model). Think about how this compares to your validation set accuracy.