NN

Slide 1

• Scaling these models to large inputs such as high-
• resolution images or long temporal sequences requires specialization. We introduce
• the convolutional network for scaling to large images and the recurrent neural
• network for processing temporal sequences. Finally, we present general guidelines
• for the practical methodology involved in designing, building, and configuring an
• application involving deep learning and review some of its applications
• One way to understand feedforward networks is to begin with linear models
• and consider how to overcome their limitations. Linear models, such as logistic
• regression and linear regression, are appealing because they can be fit efficiently
• and reliably, either in closed form or with convex optimization. Linear models also
• have the obvious defect that the model capacity is limited to linear functions, so
• the model cannot understand the interaction between any two input variables.
• To extend linear models to represent nonlinear functions of
• x
• , we can apply
• the linear model not to
• x
• itself but to a transformed input
• φ
• (
• x
• )

Slide 2

• This means that neural networks are usually
• trained by using iterative, gradient-based optimizers that merely drive the cost
• function to a very low value, rather than the linear equation solvers used to train
• linear regression models or the convex optimization algorithms with global conver-
• gence guarantees used to train logistic regression or SVMs.
• Stochastic gradient descent applied to
• nonconvex loss functions has no such convergence guarantee and is sensitive to the
• values of the initial parameters. For feedforward neural networks, it is important to
• initialize all weights to small random values.

Slide 3

• One recurring theme throughout neural network design is that the gradient of
• the cost function must be large and predictable enough to serve as a good guide
• for the learning algorithm. Functions that saturate (become very flat) undermine
• this objective because they make the gradient become very small. In many cases
• this happens because the activation functions used to produce the output of the
• hidden units or the output units saturate. The negative log-likelihood helps to
• avoid this problem for many models. Several output units involve an
• exp
• function
• that can saturate when its argument is very negative. The
• log
• function in the
• negative log-likelihood cost function undoes the
• exp
• of some output units. We will
• discuss the interaction between the cost function and the choice of output unit in
• section 6.2.2.

Slide 4

• One unusual property of the cross-entropy cost used to perform maximum
• likelihood estimation is that it usually does not have a minimum value when applied
• to the models commonly used in practice. For discrete output variables, most
• models are parametrized in such a way that they cannot represent a probability
• of zero or one, but can come arbitrarily close to doing so. Logistic regression
• is an example of such a model. For real-valued output variables, if the model
• can control the density of the output distribution (for example, by learning the
• variance parameter of a Gaussian output distribution) then it becomes possible
• to assign extremely high density to the correct training set outputs, resulting in
• cross-entropy approaching negative infinity. Regularization techniques described
• in chapter 7 provide several different ways of modifying the learning problem so
• that the model cannot reap unlimited reward in this way

Slide 5

• Any time we wish to represent a probability distribution over a discrete variable
• with
• n
• possible values, we may use the softmax function. This can be seen as a
• generalization of the sigmoid function, which was used to represent a probability
• distribution over a binary variable.
• More rarely, softmax functions
• can be used inside the model itself, if we wish the model to choose between one of
• n different options for some internal variable

Slide 6

• he name “softmax” can be somewhat confusing. The function is more closely
• related to the
• arg max
• function than the max function. The term “soft” derives
• from the fact that the softmax function is continuous and differentiable. The
• arg max
• function, with its result represented as a one-hot vector, is not continuous
• or differentiable. The softmax function thus provides a “softened” version of the
• arg max
• . The corresponding soft version of the maximum function is
• softmax
• (
• z
• )
• 
• z
• .

Slide 7

• he functions used in the context of neural
• networks usually have defined left derivatives and defined right derivatives. In the
• case of
• g
• (
• z
• ) =
• max{
• 0
• , z}
• , the left derivative at
• z
• = 0 is 0, and the right derivative
• is 1. Software implementations of neural network training usually return one of
• the one-sided derivatives rather than reporting that the derivative is undefined or
• raising an error. This may be heuristically justified by observing that gradient-
• based optimization on a digital computer is subject to numerical error anyway.
• When a function is asked to evaluate
• g
• (0), it is very unlikely that the underlying
• value truly was 0. Instead, it was likely to be some small value
• 
• that was rounded
• to 0. In some contexts, more theoretically pleasing justifications are available, bu

Slide 8

• Absolute value
• rectification
• fixes
• α
• i
• =
• −
• 1 to obtain
• g
• (
• z
• ) =
• |z|
• . It is used for object recognition
• from images (Jarrett et al., 2009), where it makes sense to seek features that are
• invariant under a polarity reversal of the input illumination. Other generalizations
• of rectified linear units are more broadly applicable. A
• leaky ReLU
• (Maas et al.,
• 2013) fixes
• α
• i
• to a small value like 0.01, while a
• parametric ReLU
• , or
• PReLU
• ,
• treats α
• i
• as a learnable parameter (He et al., 2015)

Slide 9

• Maxout units
• (Goodfellow et al., 2013a) generalize rectified linear units
• further. Instead of applying an element-wise function
• g
• (
• z
• ), maxout units divide
• z
• into groups of
• k
• values. Each maxout unit then outputs the maximum element of
• one of these groups:

Slide 10

• Each maxout unit is now parametrized by
• k
• weight vectors instead of just one,
• so maxout units typically need more regularization than rectified linear units. They
• can work well without regularization if the training set is large and the number of
• pieces per unit is kept low (Cai et al., 2013)

Slide 11

• One must be careful not to apply transformations that would change the correct
• class. For example, optical character recognition tasks require recognizing the
• difference between “b” and “d” and the difference between “6” and “9,” so horizontal
• flips and 180
• ◦
• rotations are not appropriate ways of augmenting datasets for these
• tasks.
• There are also transformations that we would like our classifiers to be invariant
• to but that are not easy to perform. For example, out-of-plane rotation cannot be
• implemented as a simple geometric operation on the input pixels.

Slide 12

• Bagging
• (short for
• bootstrap aggregating
• ) is a technique for reducing general-
• ization error by combining several models (Breiman, 1994). The idea is to train
• several different models separately, then have all the models vote on the output for
• test examples. This is an example of a general strategy in machine learning called
• model averaging
• . Techniques employing this strategy are known as
• ensemble
• methods.
• The reason that model averaging works is that different models will usually
• not make all the same errors on the test set.