DNN_M3_Optimization.pdf

DEEP LEARNING
MODULE # 3 : OPTIMIZATION
DL Team, BITS Pilani

The instructor is gratefully acknowledging
the authors who made their course
materials freely available online.
Deep Learning 2 / 80

What we learn ...
1 3.1 CHALLENGES IN NEURAL NETWORK OPTIMIZATION - SADDLE
POINTS AND PLATEAU
2 3.2 NON-CONVEX OPTIMIZATION INTUITION
3 3.3 OVERVIEW OF OPTIMIZATION ALGORITHMS
4 3.4 MOMENTUM BASED ALGORITHMS
5 3.5 ALGORITHMS WITH ADAPTIVE LEARNING RATES

Variance vs Bias
Figure: Underfitting; High Bias Figure: Low variance and bias Figure: Overfitting; High variance
Our aim is to achieve low variance and bias.

Deep Learning Recipe (Andrew Ng)
Train an
initial Model
High
Bias?
Optimization
Bigger network
Try different
Architecture
High
Variance?
Regularization
Get more data
Try different
Architecture
Stop
no
yes yes
no

In This Segment
1 Optimization
2 Challenges in Optimization
3 Optimization Algorithms
Gradient Descent
Effect of Learning Rate on Gradient Descent
Stochastic Gradient Descent
Dynamic Learning Rate
Minibatch Stochastic Gradient Descent
Momentum
Adagrad
RMSPRop
ADAM

Optimization
Training of DNN is an iterative process.
We may be training the DNN on a very large dataset or big data.
The aim of DNN is improve the performance measure evaluated on the
training set, i.e. reduce the bias.
The performance measure P is optimized indirectly. Train the DNN to find the
parameters θ that significantly reduce the cost function J(θ), so that P
improves.

Optimization Algorithms
Optimization algorithms train deep learning models.
Optimization algorithms are the tools that allow
I continue updating model parameters
I to minimize the value of the loss function, as evaluated on the training set.
In optimization, a loss function is often referred to as the objective function of
the optimization problem.
By tradition and convention most optimization algorithms are concerned with
minimization.

Optimization

Why Optimization Algorithms?
The performance of the optimization algorithm directly affects the model’s
training efficiency.
Understanding the principles of different optimization algorithms and the role
of their hyperparameters will enable us to tune the hyperparameters in a
targeted manner to improve the performance of deep learning models.
The goal of optimization is to reduce the training error. The goal of deep
learning is to reduce the generalization error, this requires reduction in
overfitting also.

In This Segment
1 Optimization
Gradient Descent
Momentum
Adagrad
RMSPRop
ADAM

Challenges in Optimization
Local minima
Saddle points
Vanishing gradients

Finding Minimum of a Function
Find the of x at which f0
(x) = 0. i.e
df(x)
dx
= 0.
Solution is at a turning point.
A turning point is a value where the
derivatives turn from positive to negative
or vice versa.
Turning points occur at minima, maxima
or inflection (saddle) points.

Derivatives of a Function
Both maxima and minima are turning points.
Both maxima and minima have zero derivatives.
The second derivative f00
(x) is negative at maxima and positive at minima.

All locations with zero derivative are
critical points.
I These can be local minima, local
maxima or saddle points.

All locations with zero derivative are
critical points.
I These can be local minima, local
maxima or saddle points.
The second derivative is
I ≥ 0 at minima
I ≤ 0 at maxima
I = 0 at saddle point
It is more complicated for functions of
multiple variables.

Finding Minimum of a Function
Find the of x at which f0
(x) = 0. i.e
df(x)
dx
= 0.
Solution xsoln is at a turning point.
Check the double derivative f00
(xsoln) i.e
f00
(xsoln)df0(x)
dx
.
If f00
(xsoln) is positive, xsoln is minimum.
If f00
(xsoln) is negative, xsoln is maximum.
If f00
(xsoln) is zero, xsoln is saddle point.

Local Minima
For any objective function f(x), if the
value of f(x) at x is smaller than the
values of f(x) at any other points in the
vicinity of x, then f(x) could be a local
minimum.
If the value of f(x) at x is the minimum of
the objective function over the entire
domain, then f(x) is the global minimum.
In minibatch stochastic gradient descent,
the natural variation of gradients over
minibatches is able to dislodge the
parameters from local minima.

Saddle Points
In high dimensional space, the
local points with gradients is
closer to zero are known as
saddle points.
Saddle points tend to slow down
the learning process.
A saddle point is any location
where all gradients of a function
vanish but which is neither a
global nor a local minimum.
Eg: f(x, y) = x2
− y2
saddle point at (0, 0); maximum
wrt y and minimum wrt x

Gradient of a Scalar Function with Multiple
Variables
The gradient ∇f(X) of a
mutli-variate input X is a
multiplicative factor that gives us
the change in f(X) for tiny
variations in X.
df(X) = ∇f(X)d(X)

Variables
Consider f(X) = f(x1, x2, . . . , xn))
The gradient is given by
∇f(X) =

∂f(X)
∂x1
,
∂f(X)
∂x2
, . . . ,
∂f(X)
∂xn

The second derivative of the function is given by the Hessain.
∇2
f(x) =





∂2
f
∂x2
1
∂2
f
∂x1x2
. . . ∂2
f
∂x1xn
∂2
f
∂x2x1
∂2
f
∂x2
2
. . . ∂2
f
∂x2xn
. . .
∂2
f
∂xnx1
∂2
f
∂xnx2
. . . ∂2
f
∂x2
n






Variables
The optimum point is the turning point. The gradient is zero.

Solution of Unconstrained Minimization
Solve for X where the gradient equation equal to zero.
∇f(X) = 0
Compute the Hessian matrix ∇2
f(X) at the candidate solution and verify that
Local Minimum
I Eigenvalues of Hessian matrix are all positive.
Local Maximum
I Eigenvalues of Hessian matrix are all negative.
Saddle Point
I Eigenvalues of Hessian matrix at the zero-gradient position are negative and
positive.

Example
Minimize
f(x1, x2, x3) = x2
1 + x1(1 − x2) + x2
2 − x2x3 + x2
3 + x3
Gradient
∇f(X) =


2x1 + 1 − x2
−x1 + 2x2 − x3
−x2 + 2x3 + 1



Example
Set Gradient = 0
∇f(X) =


2x1 + 1 − x2
−x1 + 2x2 − x3
−x2 + 2x3 + 1

 =


0
0
0


Solve the 3 equations in 3 unknowns.
X = (x1, x2, x3) =


−1
−1
−1



Example
Compute the Hessian
∇2
f(X) =


2 −1 0
−1 2 −1
0 −1 2


Evaluate the eigenvalues of the Hessain matrix
λ1 = 3.414 λ2 = 0.586 λ3 = 2
All Eigen vectors are positive.
The obtained solution is minimum.
X = (x1, x2, x3) =


−1
−1
−1



Vanishing Gradients
Function f(x) = tanh(x)
f0
(x) = 1 − tanh2(x)
f0
(4) = 0.0013.
Gradient of f is close to nil.
Vanishing gradients can cause
optimization to stall.
Reparameterization of the
problem helps.
Good initialization of the
parameters.

In This Segment
1 Optimization
Gradient Descent
Momentum
Adagrad
RMSPRop
ADAM

How to find Global Minima?
It is not possible to solve ∇f(X) = 0.
I The function to minimize / maximize may have a intractable form
In these situation, iterative solutions are used.
I Begin with a guess for the optimal X and refine it iteratively until the correct value
is obtained.

Find Global Minima Iteratively
Iterative solutions
I Start with an initial guess X0.
I Update the guess towards a better value of f(X).
I Stop when F(X) no longer decreases.
Challenges
I Which direction to step in?
I How big the steps should be?

Gradient Descent
Iterative Solution
I Start at some initial guess.
I Find the direction to shift this point to decrease error.
F This can be found from the derivative of the function.
F A positive derivative implies, move left to decrease error.
F A negative derivative implies, move right to decrease error.
I Shift the point in this direction.

Gradient Descent
Algorithm 1: Gradient Descent Algorithm
1 Initialize x0
2 while f0
(xk
) 6= 0 do
3 xk+1
= xk
− step sizef0
(xk
)

Gradient Descent
First order Gradient Descent algos
consider the first order derivatives
to get the magnitude and direction
of update.
def gd ( eta , f grad ) :
x = 10.0
r e s u l t s = [ x ]
f o r i in range ( 1 0 ) :
x −= eta * f grad ( x )
r e s u l t s . append ( f l o a t ( x ) )
return r e s u l t s

Numerical Example
Consider an error function
E(w1, w2) = 0.05 +
(w1 − 3)2
4
+
(w2 − 4)2
9
−
(w1 − 3)(w2 − 4)
6
Different variants of gradient descent algorithm can be used to minimize this error
function w.r.t (w1, w2). Assume (w1, w2) = (1, 1) at time (t − 1) and after update
(w1, w2) = (1.5, 2.0) at time (t). Assume α = 1.5, β = 0.6, η = 0.3.

Numerical Example
Compute the value that minimizes (w1, w2). Compute the minimum possible value
of error.
E(w1, w2) = 0.05 +
(w1 − 3)2
4
+
(w2 − 4)2
9
−
(w1 − 3)(w2 − 4)
6
∂E
∂w1
=
2(w1 − 3)
4
−
(w2 − 4)
6
= 0
∂E
∂w2
=
2(w2 − 4)
9
−
(w1 − 3)
6
= 0
Solving w1 = 3
w2 = 4
Substituting E = 0.05

Numerical Example
Compute the value of (w1, w2) at time (t + 1) if standard gradient descent is used.
E(w1, w2) = 0.05 +
(w1 − 3)2
4
+
(w2 − 4)2
9
−
(w1 − 3)(w2 − 4)
6
∂E
∂w1
=
2(w1 − 3)
4
−
(w2 − 4)
6
∂E
∂w2
=
2(w2 − 4)
9
−
(w1 − 3)
6
at (t + 1) w1(t+1) = w1(t) − η
∂E
∂w1
= 1.5 − 0.3 ∗

(1.5 − 3)
2
−
(2 − 4)
6

= 1.625
w2(t+1) = w2(t) − η
∂E
∂w2
= 2 − 0.3 ∗

2(2 − 4)
9
−
(1.5 − 3)
6

= 2.05

Learning Rate
The role of the learning rate is to moderate the degree to which weights are
changed at each step.
Learning rate η is set by the algorithm designer.
If the learning rate that is too small, it will cause parameters to update very
slowly, requiring more iterations to get a better solution.
If the learning rate that is too large, the solution oscillates and in the worst
case it might even diverge.

Gradient Descent
Small learning rate
I Slow Learning
I Definitely converge
Large learning rate
I Oscillations
I Worst case it might diverge

Example
Error surface is given by E(x, y, z) = 3x2
+2y2
+4z2
+6. Assume gradient descent
is used to find the minimum of this error surface. What is the optimal learning rate
that leads to fastest convergence to the global minimum?
E(x, y, z) = 3x2
+ 2y2
+ 4z2
+ 6
ηxopt = 1/6
ηyopt = 1/4
ηzopt = 1/8
optimal learning rate for convergence = min[ηxopt , ηyopt , ηzopt ] = 1/8
Largest learning rate for convergence = min[2ηxopt , 2ηyopt , 2ηzopt ] = 0.33
Learning rate for divergence 2ηopt = 2 ∗ 1/8 = 0.25

In deep learning, the objective function is the average of the loss functions for
each example in the training dataset.
Given a training dataset of m examples, let fi (θ) is the loss function with
respect to the training example of index i, where θ is the parameter vector.
Computational cost of each iteration is O(1).

Stochastic Gradient Descent Algorithm
Algorithm 2: STOCHASTIC GRADIENT DESCENT ALGORITHM
Data: Learning Rate η
Data: Initial Parameters θ
1 while stopping criterion not met do
2 for each example i do
3 Perform Forward prop on X{i}
to compute ŷ and cost J(θ)
4 Perform Backprop on X{i}
, Y{i}

to compute gradient g
5 Update parameters as θ ← θ − ηg

Trajectory of the variables in the stochastic gradient descent is much more
noisy. This is due to the stochastic nature of the gradient. Even near the
minimum, uncertainty is injected by the instantaneous gradients.
def sgd ( x1 , x2 , s1 , s2 , f grad ) :
g1 , g2 = f grad ( x1 , x2 )
# Simulate noisy gradient
g1 += np . random . normal (0.0 , 1 , ( 1 , ) )
g2 += np . random . normal (0.0 , 1 , ( 1 , ) )
e t a t = eta * l r ( )
return ( x1−e t a t *g1 , x2−e t a t *g2 , 0 , 0)

Replace η with a time-dependent learning rate η(t) adds to the complexity of
controlling convergence of an optimization algorithm.
A few basic strategies that adjust η over time.
I Piecewise constant
F decrease the learning rate, e.g., whenever progress in optimization stalls.
F This is a common strategy for training deep networks
η(t) = ηi if ti ≤ t ≤ ti+1
I Exponential decay
F Leads to premature stopping before the algorithm has converged.
η(t) = η0e−λt
I Polynomial decay
η(t) = η0(βt + 1)−α

Exponential Decay
Variance in the parameters is significantly reduced.
The algorithm fails to converge at all.
def e x p o n e n t i a l l r ( ) :
global t
t += 1
return math . exp(−0.1 * t )

Polynomial Decay
Learning rate decays with the inverse square root of the number of steps.
Convergence gets better after only 50 steps.
def po ly no mia l lr ( ) :
global t
t += 1
return (1 + 0.1 * t ) ** ( −0.5)

Gap
Gradient descent
I Uses the full dataset to compute gradients and to update parameters, one pass
at a time.
I Gradient Descent is not particularly data efficient whenever data is very similar.
Stochastic Gradient descent
I Processes one observation at a time to make progress.
I Stochastic Gradient Descent is not particularly computationally efficient since
CPUs and GPUs cannot exploit the full power of vectorization.
I For noisy gradients, choice of the learning rate is critical.
I If we decrease it too rapidly, convergence stalls.
I If we are too lenient, we fail to converge to a good enough solution since noise
keeps on driving us away from optimality.
Minibatch SGD
I Accelerate computation, or better or computational efficiency.
I Averaging gradients reduce the amount of variance.

Suppose we have 4 million examples in the training set.
X = [X(1)
, X(2)
, . . . , X(m)
]
Y = [Y(1)
, Y(2)
, . . . , Y(m)
]
The gradient descent algorithm process the entire dataset of m = 4 million
examples before proceeding to the next step. This is cumbersome and time
consuming.
Hence split the training dataset.

Let the mini-batch size be 1000.
Then we have 4000 mini-batches, each having 1000 examples.
X = [X(1)
, X(2)
, . . . , X(m)
]
= [X(1)
, X(2)
, . . . , X(1000)
, X(1001)
, . . . , X(2000)
, . . . , X(m)
]
= X{1}
, X{2}
, . . . , X{m/b}
Y = [Y(1)
, Y(2)
, . . . , Y(m)
]
= [Y(1)
, Y(2)
, . . . , Y(1000)
, Y(1001)
, . . . , Y(2000)
, . . . , Y(m)
]
= Y{1}
, Y{2}
, . . . , Y{m/b}
The minibatch t is denoted as X{t}
, Y{t}
The Gradient descent algorithm will be repeated for t batches.

Algorithm
In each iteration, we first randomly sample a minibatch B consisting of a fixed
number of training examples.
We then compute the derivative (gradient) of the average loss on the minibatch
with regard to the model parameters.
Finally, we multiply the gradient by a predetermined positive value η and
subtract the resulting term from the current parameter values.
θ ←
η
|B|
X
i∈B
∂θloss(t)
(θ)
|B| represents the number of examples in each minibatch (the batch size) and
η denotes the learning rate.

Algorithm
Gradients at time t is calculated as
gt,t−1 = ∂theta
1
|B|
X
i∈Bt
f(xi , θt−1) =
1
|B|
X
i∈Bt
hi,t−1
|B| represents the number of examples in each minibatch (the batch size) and
eta denotes the learning rate.
hi,t−1 = ∂θf(xi , θt−1) is the stochastic gradient descent for sample i using the
weights updated at time t − 1.

Algorithm
Algorithm 3: MINIBATCH STOCHASTIC GRADIENT DESCENT ALGORITHM
Data: Learning Rate η
2 for t in range (1, m/batch size) do
3 Perform Forward prop on X{t}
4 Perform Backprop on X{t}
, Y{t}

5 Update parameters as θ ← θ − ηg

Minibatch Size
Typical mini batch size is taken as some power of 2; say 32, 64, 128, 256.
Better utilization of multicore architecture.
Amount of memory scales with batch size.

Leaky Average in Minibatch SGD
Replace the gradient computation by a ”leaky average” for better variance
reduction.
β ∈ (0, 1)
This effectively replaces the instantaneous gradient by one that’s been
averaged over multiple past gradients.
v is called momentum.
Momentum accumulates past gradients.
Large β amounts to a long-range average and small β amounts to only a slight
correction relative to a gradient method.
The new gradient replacement no longer points into the direction of steepest
descent on a particular instance any longer but rather in the direction of a
weighted average of past gradients.

Momentum Method Example
Consider a moderately distorted ellipsoid objective f(x) = 0.1x2
1 + 2x2
2
f has its minimum at (0, 0). This function is very flat in x1 direction.
For η = 0.4 Without momentum
The gradient in the x2 direction
oscillates than in the horizontal x1
direction.
For η = 0.6 Without momentum
Convergence in the x1 direction
improves but the overall solution
quality is diverging.

Momentum Method Example
Consider a moderately distorted ellipsoid objective f(x) = 0.1x2
1 + 2x2
2
Apply momentum for η = 0.6
For β = 0.5
Converges well. Lesser
oscillations. Larger steps in x1
direction.
For β = 0.25
Reduced convergence. More
oscillations. Larger magnitude of
oscillations.

Momentum
Compute an exponentially weighted average of the gradients and use it to
update the parameters.
The momentum algorithm accumulates an exponentially decaying moving
average of past gradients and continues to move in their direction.
The momentum introduces a velocity v that denotes the direction and speed at
which the parameters change.
The velocity is set to an exponentially decaying average of the negative
gradient.
The Momentum parameter β ∈ [0, 1) is a hyperparameter and typically takes
a value of 0.9.

SGD with Momentum

SGD with Momentum Algorithm
Algorithm 4: STOCHASTIC GRADIENT DESCENT WITH MOMENTUM
Data: Learning Rate η, Momentum Parameter β
Data: Initial Parameters θ , Inital velocities v
, Y{t}

5 Update velocities as v = βv + (1 − β)g
6 Update parameters as θ ← θ − ηv

Momentum method: Summary
Momentum replaces gradients with a leaky average over past gradients. This
accelerates convergence significantly.
Momentum prevents stalling of the optimization process that is much more
likely to occur for stochastic gradient descent.
The effective number of gradients is given by 1/(1 − β) due to exponentiated
down weighting of past data.
Implementation is quite straightforward but it requires us to store an additional
state vector (momentum v).

Numerical Example
Consider an error function
E(w1, w2) = 0.05 +
(w1 − 3)2
4
+
(w2 − 4)2
9
−
(w1 − 3)(w2 − 4)
6
Different variants of gradient descent algorithm can be used to minimize this error
function w.r.t (w1, w2). Assume (w1, w2) = (1, 1) at time (t − 1) and after update
(w1, w2) = (1.5, 2.0) at time (t). Assume α = 1.5, β = 0.6, η = 0.3. Compute the
value of (w1, w2) at time (t + 1) if momentum is used.
∂E
∂w1
=
2(w1 − 3)
4
−
(w2 − 4)
6
∂E
∂w2
=
2(w2 − 4)
9
−
(w1 − 3)
6

Numerical Example
at (t + 1) w1(t+1) = w1(t) − η
∂E
∂w1
+ β∆w1
= 1.5 − 0.3 ∗

(1.5 − 3)
2
−
(2 − 4)
6

+ 0.9 ∗ (1.5 − 1)
= 2.075
w2(t+1) = w2(t) − η
∂E
∂w2
+ β∆w1
= 2 − 0.3 ∗

2(2 − 4)
9
−
(1.5 − 3)
6

+ 0.9 ∗ (2 − 1)
= 2.958

Adagrad
Used for features that occur infrequently (sparse features).
Adagrad uses aggregate of the squares of previously observed gradients.

Adagrad Algorithm
Variable st to accumulate past gradient variance.
Operation are applied coordinate wise.
p
1/v has entries
p
1/v1 and u · v
has entries ui vi .
η is the learning rate and is an additive constant that ensures that we do not
divide by 0.
Initialize s0 = 0.
gt = ∂w loss(yt , f(xt , w))
st = st−1g2
t
wt = wt −
η
√
st +
· gt

Adagrad: Summary
Adagrad decreases the learning rate dynamically on a per-coordinate basis.
It uses the magnitude of the gradient as a means of adjusting how quickly
progress is achieved - coordinates with large gradients are compensated with
a smaller learning rate.
If the optimization problem has a rather uneven structure Adagrad can help
mitigate the distortion.
Adagrad is particularly effective for sparse features where the learning rate
needs to decrease more slowly for infrequently occurring terms.
On deep learning problems Adagrad can sometimes be too aggressive in
reducing learning rates.

RMSProp (Root Mean Square Prop)
Adapts the learning rates of all model parameters by scaling the gradients into
an exponentially weighted moving average.
Performs better with non-convex setting.
Adagrad use learning rate that decreases at a predefined schedule of
effectively O(t−1/2
).
RMSProp algorithm decouples rate scheduling from coordinate-adaptive
learning rates. This is essential for non-convex optimization.

RMSProp Algorithm
Use leaky average to accumulate past gradient variance.
Parameter γ 0.
The constant is set to 10−6
to ensure that we do not suffer from division by
zero or overly large step sizes.
st = γst−1 + (1 − γ)g2
t
wt = wt −
η
√
st +
gt

RMSProp Algorithm
Algorithm 5: RMSPROP
Data: Learning Rate η, Decay Rate γ
Data: Small constant = 10−7
to stabilize division
1 Initialize accumulated gradient s = 0
, Y{t}

6 Compute accumulated squared gradient s = γs + (1 − γ)g2
7 Update parameters as θ ← θ − η
√
s+
g

RMSProp Example
def rmsprop 2d ( x1 , x2 , s1 , s2 ) :
g1 , g2 , eps = 0.2* x1 , 4*x2 , 1e−6
s1 = gamma* s1 +(1−gamma) * g1**2
s2 = gamma* s2 +(1−gamma) * g2**2
x1 −= eta / math . sqrt ( s1 + eps ) * g1
x2 −= eta / math . sqrt ( s2 + eps ) * g2
return x1 , x2 , s1 , s2

RMSProp: Summary
RMSProp is very similar to Adagrad as both use the square of the gradient to
scale coefficients.
RMSProp shares with momentum the leaky averaging. However, RMSProp
uses the technique to adjust the coefficient-wise preconditioner.
The learning rate needs to be scheduled by the experimenter in practice.
The coefficient γ determines how long the history is when adjusting the
per-coordinate scale.

Review of algorithms learned so far
1 Stochastic gradient descent
I more effective than Gradient Descent when solving optimization problems, e.g.,
due to its inherent resilience to redundant data.
2 Minibatch Stochastic gradient descent
I affords significant additional efficiency arising from vectorization, using larger
sets of observations in one minibatch. This is the key to efficient multi-machine,
multi-GPU and overall parallel processing.
3 Momentum
I added a mechanism for aggregating a history of past gradients to accelerate
convergence.

Review of algorithms learned so far
1 Adagrad
I used per-coordinate scaling to allow for a computationally efficient
preconditioner.
2 RMSProp
I decoupled per-coordinate scaling from a learning rate adjustment.

Adam (Adaptive Moments)
Adam combines all these techniques into one efficient learning algorithm.
Very effective and robust algorithm
Adam can diverge due to poor variance control. (disadvantage)
Adam uses exponential weighted moving averages (also known as leaky
averaging) to obtain an estimate of both the momentum and also the second
moment of the gradient.

Adam Algorithm
State variables
vt ← β1vt−1 + (1 − β1)gt
st ← β2st−1 + (1 − β2)g2
t
β1 and β2 are nonnegative weighting parameters. Common choices for them
are β1 = 0.9 and β2 = 0.999. That is, the variance estimate moves much
more slowly than the momentum term.
Initialize v0 = s0 = 0.

Adam Algorithm
Normalize the state variables
v̂t =
vt
1 − βt
1
ŝt =
st
1 − βt
2
Rescale the gradient
ĝt =
ηv̂t
√
ŝt +
Compute updates
wt ← wt−1 −
ηv̂
√
ŝ +

Adam Algorithm
Algorithm 6: ADAM
Data: Learning Rate η = 0.001, Momentum parameter β1 = 0.9, Decay Rate
β2 = 0.999, Small constant = 10−7
to stabilize division
Data: Initial Parameters θ; Initialize first and second moments v = 0, s = 0
, Y{t}

5 Update first moment as v ← β1v + (1 − β1)g
6 Update second moment as s = β2s + (1 − β2)g2
7 Perform bias correction as v̂ = v
1−βt
1
and ŝ = s
1−βt
2
8 Update parameters as θ ← θ − ηv̂
√
ŝ+

Adam: Summary
Adam combines features of many optimization algorithms into a fairly robust
update rule.
Adam uses bias correction to adjust for a slow startup when estimating
momentum and a second moment.
For gradients with significant variance we may encounter issues with
convergence. They can be amended by using larger minibatches or by
switching to an improved estimate for state variables.
Yogi algorithm offers such an alternative.

Numerical Example
The training data (x1, y1) = (3.5, 0.5) for a single Sigmoid neuron and initial values
of w = −2.0, b = −2.0, η = 0.10, β1 = 0.90, β2 = 0.99, = 1e − 8, sW =
0 and sB = 0 are provided. Showing all the calculations, find out the values of
w, b, sW , sB, rW , rB after one iteration of Adam. Use BCE as loss function. Apply
bias-correction.

Numerical Example
Forward Pass and Calculate Loss
ŷ = σ(wx + b) = σ(−2.0 × 3.5 − 2.0) ≈ 0.0180
L = −[y × log(ŷ) + (1 − y) × log(1 − ŷ)
L = −[0.5 × log(0.0180) + (1 − 0.5) × log(1 − 0.0180)] ≈ 4.0076
Calculate Gradients
∇w L = (ŷ − y) × x = (0.0180 − 0.5) × 3.5 ≈ −0.493
∇bL = ŷ − y = 0.0180 − 0.5 ≈ −0.482
Update First Moments
sW = β1 × sW + (1 − β1) × ∇w L = 0.90 × 0 − 0.10 × (−0.493) ≈ 0.04437
sB = β1 × sB + (1 − β1) × ∇bL = 0.90 × 0 − 0.10 × (−0.482) ≈ 0.04382

Numerical Example
Update Second Moments
rW = β2 × rW + (1 − β2) × (∇w L)2
= 0.999 × 0 − 0.001 × (−0.493)2
≈ 0.0001216
rB = β2 × rB + (1 − β2) × (∇bL)2
= 0.999 × 0 − 0.001 × (−0.482)2
≈ 0.0001168
Corrected First Moments
ŝW =
sW
1 − β1
1
≈
0.04437
1 − 0.901
≈ 0.4437
ŝB =
sB
1 − β1
1
≈
0.04382
1 − 0.901
≈ 0.4382

Numerical Example
Corrected Second Moments
r̂W =
rW
1 − β1
2
≈
0.0001216
1 − 0.9991
≈ 0.0608
r̂B =
rB
1 − β1
2
≈
0.0001168
1 − 0.9991
≈ 0.0584
Update Weights and Biases
w = w −
η
√
r̂W +
× ŝW = −2.0 −
0.10
√
0.0608 + 1e − 8
× 0.4437 ≈ −2.081
b = b −
η
√
r̂B +
× ŝB = −2.0 −
0.10
√
0.0584 + 1e − 8
× 0.4382 ≈ −2.079

References
1 Dive into Deep Learning by Aston Zhang, Zack C. Lipton, Mu Li, Alex J. Smola
https://d2l.ai/chapter_introduction/index.html
2 Deep Learning by Ian Goodfellow, YoshuaBengio, Aaron Courville
https://www.deeplearningbook.org/

DNN_M3_Optimization.pdf

More Related Content

Similar to DNN_M3_Optimization.pdf

Recently uploaded

DNN_M3_Optimization.pdf