Machine learning paper explained

Página 1

LOOK AHEAD OPTIMIZER : K STEPS FORWARD 1 STEP BACKWARD

https://www.youtube.com/watch?v=ypqf7UUird4

Stochastic weight optimizing : taking weight average when in weight space

Regularized non linear acceleration(RNA)

tries to find lace where gradient is zero

solve linear system of eqn based on last k iterates

O(K^2d+K^3) : K times more memory and compute. K no of iterate

Noisy Quadratic analysis :

loss is taken as quadratic version of the parameter.

step is then equal to actual gradient + some noise ci ~N(0,var)

variance of lookahead is lower than the stochastic gradient descent. in noisy quadratic model

They introduced two new parameter, how many steps you want to look ahead(K) and then step you want to take based on that(alpha).

Página 2

EigenDamage : Structured pruning in kronecker - factored eigenbase

(https://www.youtube.com/watch?v=OymzTQYvY_I&t=2s)

Pruning of the network

Help in generalization like dropout

biological and practical perspective

way of doing the pruning in the network

structure pruning: irregular sparsity in the network.

weight pruning: introduce sparsity of the weight matrix.

Hessian based pruning network :

It requires you to mention the pruning ratio and it automatically identify best network structure.

we do an approximation of the function around optimal theta and remove those weights which will lead to least increase in overall training error.

There are two type of hessian based approximation algorithms.

Optimal brain damage

it uses a diagonal approximation of hessian to compute the importance of each weight.

It does an inaccurate approximation of the correlation matrix.

It ignores the corrrelation between features.

Optimal brain surgeon

It uses full hessian for the calculation of the importance of weights.

But it's challenging to calculate a full hessian for any neural network.

It is guaranteed to be better than OBD, if weight is pruned one by one (i.e. Hessian is calculated in multiple iterations)

But we want to prune multiple weights at a time.

If we Prune multiple Weights at a time will OBS is better than OBD ?? -- NO

both OBS and OBD faces there own issues,

Soln :

Pruning in a new coordinate system (i.e. new basis ) in which each weights are closer to independence.

Hessian is hard to calculate.

Idea :

KFAC approximation => approximation fisher information matrix => in turn approximate Hessian

KFAC approximation:

FIM of a matrix is decomposed as Kronecker product of two smaller matrices under independence assumption.

generally it is kronecker product of gradient before and after activation

We do en decomposition of the approximated FIM and do the pruning in the new eigen space.

Taylor approximation is only valid for the optimal v*, so we have to prune it and then fine tune.

Página 3

Neural Turing Machine

(https://www.youtube.com/watch?v=XwqWTX1pW4s)

IT's a simple model of a computer

Memory tape

Read and write heads

Neural turing machine has a memory and uses a differentiable network structure as a controller

XOR and other gates based function are not differentiable . It uses some sort of approximated smooth function for such gates.

Can learn general tasks.

Copy, repeat, recognize simple formal language.

Generalize

Do well at language model

DO well at Babi

Can be used to recognize balanced paranthesis problem.

Problems :

Architecture dependent

Large number of parameters

Doesn't benefit much from GPU acceleration : harder to parallelize each input is dependent on previous generated result.

Numerically unstable : making mistake in algorithm is very risky

Using memory is hard : learn not to accidentally forget

Needs smart optimization.

Gradient clipping

limit training speed of parameter

Helpful in learning long range dependencies

Use curriculum learning

Dynamic Neural computers

o index shift based addressing

Can allocate and de-lallocate memory

Can be used in inference tasks