Deep Learning Mechanisms

Deep Learning Mechanisms

1. Attention Mechanism

   • In Encoder/Decoder Networks, it allows the Decoder to "Pay Attention" to different parts of the input
     • RNN
       • Pay attention to different part of Input at every step of output sequence generation
         • Each decoder output depends on a weighted combination of ALL THE INPUT STATES, not just the last state
           • Each input state is assigned an attention weight for each output step (in a sequence of outputs for an RNN)

2. Backpropagation

3. Sparsely Attentive Backtracking (SAB)