RNN Encoder-Decoder

RNN Encoder-Decoder

1. Encode a variable-length sequence into a fixed-length vector representation

   • The encoder is an RNN that reads each symbol of an input sequence x sequentially. As it reads each symbol, the hidden state of the RNN changes according to : h(t)=f(h(t-1),x_t). After reading the end of the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary c of the whole input sequen

2. Decode a given fixed-length vector representation back into a variable-length sequence

   • The decoder is another RNN which is trained to generate the output sequence by predicting the next symbol y_t given the hidden state h(t) = f(h(t-1), y_(t-1), c) is conditionned on previous h, y and c:the summary of the input.
     • A Hidden unit that adaptively REMEMBERS (update gate) and FORGET (reset gate)
       •  As each hidden unit has separate reset and update gates, each hidden unit will learn to capture dependencies over different time scales. Those units that learn to capture short-term dependencies will tend to have reset gates that are frequently active, but those that capture longer-term dependencies will have update gates that are mostly active.
       • UPDATE gate(z): controls how much information from the previous hidden state will carry over to the current hidden stat. It selects whether the hidden state "h" is to be updated with a new hidden state"h~". This is similar to the memory cell in the LSTM and helps the RNN to remember long-term information.
       • RESET gate: It effectively allows the hidden state to drop any information that is found to be irrelevant later in the future, thus, allowing a more compact representation. It simply shut-down the PREVIOUS hidden state so that it's ignored.

3. From a probabilistic perspective, this model is a general method to learn the conditional distribution over a variable-length sequence conditioned on yet another variable-length sequence: p(y1, . . . , yT' | x1, . . . , xT) T!=T'

4. Machine Translation

Images and files in the map

• Rnn Encoder Decoder Article (application/pdf)