Variational Auto-Encoder (VAE)

Variational Auto-Encoder (VAE)

1. What is it?

   • The basic idea behind autoencoders is to encode information (as in compress, not encrypt) automatically, hence the name.
     • In neural net language, a variational autoencoder consists of an encoder, a decoder, and a loss function.
       • Encoder
         • The encoder is a neural network. Its input is a datapoint x, its output is a hidden representation z, and it has weights and biases θ. We note that the lower-dimensional space is stochastic: the encoder outputs parameters to qθ(z|x), which is a Gaussian probability density. We can sample from this distribution to get noisy values of the representations z.
       • Decoder
         • The decoder is another neural net. Its input is the representation z, it outputs the parameters to the probability distribution of the data, and has weights and biases ϕ.
           • The decoder is denoted by pϕ(x|z).
       • Loss function
         • The loss function of the variational autoencoder is the negative log-likelihood with a regularizer. Because there are no global representations that are shared by all datapoints, we can decompose the loss function into only terms that depend on a single datapoint li ​​.
           • The total loss is then ∑^N ​i=1 ​​li ​​ for N total datapoints.

2. Architecture

   • Some variations...
     • It is somewhat similar to FFNNs as it is more like a different use of FFNNs than a fundamentally different architecture.
   • Shares the same architecture as AE's, however, an approximated probability distribution of the input samples is used, based on Bayesian mathematics regarding probabilistic inference and independence.

3. A little history...

   • Variational autoencoders (VAEs) were defined in 2013 by Kingma et al. and Rezende et al.
     • They married ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning.
     • Their algorithm introduced a recognition model to represent approximate posterior distributions, and that acts as a stochastic encoder of the data.
     • With this...
       • They developed stochastic back-propagation -- rules for back-propagation through stochastic variables -- and used this to develop an algorithm that allows for joint optimisation of the parameters of both the generative and recognition model.

4. What is it used for?

   • Variational autoencoders are cool. They let us design complex generative models of data, and fit them to large datasets.
     • These models also yield state-of-the-art machine learning results in image generation and reinforcement learning.
     • For instance, they can generate images of fictional celebrity faces and high-resolution digital artwork

5. Learning rules

   • The basic idea is: Take influence into account.
     • Things are not always related, so if that is the case, then the error propagation should consider that.
     • We train the variational autoencoder using gradient descent to optimize the loss with respect to the parameters of the encoder and decoder θ and ϕ. For stochastic gradient descent with step size ρ, the encoder parameters are updated using θ←θ−ρ ​∂θ/∂l ​​ and the decoder is updated similarly.

6. Some real life experiments

   • Kingma and Rezende et al. demonstrated on several real-world data sets that the model generates realistic samples, provides accurate imputations of missing data and is a useful tool for high-dimensional data visualisation.
     • They trained generative models of images from the MNIST and Frey Face MATLAB datasets and compared learning algorithms in terms of the variational lower bound, and the estimated marginal likelihood.
       • Computation took around 20-40 minutes per million training samples with a Intel Xeon CPU running at an effective 40 GFLOPS.