Representation Learning

Representation Learning

1. Representation Learning: A Review and New Perspectives (Yoshua)

   • why representation learning?
     • make learning algorithms less dependent on feature engineering
   • what makes a representation good?
     • data priors for representation learning in IA
       • smootheness
       • multiple explanatory factors
       • hierarchical organization of explanatory factors
       • semi-supervised learning
       • shared factors across tasks
       • manifolds
       • natural clustering
       • temporal and spatial coherence
       • sparsity
       • simplicity of factor dependencies
     • deep architectures
       • promote the reuse of features
       • leed to progressively more abstract features
       • how to build deep architectures?
         • A Fast Learning Algorithm for Deep Belief Nets (Hinton)
         • Greedy Layer-Wise Training of Deep Networks (Yoshua)
         • Efficient Learning of Sparse Representations with an Energy-Based Model (LeCunn)
         • Learning Deep Architectures for AI (Yoshua)
         • learn a hierarchy of features one level at a time
     • distange factors of variation
   • applications
     • speech recognition and signal processing
     • object recognition
     • natural language processing
     • multi-task and transfer learning
     • image reconstruction/recovery?

2. generative models

   • model for randomly generating observable-data values, typically given some hidden parameters
   • probabilistic models
     • directed graphical models
       • p(x,h) = p(x|h)p(h) : parametrizes the likelihood and the prior separately
         • Gaussian Mixture Models (GMM)
         • Naive Bayes
         • PCA (probabilistic interpretation)
           • p(h) = N(h;0;\sigma^2 I )
           • p(x|h) = N(x; Wh+\mu_x, \sigma^2 I )
           • latent representation h
           • linear mapping W to the data x
         • Sparse Coding (probabilistic interpretation)
           • latent representation h
           • linear mapping W to the data x
           • includes a penalty to ensure a sparse activation of h is used to encode x
           • dictionary learning: find the most suited linear mapping W
             • K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation (Aharon)
           • application on image reconstruction/recovery
         • Spike-and-Slab Sparse Coding
     • undirected graphical model
       • Markov Random FIelds (MRFs) parametrize p(x,h) through a product of unnormalized non-negative clique potentials
       • Restricted Boltzman Machine (RBMs)
         • p(x,h) defined with clique potentials constrained to be positive
           • energy based models
             • A Tutorial on Energy-Based Learning (LeCunn)
         • positive phase computes p(h|x)
         • negative pahse computes p(x|h)
         • hiden layer and visible layer are binary
         • may be stacked to compose a deep architecture
         • generalization of the RBM to real-valued data are not ultimately solved yet
           • Gaussian RBM (GRBM)
           • mcRBM
           • mPot
           • ssRBM
   • models based on parametric map from input to representation
     • Auto-Encoders
       • defines an encoder function h = f(x)
       • defines a decoder function x = g(h)
         • g(h) está explicitamente associado à reconstrução do modelo (Yoshua)
       • regularized Auto-Encoders
         • prevents the auto-encoder to learn the identity function h = x
         • Sparse Auto-Encoder (SAE)
           • introduces sparsity regularization in the hidden unit
         • Denoising Auto-Encoder (DAE)
           • objective: recontruct the original data from a corrupted input
           • learns more robust features than SAE
           • DAE features perfoms similarly or better than RBM features (Yoshua)
         • Contarctive Auto-Encoders
       • relation with RBMs
         • f(x) plays the role of P(h|x)
         • g(h) plays the role of P(x|h)

Bilder und Dateien der Mindmap