UFLDL Tutorial

UFLDL Tutorial

1. sparse autoencoder

2. preprocessing: pca and whitening

   • whitening
     • is a preprocessing
       • why?
         • images are highly redudant since adjacent pixels are very correlated
         • to make image input less redudant
     • will provide to the learning algorithm
       • (image with) features less correlated with each other
       • features having the same variance
     • PCA Whitening
       • components are divided by the eigenvalues (variance) sqrt
       • is possible to also reduce the data dimension
     • ZCA Whitening
       • X_{ZCAwhite} = U*X_{PCAwhite}
         • U is the matrix of the data eigeinvalues (PCA basis change)
         • X_{ZCAwhite} is the closest as possible to the original input X
       • is usually not used with dimension reduction
       • this is also a rough model of how the biological eye (retina) process images

3. softmax regression

   • generalizes logistic regression to classification problems (for n classes)
     • logistic regression
       • labeled examples (supervised problem)
       • binary classification setting
         • h_\theta(x) = sigmoid(-theta^{T}.x)
           • output of a single neuron
     • calculates p(y^{(i)}=k|x^{(i);\theta} for each input x^{(i)}
       • a postriori probability
         • is clearly a generative model
       • \theta is an association for distincts '\theta_i's for each class

4. self-taught learning

   • we can get our algorithm to learn from unlabeled data
   • a single unlabeled example is less informative than a single labeled example
     • with a large ammount of unlabeled data - and our algorithm can explot this unlabeled data effectively - the may do better than massive hand engineering features and massive-labeling data
   • large amout of unlabeled data may be used to learn features and a small amount of labeledt data may be used to train the solver for the classification problem

5. deep networks

   • each hidden layer computes a non-linear transformation of the previous layer
   • deep net can have grater rerpesentation power
   • cortical computations also have multiple layers of processing
   • why deep learninig did't worked well before?
     • standard approach
       • iniatialize the weights radomly
       • training using labeled data (x^(i), y^(i)) to minimize the error measure
         • gradient descend to drive down the error mountain
     • o usually had too few labeled data to fit the parameters of a complex approach
       • training using unsuficient data would lead to overfiting
     • in a deep architecture, a gradient descend algorithm tends to be stucked in a local otima
     • in the standard architecture, using backpropagation and gradient descent, the gradient that propagates backwards rapidly dimiinish in magnitude as depth of the network increasses
     • how can we train a deep network?
       • greedy layer-wise training: one layer at the time
         • initially unsupervised and then, supervised (fine-tuning)