Deep Learning

Deep Learning

1. Applied Mathematics

   • Linear Algebra
     • Scalars
     • Vectors
     • Matrices
     • Tensors
     • Multiplying Matrices and Vectors
     • Identity and Inverse Matrices
     • Linear Dependence and Span
     • Norms
     • Special Matrices and Vectors
     • Eigendecomposition
     • Singular Value Decomposition
     • The Moore-Penrose Pseudo-inverse
     • The Trace Operator, and the Determinant
   • Probability and Information Theory
     • Random Variables
     • Probability Distributions
     • Marginal Probability
     • Conditional Probability
     • Chain Rule of Conditional Probabilities
     • Independence and Conditional Independence
     • Expectation
     • Variance and Covariance
     • Common Probability Distributions
     • Useful properties of common functions
     • Bayes’ Rule
     • Continuous Variables
     • Information Theory
     • Probabilistic Models
   • Numerical Computation for Machine Learning
     • Overflow and Underflow
     • Conditioning
     • Gradient-Based Optimization
     • Constrained Optimization

2. Machine Learning Basics

   • Learning Algorithms
   • Capacity
   • Overfitting and Underfitting
   • Hyperparameters
   • Validation Sets
   • Estimators
   • `Bias and Variance
   • Maximum Likelihood
   • Bayesian Statistics
   • Supervised Learning Algorithms
   • Unsupervised Learning Algorithms
   • Stochastic Gradient Descent
   • Building a Machine Learning Algorithm

3. Deep Learning Modern Practices

   • Deep Feedforward Networks
     • Gradient-Based Learning
     • Hidden Units
     • Architecture Design
     • Back-Propagation and other Differential Algorithms
   • Regularization for Deep Learning
     • Parameter norm penalties
     • Norm penalties as constrained optimization
     • Regularization and Under Constrained problems
     • Dataset Augmentation
     • Noise Robustness
     • Semi-Supervised Learning
     • Multitask Learning
     • Early Stopping
     • Parameter Tying and Parameter Sharing
     • Sparse Representations
     • Bagging and other Ensemble Methods
     • Dropout
     • Adversarial Training
     • Tangent Distance
     • Tangent Prop
     • Manifold Tangent Classifier
   • Optimization for Training Deep Models
     • Learning vs Pure Optimization
     • Challenges in Neural Network Optimization
     • Basic Algorithms
     • Parameter Initialization Strategies
     • Algorithms with Adaptive Learning Rates
     • Approximate Second-Order Methods
     • Optimization Strategies and Meta-Algorithms
   • Convolutional Neural Networks (CNNs)
     • The convolution operation
     • Motivation
     • Pooling
     • Convolution and Pooling
     • Variants of the Basic Convolution Function
     • Variants of the Basic Convolution Function
     • Structured Outputs
     • Data Types
     • Efficient Convolution Algorithms
     • Random or Unsupervised Features
   • Recurrent Neural Networks (RNNs)
     • Unfolding Computational Graphs
     • Bidirectional RNNs
     • Encoder-Decoder Sequence-to-Sequence Architectures
     • Deep Recurrent Networks
     • Recursive Neural Networks
     • The challenge of long-term dependencies
     • Echo State Networks
     • Leaky Units and other strategies for multiple time scales
     • The Long Short-Term Memory (LSTM) and other Gated RNNs
     • Optimization for Long-Term Dependencies
     • Explicit Memory
   • Deep Learning Methodology
     • Performance Metrics
     • Default baseline models
     • Determining whether to Gather More Data
     • Selecting Hyperparameters
     • Debugging Strategies
   • Deep Learning Applications
     • Large-Scale Deep learning
     • Computer Vision
     • Speech Recognition
     • Natural language processing

4. Deep Learning Research Topics

   • Linear Factor Models
     • Probabilistic PCA and Factor Analysis
     • Independent Component Analysis (ICA)
     • Slow Feature Analysis
     • Sparse Coding
     • Manifold Interpretation of PCA
   • AutoEncoders
     • Undercomplete Autoencoders
     • Regularized Autoencoders
     • Representational Power
     • Layer size and Depth
     • Stochastic Encoders and Decoders
     • Denoising Autoencoders
     • Learning Manifolds with autoencoders
     • Contractive Autoencoders
     • Predictive Sparse decomposition
     • Applications of Autoencoders
   • Representation Learning
     • Greedy Layer-Wise Unsupervised PreTraining
     • Transfer Learning and Domain Adaptation
     • Semi-Supervised Disentangling of Causal Factors
     • Distributed Representation
     • Exponential Gains from Depth
     • Clues to discover underlying causes
   • Structured Probabilistic Models For Deep Learning
     • Challenges of Unstructured Modeling
     • Using Graphs to Describe Model Structure
     • Sampling from Graphical Models
     • Advantages of Structured Modeling
     • Learning about Dependencies
     • Inference and Approximate Inference
     • The Deep Learning Approach to Structured Probabilistic Models
   • Monte Carlo Methods
     • Sampling and Monte Carlo Methods
     • Importance Sampling
     • Markov Chain Monte Carlo Methods
     • Gibbs Sampling
     • The Challenge of Mixing between Separated Modes
   • The Partition Function
     • The Log-Likelihood Gradient
     • stochastic Maximum Likelihood and Contrastive Divergence
     • Psuedolikelihood
     • Score Matching and Ratio Matching
     • Denoising Score matching
     • Noise-Contrastive Estimation
     • Estimating the Partition Function
   • Approximate Inference
     • Inference as optimization
     • Expectation Maximization
     • Map Inference and Sparse Coding
     • Variational Inference and Learning
     • Learned approximate inference
   • Deep Generative Models
     • Boltzmann Machines
     • Restricted Boltzmann Machines
     • Deep Belief Networks
     • Deep Boltzmann Machines
     • Boltzmann Machines for Real-Valued Data
     • Convolutional Boltzmann Machines
     • Boltzmann Machines for Structured or Sequential Outputs
     • Other Boltzmann Machines
     • Back-Propagation through Random Operations
     • Directed Generative Nets
     • Drawing samples from Autoencoders
     • Generative Stochastic Networks
     • Other Generation Schemes
     • Evaluating Generative Models