Flashcards

Machine Learning

Oliver Boom English 101 cards

Cards in this deck (101)

1Supervised vs UnsupervisedLabelled vs no labels
Unsupervised is to recognise patterns within the data
2Examples Supervised vs Non-supervisedRegression/Classification
Clustering/Dimensionality reduction
3Linear RegressionFits a linear function called the hypothesis to the data
Minimise the cost function of this hypothesis
The weights of the hypothesis function called the parameters
The hypothesis has a bias term that is not associated with values
4Cost function for linear regressionMinimsing the least squares difference between the hypothesis and the actual value
Convex
Optimum can be found by gradient descent
5Gradient descentDifferentiate the cost function wrt parameters and head in direction of steepest descent according to some learning rate
6The normal equationsBecause the cost function is convex can differentiate the cost function wrt the parameters and set to zero. Then solving for the parameters will give the analytical solution to the minimum of the convex function
7Logistic regressionUsed for classification problems
Place the hypothesis function into a sigmoid function to give the probability of a class
To solve take the derivative of the cost function wrt the parameters and apply gradient descent
8Logistic Regression cost functionThe cross-entropy loss function
9Multiclassification problemsYou classify each class in a one vs the rest manner
Then logistic regression to get the probability of each class vs the rest and select the most probable.
10Decision boundariesWill be a polynomial that separates the different classification areas
11K-means - definition and algorithmA clustering approach
1 - randomly initialise centroids
2 - Assign cluster points nearest to centroids
3 - Move centroid location to the centre of the weights of these clusters
4 - Repeat until stopping condition (normally all points classified and centroids stable)
12Problem with k meansInitial centroids are random and clusters can be non-unique
13PCA Definition and algorithmDimensionality reduction
1 - Normalize data
3 - Calculate the covariance matrix
4 - Calculate the eigenvalues of the covariance matrix
5 - Keep the largest eigenvalues according to how many dimensions it is being reduced to. Project onto this dimensional space
14How to normalize data1 - Calculate the mean and std of the training vector
2 - Normalise input vectors by (X-mean)/std
15Hypothesis function and g(theta*x)g is the activation function
16Non-linear activation functions and their characteristicsReLU - Non-vanishing gradient
Sigmoid - 0-1 limit. Mainly used for final output layer. Gives a probability
Tanh - Mean of 0 compared. Stronger gradients that sigmoid
17Why use hidden layersBecause some complex logical operations cannot be represented purely from logistic regression
18Do Single Neuron Example Do parameters in network example Linear separabilityShould be able to draw a straight decision boundary
19Number of parameters in a networkN = (number of neurons in input layer + bias) * nodes in hidden layer + ....
for as many layers as needed
20Cost function in the binary classification caseUse the cross-entropy loss function
21Softmax functionTurns the output layer into a probability vector.
Selects the most likely probability
22BackpropagationDifferentiate the cost function with respect to the weights
This is done using chain rule
23Types of gradient descentBatch, mini-batch, stochastic
24Batch Mini-batch Stochastic Epoch1 - Does gradient descent calculation on the whole dataset. Computationally expensive
2 - Performs it in small batches
3 - Does it one training datapoint at a time, randomly. Better for escaping local minima
4 - A iteration over the whole training set
25Neural network algorithm1 - Randomly assign training weights
2 - Perform forwards pass to calculate the hypothesis function
3 - Calculate the loss function
4 - Calculate the gradients using backpropagation
5 - Perform gradient descent to re-assign weights
26OverfittingWhere the model performs well on the training set but fails to generalize well to new data
The hypothesis function has too many degrees of freedom considering the variability of the data
Also called variance
27UnderfittingThe data variations are too simple considering the actual variability of the data
Also called bias
28The capacity of a modelHow complex a relationship can be modelled
If a model has low capacity it will underfit due to not enough parameters
High capacity will overfit due to too many parameters
29GeneralizationThe ability of a model to perform well on unseen data
30What will a good ML algorithm do with respect to training error and test errorMake the training error small and the discrepancy between training error and test error also small.
Failure to do the first is underfitting, failure to do the second is overfitting
31RegularisationAny modification which improves generalization while not decreasing the training error
Want to decrease the variance while not significantly increasing the bias
32Regularisation examplesL1/L2
Dropout - drop hidden layers according to probability. Like sampling from an ensemble. Don't use at test time
Data Augmentation -
33Types of regularisationL1 - The absolute value of the weights
Also called ridge regression
L2 - The squared value of the weights
Also called lasso regression
Gamma - the weight decay term
The weights are summed and added to the cost function to incentivise having a small number of weights
34Effect of changing the regularisation parameterIncreasing gamma results in the weights tending to zero and therefore the function approximating a linear function.
35HyperparameterA value who's set before the training parameters begin
36Why the training/validation splitTraining - Used to train the nn
Validation - Used to test the trained set to determine optimal hyperparameters
The choose the optimal hyperparameters/architecture to take onto the test set
37Sampling from the training set in classification problems How to do this for big/small datasetsWant an even representation of each class in the training, validation and ideally test sets
Smaller - Random stratified sampling (first sort population into strata then randomly sample)
Large - Can just randomly shuffle, as the sample size will mean that the randomness will even out
38Bias vs variance Error vs Parameters plot
39Accuracy vs Regularization strength
40Large vs small neural networks Pros/ConsLarge - Computationally expensive. Overfit. Combat with regularisation
Small - Computationally cheap. Underfit. Decrease regularisation underparameters, or add more features
41K fold validation - whyOnly one validation set is not very robust
Especially for smaller datasets, the variability could result in inaccurate validation scores
42Issues with feed forwards neural networksThey lose their spatial organisation.
Unlike convolutional neural networks
43Do convolution example
44Idea behind cnnsParameterize the filter and optimise it's coefficients
W (filter) convolve x (input) = y (output)
45Bias and non-linearityAdd a bias term for each filter always unless specified
Apply an activation function to use non-linearity
46Key parametersStride and padding
47Equation to decide the convolution parametersCan also have a filter for each rgb channel. However the output is a singular channel
48If my input Image is 50x50 with 10 channels, and if I apply 25 filters each of size 3x3x10 with a stride of 1, no padding, what is the size and number of channels of the output image? Is the bias included48x48 with 25 channels
The bias has nothing to do with the size of the output image, it is to do with the weights
49How many weights are adjusted per cnn layerNumber of filters x (size of the filter + 1 for bias)
50Is the number of parameters related to the input image sizeNo it is not
The number of parameters is the same as the degrees of freedom
51How many weight terms for this network27301
52Types of layers in a convolutional networkConvolutional layers
Fully connected layers
Pooling layer
53How are the parameters optimizedIn the same way as feed forwards networks. Backpropagate to find gradients and perform gradient descent.
54Transfer learningTrain on existing network and then retrain final layer of network on images of interest
55Random variableA variable that is associated with a non-deterministic process. Ie it's picked according to a distribution
56Probability density function
57Mean of pdf Std of pdf
58Covariance of two distributions rhoRho is the correlation coefficient
if rho = 0 no correlation
if rho = 1 or -1 perfect linear correlation
59SigmaThe variance-covariance matrix
60Discrete random variablesTake finite values associated to certain probabilities
61Expectation of DRVE(X) = p * x
E(X^2) = p * x^2
Var = E[X^2] - mu^2
62Bernoulli Variablemean - p
Variance - p*q
63IIDIndependently and identically distributed
If each RV has the same distribution and are all mutually independent
64Maximum likelihoodMeans choosing the parameters such that m samples maximises their likelihood
65Likelihood of an image Maximum likelihood ML of a gaussianpdf(image input)
So maximum likelihood is just choosing weights that maximise the sum of all the likelihoods across the data set
Maximising likelihood leads to minimising cross-entropy loss for classification or L2 norm for regression
66—Do exercise
67Lognormal distributionX is lognormally distributed if log(X) is normally distributed
68Monte Carlo and RVsYou can use Monte Carlo simulation to pick RVs according to some distribution
69KL divergenceA method to compare two RVs - Seeing the similarity between two distributions
It's value is 0 if the RVs are identical
It is assymetrical - think about the normal gaussian videos as they came closer togehter
70KL vs Maximum likelihoodFinding the parameters which maximise likelihood is the equivalent to finding the parameters which minimize KL divergence
71PCA vs AutoencoderPCA can only represent linear mapping
Autoencoder can do non-linear
72Fully connected Layer in CNNSame as in a forward neural network. All the neurons in the input layer connect to all the neurons in the output layers
73Pooling LayersTakes the maximum values in a given filter size. Makes the feature dimensions smaller while still recognising key features. It also reduces the number of parameters and computations
74Latent spaceWant to represent an image by a set of vectors which are much smaller than the dimension of the original image
Like PCA but also works for non-linear manifolds
75Autoencoders Process
76Difference between encoder and decoder convolutionEncoder reduced dimensionality
Decoder increases dimensionality
77Does the convolution operation include a bias termYes it does
78How to get the decoded x values from y and W
79Example Calculate the direct convolutionNext to the transpose
80How to calculate the transposed convolution
81What do VAE generateNew images that belong to the same general pdf as those of the training set
82Encoding StepThe recognition network
83Decoding StepThe generation network
84How do you calculate mean and std of xUsing a neural network where the parameters maximise the likelihood
85ELBOThe evidence lower bound. The lower bound of the log likelihood for a single image
Made up of the reconstruction likelihood and the KL divergence
86Reconstruction likelihoodQuantifying the quality of the trip from training image to latent vector pdf
87KL Divergence for ELBOAims to spread the range of latent vectors associated to each image
88How can you calculate the expectation of a pdf in variational autoencodingUse monte carlo to sample enough times to get the expectation
89VAE Flowchat
90MMD for GANSMaximum mean discrepancy compares two samples by comparing the mean in the feature space
91GANs discriminator cost functionBinary cross entropy equivilent
92Why might generators not be greatMiss modes of the distribution - called mode collapse
93Which is the best method of GANUnrolled is better than GAN
94Smooth class GANsCan incorporate class labels into GANs and have smooth interpolation to have smooth inter-class results
95Unpaired domain transferUsed to map one domain onto another
96VAE vs GANs
97Are kernels linear or non-linearThey are non-linear because a transformation is applied
98Number of output neurons in a convolutional filterIs the same as the output shape
99Parameters/weights vs Output Neurons for a convolutional neural networkParameters/weights is the number of filters * (size of filter + 1) it has nothing to do with the size of the input image. Because it is the convolutional filter that is being parameterized.
The number of neurons which is a much larger number and is proportional to the size of the input image
100Normal distribution
101Log likelihood for a variaitional autoencoderThe measure of the effectiveness with which the decoder has learned to reconstruct an input image xx given its latent representation zz.