Lecture1 notes

Page 1

If we want to penalise the formula why to use the absolute value rather than the square?

L1 constraint is another name for the Lasso, while l2 might describe the ridge regression

Ridge is like the real ridge of the mountain, while in the same line you will always get the same values

There're some alternatives for finding the budget with cross-validation is by using Baysian- Lablasian

Are Baysian methods always computationally demanding?

Relaxed Lasso: think about the bias-variance trade-off

Trying to solve the Lasso problem will make us run into the problem of calculating the gradient for zero which is not defined so instead in the lecture the prof. use math formulas for the subgradient

Page 2

Excerise

- Generate random weights with many of them set to zero

- Generate fake data

- Write out coordinate descent algorithm for Lasso

- Show lambda = 0 and compare with ground truth

- Solve also with other lambdas and compare

- Play with numbers of features versus number of data points (the 'width' of the data) as well as the strength of the non-zero weights and the noise term