gradient descent

Page 1

:why is gradient descent needed?

to find the minimum of a function...as in the cost function J(theta0, theta1)

how does it work?

start with theta0 and theta1

compute J(theta0, theta1)

modify theta0 , theta1

to minimize J(theta0,theta1)

Page 2

gradient descent function : for theta-0, theta-1

theta-j := theta-j -alpha * d/dthetaj * J( theta-0,theta1)

alpha - learning rate

theta-0, theta-1 have simultaneous update

ex.,

tempo-0 := theta-0 -alpha * d/dtheta-0 * J( theta-0,theta1)

tempo-1 := theta-1 -alpha * d/dtheta-1 * J( theta-0,theta1)

:= signifies assignment operator; like set parameter in programming;

different from = ; a:=a+1 is assignment ;

where as = is an assertive statement;

theta-0: = tempo-0

theta-1: = tempo-1

Page 3

example : cost function with one variable : J(theta1)

theta1 := theta1 - alpha * d/dtheta1 * J(theta1)

d/dtheta represents the slope of the tangent line at J(theta1) , theta1

alpha is the learning parameter ; it is positive;

if alpha is too small, the descent will be slow

alpha too high, fail to converge or even diverge

choose an alpha and no need to change it over the iterations;

Page 4

first machine learning algorithm: linear regression:

h(x) = theta-0 + theta-1 * x;

J(theta) = 1/2m * (sum of squared errors)

mean sum of squared errors = 1/2m * (sum of squared errors)

sum of squared errors = sigma (1->m) error^2

error = (theta0 + theta1*xi) - yi

repeat until convergence :

theta0 := theta0 - alpha * d/dtheta0 * J(theta0, theta1)

theta1 := theta1 - alpha * d/dtheta1 * J(theta0, theta1)

Gradient descent for linear regression : above statement looks as :