week 9

Page 1

what is anamoly detection?

take an example of aircraft engine;

you have collected test data normal engines for heat and vibration levels;

now for a new engine : (test) : how do you say that the engine will function normally or does it have an anomaly?

how do you do that?

Probablity of test data <= threshold vale => test is an anomaly;

Page 2

anomaly detection steps:

choose X features (x1,x2,....xm) that you think may be indicative of anamoly

for each feature : find mu1....mu.m ; sigma1....sigmam;

using gaussian distribution :

find p(x1), p(x2)...p(xm) and find the product;

if product <= epsilon ; anamoly ;

Page 3

practical steps : anomaly detection:

fit model with training set data

on Cross validation set :

predict y=1 - normal

; y= 0; anomaly

now prepare evaluation matrix:

true+ , false + , false -,true -

Fscore

precision / recall

Page 4

recommender systems:

ex., books - amazon

movies - NetFlix

problem formulations:

Nu - no of users

Nm - No of movies

ri = 1 , if user viewed movies

y = rating given to user j to movie i

Page 5

content based recommendations: similar to linear regression

recommendation : Theta-Transpose * X ; where X is a feature vectorbased on content;

X1 => romance

X2=> action etc;

Page 6

collaborative filtering:

how to get feature vectors for movie ratings?

given user preferences :

X = {1 romance action comedy}

remember user preference = {Theta-Transpose}{X} ;

this is given theta find X ;

previously we determined , given X ,find Theta ;

actually using minimization : we can go : given Theta => X => Theta => X ...and so on

to converge to a solution;

Page 7

low rank vectorization:

take Y = [matrix of users and preference] => Yi = Theta-transpose * Xi

Y matrix => X*Theta-Transpose ;

if you want to find related movies to movie Xi

find distance ||Xi - Xj||;

smaller the distance , similar the movies;

Page 8

mean normalization:

when an user has not provide any ratings to any movie : use mean normalization;

ie. Y = [1 0 0 ?

2 1 1 ?

3 4 5 ?

6 7 8 ? ]

by default, algorithm would assign rating of 0;

find average of given ratings for a movie: (row mean) Mui;

subtract from Y; get new Y;

find Theta-Transpose * Xi + Mui => to get ratings for unrated user;