Classification and vector spaces

Page 1

Sentiment analysis :

say you have 1000 product reviews :

go through the product review and classify how much % is good and how much is bad?

In the first week, you'll learn how to represent text as a vector and build a classifier that will classify whether a sample text is a positive sentiment or a negative one. You will use logistic regression for that

In the second week, you'll use the Naive Bayes classifier on the same problem. I

In the third week, you'll learn about vector space models. You'll learn how to represent text documents like tweets, articles, queries, or any object that contains of text as a vector. This is important in information retrieval, in indexing, in relevancy ranking, and also in information filtering.

Finally, in week 4, you will build your first simple machine translation system and you'll make use of locality sensitive hashing to improve the performance of nearest neighbor search.

Page 2

hSupervised ML : input variable X: Output variable : Y

Sentiment analysis of tweet :

Input is the tweet : "I am happy learning the NLP"

Output variable Y : Positive -1; Negative - 0;

so we can use the Logistic regression;

process :

how do you do that?

you have input : X

then you define prediction function : made of parameters : theta

prediction output : Y-cap;

Cost = Y predicted - Y actual;

then update your parameters :

so that the cost is minimized

this process of updating the theta parameters in the prediction function to minimize cost is called "Training"

Page 3

How to extract feature from a Tweet?

First you build a vocabulary of all unique words from all tweets : call it V

I am happy learning NLP

I hate History

V =[ I , am , happy, learning, NLP, hate, History]

then : for a particular tweet say,

Ex.,

I am happy learning NLP

feature represtnation goes like this :

for every word in vocabulary assign 1; for every word in vocabulary not in tweet assign 0

[1 1 1 1 1 0 0 0]

so logistic regression : will be n+1 model ???, where n is size of vocabulary

Page 4

create a corpus of 4 tweets : both positive and negative tweets

create a vocabulary out of 4 tweets

now take positive tweets:

and count the frequency of each word in vocabulary:

similarly

take negative tweets :

and count the frequency of each word in vocabulary:

see example below :

Page 5

Feature extraction:

you need to represent a tweet as vector of 3 :

for a particular tweet :

I am sad i am not learning NLO

[1 , count of positive frequency for each word, count of negative frequency for each word]

Answer : [1,8,11]

Page 6

Preprocessing:

one of the thing you need to do with text is to : preprocess:

consists of following steps:

removing words like and , is, or,

removing punctuations

lower case:

removing special characters

stemming :

ex., tuning GREAT AI Models!!!

stemming : tuning => tun which can be related to tune,tuning, tuned

GREAT => great

AI => ai

tun great ai models

Page 7

Algorithm to build X matrix:

build vocabulary:

create frequency dictionary mapping

take tweet:

I am happy learning NLP @deeplearning

|

preprocess

|

[happy,learn,nlp]

|

X vector => [1,sum of number of positive tweets, sum of number of negative tweets]

=> [1,4,2]

so for 'm' number of tweets => you will have matrix of size : m x 3 :

[1 X1(1) X1(2) , 1 X2(1) X2(2) , ..... 1 Xm(1) Xm(2)]

So Algorithm is :

#build frequencies dictionary

freqs = build_freqs (tweets, labels)

X = np,zeroes((m,3)) ; initialize matrix m

for i= 1 to m :

process every tweet to fill matrix X

X[i,:] = extract_features

Page 8

Next is to use extracted features X in Logistic Regression:

X => Function (of Theta) => Output Y -cap = > Cost = Output Y - Label Y => update theta to minimize cost

in logistics regression : Function F -> Sigmoid function:

F : h (xi) , theta) = 1/(1 + exp( -Theta transpose * x(i))

so you train theta , using training data to reduce cost function

Page 9

Initialize Theta parameters :

use logistic function :

find Y-cap:

Get gradient:

Update Theta

Get loss

Iterate

Page 10

testing logistic regression:

get X val and Y val and Theta : for training tweets

then

use sigmoid function on Xval

pred = h(Xval, Theta) > = 0.5 ;

so, pred = 1; if h < 0.5

pred =1 ; if h>=0.5

now do that for m tweets;

compare : prediction to Y val;

if both are equal : create a new vector => 1 ; you predicted right for the tweet

if prediction not equal to Y => vector value = 0 you predicted wrong for

so for m tweets => [1 0 1 1 0] ;

acciracy = sum of 1s / m = 3/4 = 60%

Page 11