Natural Language Processing

Seite 1

Commands to remember :

Use this command in anaconda prompt to open your notebook in a different location: jupyter notebook --notebook-dir=\\timing6\aashna\scratch\python

Decision Tree, Bayesian Net, Random Forest, Honey Bee Algorithm, Swarm Intelligence, Simulated Annealing, Genetic Algorithm

Text Mining /Text Analytics / NLP – Application: Sentiment Analysis, Chatbot, Speech recognition, Machine Translation ( Google translate), Extract information from a doc.

Types of NLP – Natural Language Understanding and Natural Language Generation

PROCESS involved in Understanding

Tokenization -> Stemming(Normalize words to base form, could be inaccurate) -> Lemmatization(need a detailed dictionary to link back the word, output is a proper word) ->POS (Parts of Speech) tags-> Named Entity Recognition (NER- to detect things like movie, organization, location, person, 3stpes- Noun Phrase Identification, Phrase classification, Entity disambiguation)->Chunking(Picking up individual pieces of Information and grouping them into bigger pieces )

NLU Ambiguity: Lexical Ambiguity, Syntactic Ambiguity, Referential Ambiguity

Python NLTK

Tokenization: First Step

Break a complex sentence into words

Understand the importance of each word with respect to the sentence

Produce Structural description on input sentence.

Bigrams - Token of 2 consecutive written words

Trigrams - Token of 3 consecutive written words

Ngrams - Token of N consecutive written words

Normalize: Normalise words into its base form

Stemming: Remove the prefix and suffix to make a shorted word

Lemmatization: Logical analysis to get out a meaningful word.

Lemmatizer keeps the words as it is unless you assign POS(Parts of speech) tags to it.

NLTK has its own set of STOP WORDS (connectors with no actual meaningfulness), which can be imported from the corpus. So remove the STOPWORDS before processing the text and POS.

Named Entity Recognition

Noun Phrase identification - extract noun phrase POS tagging and dependency parsing

Phrase classification - extracted noun phrases are classified into location, person

Entity disambiguation - Validation layer on top of the result.

(Process: Tokenize, LEmmitise, POS, NER)

Syntax tree: Defines what POS comes after what

Chunking

Pass the un phrase string to chunk parser using the RegexParser to create chunks. Use ghost script to form the syntax tree(Otherwise it gives an error).

Check out naives byes algo

LECTURE

TOPIC MODELLING - could be used to tag parts of data to something you are looking for

TEXT GENERATION - could be used for test generation

steps before doing this - a lot of cleaning up of data

Skills -

Programming - Data NLP - pandas, sklearn , re, nltk, TextBlob, gensim(for topic modeling)

Maths - clean EDA(exploratory data analysis) NLP - corpus, document-term matrix, word counts, sentiment analysis, topic modeling, text generation

Communication- Design domain - scope, visualize, extract insights

STEPS

Start with a question > Get and Clean data > Perform EDA(in a scatter plot, look for correlation and outliers), Apply NLP techniques, Share insights

Data Gathering - Requests, Beautiful Soup, Pickle (get data from http)

Data Cleaning -

1. Corpus - is a collection of texts. Pandas, DataFrame - a pandas object/table

2. Document term matrix -

clean text(remove punctuation, use lowercase- re (regular expressions in python, do something for matched data))

tokenize text - tokenize, remove stop words(which can be modified),

put it into matrix form- using scikit learn, count vectorizer with sklearn + you can remove stop words along the way

to read an HTML document - BeautifulSoup

Common data cleaning steps on all text: Make text all lower case, Remove punctuation, Remove numerical values, Remove common non-sensical text, Tokenize text, Remove stop words

Keep the data - key, and value, in pandas DataFrame.

Can make a corpus before creating Document term matrix >> text cleaning (import re and string) >> corpus. pickle >> use countVectorizer using sci-kit learn and pickle it

EDA - Summarize main characteristics of data, often using visual methods. (input- corpus/DTM , output - trend with some correlation and outliers). Find - Top words, vocabulary, define after you see frequency...plot using scatter plot, matplotlib, wordcloud,

Insights - find most used words, modify stop words,

You can use multiple things to get the insights - like the total number of uniques words, number of correspondence

Sentiment Analysis

Since the order of words matter, we use Corpus.

TextBlob is a library built on top of nltk. It gives result in terms of Polarity and Subjectivity. This is a rules-based approach, you can also use a statistical method like Naive Bayes

Topic Modelling

A document-term matrix. Order of words doesn't matter so we can use a bag of words format.

Library- gensim. We can use LDA (Latent Dirichlet Allocation) and POS tagging.

For LDA- Input is DTM , number of topics, number of iterations

other methods - LSI(Latent Semantic Indexing) , NMF(Non-negative Matrix Factorization)

Try spacy

Text Generation

Input - is a corpus, we want to preserve the order and the punctuation.

Python - Markov Chains - represent how systems change over time, they are memoryless,

keys- current word , values - probable next words

Better way - Long short term memory(LSTM) ,