COMP2611 - Natural LanguageProccessing

COMP2611 - Natural LanguageProccessing

1. Formal Grammers

   • Context-Free Grammer
   • Tokenization
     • Breaking down a stream of characters into items of interest.
     • Regular Expressions
   • A way of specifying what things are.
   • Morphology
     • Inflexional Affixes
       • Morphemes that are added to a word root to express general logical and grammatical aspects of meaning
     • Derivational Affixes
       • Combine affixes with a word root to produce a new word of a different meaning.
     • Compounding
       • Combining multiple word stems together
     • Cliticization
       • A clitic is a morpheme that acts like a word in reduced from
         • I've == I have
     • Stemming
       • Stripping off affixes and leaving stem
     • Lemmatization
       • Contrasted with stemming, want the underlying root word
     • Affix Idiosyncrasy
       • The possibility of adding affix to a word does not allows follow general rules

2. Statistical Approach

   • Consider linguistic properties that are averaged over large quantities of linguistic data
   • Zipf's Law
     • Word Frequency X Rank = Constant For Corpus
   • N-Grams
     • Bi- / Tri- / Quad-
     • Compute the probability that a given word follows from a given N-1 sequence
   • Bag Of Words (multiset)
     • Popular in spam filters
     • An un-ordered collection of words so meaning of corpus is often lost
     • Naive Bayes Classifier
   • Information Retrieval
     • Precision
       • The proportion of retrieved documents that are relevant
     • Recall
       • The proportion of all relevant documents that are retrieved
     • Analysis of search engines
   • Factoid Question Answering

Images and files in the map