Flashcards
PoS
Flashcards do conjunto (28)
- 1What is the aim of Term Normalisation? Give an example too!To make tokens with similar or identical semantics identical
USA U.S.A => usa - 2How does Term Normalisation work? (Process)- Remove punctuation
- Make lower case
- Stem to remove syntactic variants of same word - 3What is the aim of stemming? Give an example too!To eliminate different variants of the same word
e.g. connect, connects, connection, connections, etc - 4Stemming should give better ______ and ________ _________ _________- Recall
- Relevant documents retrieved - 5Name 3 Stemming Strategies- Affix Removal
- Table Lookup
- N-gram - 6What is Affix Removal?Suffix (after stem word) removal
- 7What is the most well know Affix algorithm and what is its main problem?Porter Stemmer
Often too aggressive - 8What is the best way to develop a stop word list?Look at term frequencies in corpus
- 9Give an example of a popular stop word listThe Brown Stopword List
- 10Issues with Stopwords- May not represent entire corpus vocab.
- Queries may be difficult to answer if stop words removed - 11What does PoS stand for?Parts of Speech
- 12What are the main uses of PoS tagging?- Limit range of meanings
- Helps stemming
- Limits word choice in speech recognition
- Helps identify terms for IR
- Language pattern recognition made easier - 13How do PoS tag words?- Look at each word in a sentence
- Assign a tag to each word - 14What are the PoS closed classes?- Prepositions (on, under, over, by)
- Determiners (a, an, the)
- Pronouns (he, she, it, I)
- Conjunctions (and, but, or)
- Particles (up, down, on, off)
- Numerals (1, 2, 3, first, second)
- Auxiliary Verbs (can, may, should, are) - 15What are the PoS open classes?- Nouns (people, places, things)
- Verbs (actions or processes)
- Adverbs (modifiers or verbs)
- Adjectives (describe properties/qualities) - 16What are the 2 most common tag sets for English and how many tags do they have?- Brown Corpus Tagset (87 tags)
- Penn Treebank (45 tags) - 17How does a computer/algorithm tag words (input and output)?Input:
- A tagset
- A string of words
- Dictionary of possible tags for each word
Output:
- A string of words each marked with most likely tag - 18What are the 3 different tagging algorithms and what are their time complexities?- Rule based: O(n)
- Stochastic: O(n)
- Transformation-based: O(n) - 19Rule-base tagging process1. Use dictionary to assign list of possible tags to each word
2. Apply rules to eliminate infeasible tag sequences - 20What is the aim of Stochastic tagging and what is the more realistic aim?- P(tag | word)
- P(word | tag) * P(tag | previous n tags) - 21What algorithm is most commonly used for Stochastic tagging?Viterbi Algorithms
- 22What is Transformation-Based Tagging also known as?Brill tagging
- 23Transformation-Based Tagging uses a ______ approach, this consists of...Hybrid
- Use of rules
- Supervised learning from pre-tagged training set - 24Transformation-Based Tagging: Tagging process1. Apply most likely tag
2. Apply narrower rules (e.g. change NN to VB when previous tag is TO)
3. Repeat rules until no more improvements/repetition insufficient to justify continuation - 25Transformation-Based Tagging: What are the 3 main rules to be applied to each word/tag?1. Examine all transformations
2. Select one giving most improved tagging
3. Re-tag using this rule - 26Transformation-Based Tagging: Steps for unknown/new words1. Assume all tags are equally probable
2. Assume distribution is similar to singleton words
3. Use best to use rules... - 27Transformation-Based Tagging: Best to use rules for unknown words- Words ending in -s most probably NNS
- Words ending in -ed most probably VBN
- Capitalised words most probably NNP - 28Average tagger accuracy91%-99%