LEXICAL Features

LEXICAL Features

1. Vocabulary Richness

   • type-token
     • unique token over total number of tokens
   • hapax legomena
     • word that occur once
   • unreliable to use alone
   • vocab size depends on text length

2. Word Frequencies

   • To present style of text
   • Bag-of-words approach
     • simple and efficient
     • disregard the word order or content of the text
   • function words / common words
     • among best feature to discriminate authors
     • selection are based on arbitrary criteria and language-dependent expertise
     • do not carry any semantic information
     • use in unconscious manner by the authors
   • most frequent words
     • work best to define lexical features
     • 100 features are adequate
     • content-specific words may included if the dimensionality increase

3. Word N-grams approach

   • provide word occurences or collocation
   • low accuracy compare to individual words
   • tend to capture content-specific rather than stylistic information

4. Writing Errors

   • capturing idiosyncrasies of author's style
   • the availability of accurate spell checker is still problematic
   • spelling errors
   • formatting errors

5. consist of token-based features

   • word length
   • sentence length
   • cab be applied to any language and any corpus
   • depends on the availability of a tokenizer
   • presence of too many abbreviations or acronyms may introduce noise in themeasures