Word Embeddings

Word Embeddings

1. Map Word/Document into Vector Space - Each dimension aims to capture meaning (ex: gender dimension, happiness dimension)

   • Learning the Vector Values (for each word)
     • Learning a dense distributed representation for each word = 10/100 dimensions vs 100000 dimensions with bag-of-words
       • Words that are used similar ways = similar representations
         • Embedding Layer
           • Embedding that is learned jointly with a neural net on a specific NLP task
             • • Word2Vec
                 • CBOW
                   • predicts current word based on surrounding words
                 • Skip Gram
                   • Predicts surrounding words based on current word
                 • http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
                 • What is the probability of finding a certain word next to / surrounding (== context!!!) my input word?
               • GloVe
                 • Adds matrix factorization techniques like LSA to Word2Vec
               • Bert
               • Keras Embedding Layer
                 • Each word is one-hot-encoded
                 • keras.layers.Embedding
                   • First Hidden Layer of a NN
                     • Input: pos. integer encoded words
                       • input_dim: size of vocab, output_dim: size of vectors, input_length: #words of each input document
                         • Output: 2D vector with embedding for each word in input document
                           • To connect to Dense Layer: First flatten to 1D Vector. Connect to a flatten layer model.add(Flatten())
           • used alone to learn embedding
           • part of a model where the embedding is learned along with the model
           • improve a pre-trained embedding layer
           • First Hidden Layer in the NN (when learning the embedding as part of a task)
           • Stand-alone layer when not training along a task

2. Distributional Semantics: Categorizing semantic similarities based on semantic items that are distributed across a text