Pairwise Neural Machine Translation Evaluation

Pairwise Neural Machine Translation Evaluation

1. Introduction

   • Automated Machine Translation (MT)
     • Evaluation
       • Needed
         • Developing a new MT
         • Comparing two MT
       • Reference based MT
         • Comparing the system output to one or more human reference tranlations
         • Most Common
         • Compute
           • Absolute quality score
             • Computing similarity between the machine and human translation
           • Simplest case
             • Computing word N-gram matches between the translation and the reference
               • BLUE
           • More advanced
             • Take into account various aspects of linguistic similarity
               • Better correlation with human judjment
       • Human ranking
         • Can be used to train automatic metrics
           • Can be oriented to predict absolute scores
             • Using
               • Regression
                 • ?
               • Ranking
                 • Special case
                   • Compare two hypotheses and referenec
                     • Decide which hypothesis is better
                       • Recent result
                         • Guzman
                           • Learning framework
                             • Using
                               • Preference kernel
                                 • ?
                               • Vector machines (SVM)
                               • Syntactic structures
                               • Discourage-based structures
                             • High computational costs
                               • Training
                               • Testing
                               • Due to
                                 • Using convolution kernels
                                   • ?
                                 • Over complex structures
                               • Simplification is needed!

2. Research

   • Framework for machine translation evaluation
     • Novel!
     • Goal
       • Select a better translation from a pair of hypothesis, given the reference translation
     • Using
       • Neural networks
         • Multi-layer
           • Input layer
             • Semantic info
             • Syntactic info
             • Lexical info
           • Hidden layer
             • Captures the interactions between the relevant input components
         • Models the interaction between
           • The two hypothesis
           • Reference and Hypothesis
       • Distributed vector representations
         • Used for storing
           • The Two hypothesis
         • Based on
           • Word embedding
           • Sentence embedding
         • Learned from
           • Neural Networks
             • Novel!
             • Can be trained to optimize task-specific cost function
         • Efficient
           • Vector-based compression
             • ?
     • Experiments
       • WMT12 metrics task
       • Better results than by Guzman
       • High correlation with human judjment
       • Comparable with the best!
         • DiscoTK
           • Metric
             • Combination based
               • Much heavier
       • Embeddings
         • Syntactically oriented
         • Semantically oriented
         • Cumulative performance gains
           • Over
             • BLUE
             • NIST
             • METEOR
             • TER
   • Simplification is needed!