3240236
Description
Mind Map by Ivan Zapreev, updated more than 1 year ago
|
Created by Ivan Zapreev
over 8 years ago
|
|
Trans-dimensional Random
Fields for Language Modeling
- Introduction
- language Modelling (LM)
- Joint probability of words
- Dominant
- Conditional approach
- Represent joint
probability in
terms of
conditionals
- Represent joint
probability in
terms of
conditionals
- Conditional approach
- Alternatives
- Random field
(RF)
- Used in
- Whole-sentence
maximum entropy
(WSME LMs)
- Is a Markov Random Field
- ?
- ?
- Challenge in Fitting
- Evaluating
gradient of log
likelihood
- ?
- Approximate
- Used Sampling methods
- Gibbs
- ?
- ?
- Independent
Metropolis-hashing
- ?
- ?
- Importance
- ?
- ?
- Can not work
efficiently with
complex
high-dimensional
distributions
- Empirical results
- Not satisfactory
- Poor fitted to the data
- Poor fitted to the data
- Not satisfactory
- Empirical results
- Gibbs
- Used Sampling methods
- Exact
- Requires
high-dimensional
integration
- ?
- ?
- Requires
high-dimensional
integration
- ?
- Poor fitted to the data
- Evaluating
gradient of log
likelihood
- Is a Markov Random Field
- Whole-sentence
maximum entropy
(WSME LMs)
- Potential benefits
- Naturally express
sentence level
phenomena
- ?
- ?
- Integrate features
from variety
knowledge sources
- ?
- ?
- Naturally express
sentence level
phenomena
- Used in
- Random field
(RF)
- Dominant
- Crucial for
- Computational linguistics
- Speech recognition
- Information retrieval
- Etc.
- Speech recognition
- Computational linguistics
- Joint probability of words
- language Modelling (LM)
- Research
- Revisit
- Random field
(RF)
- Innovations
- Propose
- Trans Dimensional
RF model (TDRF)
- Idea
- Take
account of
empirical
distributions
of lengths
- ?
- ?
- Allows to develop
- Markov Chain Monte
Carlo technique
- Trans-dimensional
mixture sampling
- Trans-dimensional
mixture sampling
- Markov Chain Monte
Carlo technique
- Take
account of
empirical
distributions
of lengths
- Idea
- Trans Dimensional
RF model (TDRF)
- Develop
- Training Algorithm
- Using
- Trans-dimensional
mixture sampling
- ?
- ?
- Stochastic
Approximation
(SA) framework
- ?
- ?
- Additional innovations
- Estimation of
Diagonal
elements of
hessian matrix.
- Estimated during SA
iterations to rescale
the gradient
- Improves convergence
- Improves convergence
- Estimated during SA
iterations to rescale
the gradient
- Word classing
- ?
- Accelerate sampling
- Improve the
smoothing
behavior
- Sharing statistical
strength between
similar woords
- Sharing statistical
strength between
similar woords
- ?
- Using multiple CPUs
- Parallelize training of RF model
- Parallelize training of RF model
- Estimation of
Diagonal
elements of
hessian matrix.
- Trans-dimensional
mixture sampling
- Fitting to the data
- Simultaneously update
- Model parameters
- Normalizing constants
- Model parameters
- Using
- Training Algorithm
- Propose
- Experiments
- Wall Street
Journal 92 data
- 1000 best lists
- Oracle WER is 3.4%
- Kaldi toolkit,
DNN acoustic
model
- Kaldi toolkit,
DNN acoustic
model
- Oracle WER is 3.4%
- MORE: In paper!
- 1000 best lists
- Comparison
- TDRF
- Performance
- As Good As
- Recurrent neural networks
- Recurrent neural networks
- As Good As
- Computational
- More efficient than
- Recurrent neural networks
- Computing
sentence
probability
- Recurrent neural networks
- More efficient than
- Performance
- Recurrent neural networks
- ?
- ?
- TDRF
- Wall Street
Journal 92 data
- Random field
(RF)
- Revisit
Want to create your own Mind Maps for free with GoConqr? Learn more.