Learning Continuous Phrase Representations for Translation Modeling

By Jianfeng Gao, Xiaodong He, Wen-tau Yih, Li Deng. 2014

123 citations recorded by the source corpus.

This paper tackles the sparsity problem in estimating phrase translation probabilities by learning continuous phrase representations, whose distributed nature enables the sharing of related phrases in their representations. A pair of source and target phrases are projected into continuous-valued vector representations in a low-dimensional latent space, where their translation score is computed by the distance between the pair in this new space. The projection is performed by a neural network whose weights are learned on parallel training data. Experimental evaluation has been performed on two WMT translation tasks. Our best result improves the performance of a state-of-the-art phrase-based statistical machine translation system trained on WMT 2012 French-English data by up to 1.3 BLEU points.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “…multi-task framework is extensible to other meth-ods for sentence/document representations, such as those based on convolutional networks (Kalchbren-ner et al., 2014; Shen et al., 2014; Gao et al., 2014b), parse tree structure (Irsoy and Cardie, 2014), and run-time inference (Le and Mikolov, 2014).”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, background): “Recent progress in SMT stemming from the use of neural language models (Sutskever et al., 2014; Gao et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) has inspired attempts to extend these neural techniques to response generation.”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, methodology): “Continuous representations of words and phrases estimated by neural network models have been applied on a variety of tasks ranging from Information Retrieval (IR) (Huang et al., 2013; Shen et al., 2014), Online Recommendation (Gao et al., 2014b), Machine Translation (MT) (Auli et al., 2013; Cho et a”
Neural Approaches to Conversational AI (2018, neutral): “wide range of tasks depending on the deﬁnition of (x;y). For example, (x;y) is a query-document pair for Web search ranking (Huang et al., 2013; Shen et al., 2014), a document pair in recommendation (Gao et al., 2014b), a question-answer pair in QA (Yih et al., 2015a), a sentence pair of different l”
A Persona-Based Neural Conversation Model (2016, background): “Progress in SMT stemming from the use of neural language models (Sutskever et al., 2014; Gao et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) has inspired efforts to extend these neural techniques to SMT-based conversational response generation.”
Multi-Task Learning for Multiple Language Translation (2015, background): “Much work has been done to address the data sparsity problem such as the pivot language approach (Wu and Wang, 2007; Cohn and Lapata, 2007) and deep learning techniques (Devlin et al., 2014; Gao et al., 2014; Sundermeyer et al., 2014; Liu et al., 2014).”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “Gao et al. (2014) leverage MRT for learning continuous phrase representations for statistical machine translation.”
Deep Learning: Methods and Applications (2014, background): “A more recent study on applying deep learning methods to machine translation appears in [121, 123], where the phrase-translation component, rather than the LM component in the machine translation system is replaced by the neural network models with semantic word embeddings.”
Modeling Interestingness with Deep Neural Networks (2014, neutral): “…techniques are able to automatically discover from training data the hidden structures and the associated features at different levels of abstraction useful for a variety of tasks (e.g., Collobert et al. 2011; Hinton et al. 2012; Socher et al. 2012; Krizhevsky et al., 2012; Gao et al. 2014).”
Mitigating Gender Bias Amplification in Distribution by Posterior Regularization (2020, background): “For example, Ji et al. (2012); Gao et al. (2014) design constraints based on similarity to improve question answering and machine translation, respectively.”
A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval (2014, background): “Extended from our previous work [20] [33], the CLSM and its variations have also been demonstrated giving superior performance on a range of natural language processing tasks beyond information retrieval, including machine translation [13], semantic parsing and question answering [40], entity search”
Neural Headline Generation with Sentence-wise Optimization (2016, methodology): “To the best of our knowledge, although MRT has been widely used in many NLP tasks such as statistical machine translation (Och, 2003; Smith and Eisner, 2006; Gao et al., 2014; Shen et al., 2015), it has not been well considered in the research of text summarization.”
Investigating the Usefulness of Generalized Word Representations in SMT (2014, background): “More recent research has started to set apart from the conventional maximum likelihood estimates toward neural network-based models that use continuous space representation (Schwenk, 2012; Le et al., 2012; Hu et al., 2014; Gao et al., 2014).”
Context-Dependent Translation Selection Using Convolutional Neural Network (2015, methodology): “For example, Gao et al. (2014) projected phrases from both source and target sides into a common, continuous space that is language independent.”
An Autoencoder Approach to Learning Bilingual Word Representations (2014, background): “[22] which showed that a useful linear mapping between separately trained monolingual skip-gram language models could be learned.”