Incorporating BERT into Neural Machine Translation

By Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wen-gang Zhou, Houqiang Li, Tie-Yan Liu. 2020

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For other conditional NLG tasks such as machine translation and dialogue generation, fine-tuning is also the prevalent paradigm (Zhang et al., 2020c; Stickland et al., 2020; Zhu et al., 2020; Liu et al., 2020).”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “, 2019); successful adaptations of LM prediction heads include machine translation (Zhu et al., 2020), sentiment classification (Brown et al.”
Masked Language Model Scoring (2019, methodology): “This handicapped their use in generative tasks, where they at best served to bootstrap encoder-decoder models (Clinchant et al., 2019; Zhu et al., 2020) or unidirectional LMs (Wang et al.”
Pre-trained models for natural language processing: A survey (2020, methodology): “[221] suggested using pre-trained BERT as an extra memory to facilitate NMT models.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “Following this line of research, recently, there have been initiatives to incorporate BERT inetuning for NMT [215].”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “Examples are constituency parsing [217], semantic graph parsing using Universal Conceptual Cognitive Annotation (UCCA) [1, 73] and Abstract Meaning Representation (AMR) [8, 124, 214, 226], Aspect-Based Sentiment Analysis [101], and Machine Translation (MT) [228].”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, methodology): “…al., 2019; Lewis et al., 2019) Specifically for MT, Ramachandran et al. (2017) proposed pre-training the encoder-decoder modules as two separate language models, and Yang et al. (2019a); Zhu et al. (2020) explored approaches incorporating BERT model weights into the usual seq-to-seq architecture.”
R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “We calculate the BLEU scores on these tasks for evaluation, following [80].”
TransVG: End-to-End Visual Grounding with Transformers (2021, background): “This superiority has attracted a surge of research interest in applications of transformers in NLP tasks [11, 12, 39, 66] and speech recognition [33, 50].”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “…et al., 2020; Jalili Sabet et al., 2020; Chen et al., 2020; Zenkel et al., 2020; Dou and Neubig, 2021; Imamura and Sumita, 2019; Ma et al., 2020; Zhu et al., 2020; Liu et al., 2020b; Xue et al., 2021) which use MLLMs for downstream bilingual tasks such as unsupervised machine translation,…”
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, methodology): “In the literature, several recent works [11]–[14] have tried to incorporate BERT, which is one of the most successful representatives of PLMs, into natural language generation tasks, mainly by leveraging the feature representations encoded by BERT.”
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021, methodology): “The ﬁrst category usually uses pre-trained models as feature extractors of a source language, or initializes the encoder and decoder with pre-trained models separately (Lample and Conneau, 2019; Ren et al., 2019; Yang et al., 2020a; Zhu et al., 2020).”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, methodology): “For example, leveraging BERT as contextual embedding has been proved useful for NMT by borrowing informative context from the pretrained model [176].”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “Zhu et al. (2020) use an attention model to incorporate the output of BERT into encoder and decoder of NMT.”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “In our preliminary study, we found that even a Small configuration, which is commonly used for low-resourced translation (Zhu et al., 2019), can still overfit on Multi30k.”
MathBERT: A Pre-Trained Model for Mathematical Formula Understanding (2021, methodology): “These pre-trained models performed well in general NLP tasks like text classiﬁcation [Devlin et al. , 2018], machine translation [Zhu et al. , 2020; Sundararaman et al. , 2019] and machine summarization [Miller, 2019; Xenouleas et al. , 2019].”