Facebook FAIR’s WMT19 News Translation Task Submission

By Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov. 2019

424 citations recorded by the source corpus.

This paper describes Facebook FAIR’s submission to the WMT19 shared news translation task. We participate in four language directions, English <-> German and English <-> Russian in both directions. Following our submission from last year, our baseline systems are large BPE-based transformer models trained with the FAIRSEQ sequence modeling toolkit. This year we experiment with different bitext data filtering schemes, as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific data, then decode using noisy channel model reranking. Our system improves on our previous system’s performance by 4.5 BLEU points and achieves the best case-sensitive BLEU score for the translation direction English→Russian.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, methodology): “En ↔ De/En ↔ Ru: we evaluated publicly available single model checkpoints prior to finetuning from Ng et al. (2019) on WMT2019.”
How Can We Know What Language Models Know? (2019, methodology): “We use the round-trip English-German neural machine translation models pre-trained on WMT’19 (Ng et al., 2019) for back-translation, as English-German is one of the most highly resourced language pairs.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “…al. (2016; 2017); Chen et al. (2018) explored the idea of using a small neural network to steer an LM. Yu et al. (2016), and more recently Yu et al. (2019); Yee et al. (2019); Ng et al. (2019), leveraged the Shannon Noisy Channel Theory (Shannon, 1948) for improving sequence-to-sequence modeling.”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “Fairseq Models FAIRSEQ provides a collection of MT models (Ng et al., 2019; Lewis et al., 2020) and LMs (Liu et al., 2019; Conneau et al., 2020) that demonstrate state-of-the-art performance on standard benchmarks.”
Depth-Adaptive Transformer (2019, neutral): “For example, the winning entry of the WMT’19 news machine translation task in English-German used an ensemble totaling two billion parameters (Ng et al., 2019).”
Nearest Neighbor Machine Translation (2020, methodology): “…and domain adaptation experiments, we use the WMT’19 German-English news translation task winner (Ng et al., 2019), available via the F AIRSEQ library (Ott et al., 2019 For multilingual MT, we trained a 418M parameter Transformer-based encoder-decoder model on the CCMatrix data for 100K updates.”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, methodology): “FACEBOOK FAIR Facebook AI Research (Ng et al., 2019)”
Data Augmentation using Pre-trained Transformer Models (2020, methodology): “For backtranslation, we use a pre-trained EN-DE3, and DE-EN4 translation models (Ng et al., 2019).”
With Little Power Comes Great Responsibility (2020, neutral): “TF are Transformer-based (Ott et al., 2018; Edunov et al., 2018; Ng et al., 2019) and Conv are Convolutional models (Gehring et al.”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, methodology): “Data Augmentation Parameters: For data augmentation via round-trip translation, we use a beam size of 10 and k of 10 on German and Russian translation models; fairseq provides bidirectional pretrained translation models (Edunov et al., 2018) from WMT19 (Ng et al., 2019) for these language pairs.”
FUDGE: Controlled Text Generation With Future Discriminators (2021, background): “…for style transfer (Hu et al., 2017; Lample et al., 2018b; Dai et al., 2019a), summarization (See et al., 2017; Gehrmann et al., 2018; Zaheer et al., 2020), and machine translation (Lample et al., 2018a; Ng et al., 2019; Lewis et al., 2019) can also be viewed as CCLM’s for different “attributes.””
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, methodology): “, 2017) has prevailed in sequence modeling (Ng et al., 2019; Junczys-Dowmunt, 2018).”
Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020, methodology): “, 2017) trained on En-De and En-Ru WMT’19 news translation tasks (Ng et al., 2019).”
Facebook AI’s WMT20 News Translation Task Submission (2020, methodology): “To rerank our system output, we use the noisy-channel model (Yee et al., 2019) as the scoring model (Ng et al., 2019; Chen et al., 2019).”
CoVoST 2 and Massively Multilingual Speech-to-Text Translation (2020, positive): “d on characters (there is no word segmentation). pled from the OSCAR corpus (Ortiz Sua´rez et al., 2020) for each CoVoST 2 language, except for English, Russian for which pre-trained language models (Ng et al., 2019) are utilized2. 5K lines are reserved for validation and the rest for training. BPE ”
Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias (2020, methodology): “tem of WMT19 (Ng et al., 2019), which is provided as part of the Fairseq toolkit (F-WMT).”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, methodology): “To compute the per-plexity scores, we use transformer-based LMs (Ng et al., 2019).”
Efficient Nearest Neighbor Language Models (2021, methodology): “To inspect the domain-adaptation performance, our pretrained NLM is a 12-layer transformer model trained on WMT News Crawl 6 released by (Ng et al., 2019).”