MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators

By Zhixing Tan, Xiangwen Zhang, Shuo Wang, Yang Liu. 2021

59 citations recorded by the source corpus.

Prompting has recently been shown as a promising approach for applying pre-trained language models to perform downstream tasks. We present Multi-Stage Prompting, a simple and automatic approach for leveraging pre-trained language models to translation tasks. To better mitigate the discrepancy between pre-training and translation, MSP divides the translation process via pre-trained language models into three separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre-trained language models better shift to translation tasks. We conduct extensive experiments on three translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained language models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Luke S. Zettlemoyer
3. Eric P. Xing
4. Biao Zhang
5. Zhiting Hu
6. Mohit Bansal
7. Yonatan Belinkov
8. Rico Sennrich

In Computer Vision and Pattern Recognition

1. Eric P. Xing
2. Biao Zhang
3. Zhiting Hu
4. Luc Van Gool
5. Mohit Bansal
6. Dacheng Tao
7. Trevor Darrell
8. Graham Neubig

In Natural Language Processing

1. Biao Zhang
2. Zhiting Hu
3. Graham Neubig
4. Barry Haddow
5. Eric P. Xing
6. Kyunghyun Cho
7. Alexandra Birch
8. Rico Sennrich

In Prompting & Reasoning

1. Stella Biderman
2. Eric P. Xing
3. Jason Phang
4. Jonas Oppenlaender

Citations in the corpus

• GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “There is a significant and growing body of work on the influence of the syntactic structure of finetuning data on downstream performance (Zhong et al., 2021; Tan et al., 2021; Sanh et al., 2021; Wei et al., 2021).”
• RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “…et al., 2019; He et al., 2021; Ding et al., 2022), and inspires a number of follow-up works that boost its performance (e.g., Liu et al., 2021c; Gu et al., 2021; Vu et al., 2021; Clive et al., 2021) or explore novel applications (e.g., Tan et al., 2022; Zhou et al., 2022; Levine et al., 2022).”
• Prompting Large Language Model for Machine Translation: A Case Study (2023, background): “With the popularity of prompting LLMs, researchers see value in incorporating prompts into neural MT (Li et al., 2022; Tan et al., 2021; Garcia and Firat, 2022).”