A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer

By Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, Xu Sun. 2019

188 citations recorded by the source corpus.

Unsupervised text style transfer aims to transfer the underlying style of text but keep its main content unchanged without parallel data. Most existing methods typically follow two steps: first separating the content from the original style, and then fusing the content with the desired style. However, the separation in the first step is challenging because the content and style interact in subtle ways in natural language. Therefore, in this paper, we propose a dual reinforcement learning framework to directly transfer the style of the text via a one-step mapping model, without any separation of content and style. Specifically, we consider the learning of the source-to-target and target-to-source mappings as a dual task, and two rewards are designed based on such a dual structure to reflect the style accuracy and content preservation, respectively. In this way, the two one-step mapping models can be trained via reinforcement learning, without any use of parallel data. Automatic evaluations show that our model outperforms the state-of-the-art systems by a large margin, especially with more than 10 BLEU points improvement averaged on two benchmark datasets. Human evaluations also validate the effectiveness of our model in terms of style accuracy, content preservation and fluency. Our code and data, including outputs of all baselines and our model are available at https://github.com/luofuli/DualRL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, methodology): “…Delete-only (Li et al., 2018) 81.4 28.6 606 Delete-retrieve (Li et al., 2018) 86.2 31.1 948 Unpaired RL (Xu et al., 2018) 52.2 37.2 2,750 Dual RL (Luo et al., 2019) 85.9 55.1 982 Style transformer (Dai et al., 2019) to be a little less angsty • to be about mining • to be better written • to be…”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “, 2019; J Kurisinkel and Chen, 2019), style (Fu et al., 2018; Luo et al., 2019b; Li et al., 2018), topics (Feng et al.”
Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach (2021, methodology): “Previous work has used RL for the task of sentiment transfer [35] by only using text generations as actions.”
Text Detoxification using Large Pre-trained Neural Models (2021, methodology): “Alternatively, the model DualRL by Luo et al. (2019) performs a direct transfer from the source to the target style.”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “Unsupervised style transfer is often modeled by disentangling style & content using attribute clas-siﬁers (Hu et al., 2017; Shen et al., 2017), policy gradient training (Xu et al., 2018; Luo et al., 2019) or retrieval-based approaches (Li et al., 2018).”
Text Style Transfer: A Review and Experiment Evaluation (2020, methodology): “While there are no TST models that achieved the best performance in all evaluation metrics, DualRL [48], PTO [78], B-GST [70], and PFST [20] have achieved a well-balanced trade-off among between text fluency, content preservation, and the style transfer accuracy.”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, methodology): “Following Luo et al. (2019b), we used the samples from the “Family & Relationship” (F&R) domain and restricted our focus to the informal to formal direction.”
Deep Learning for Text Style Transfer: A Survey (2020, methodology): “…(3) In training, ACO can be trained in two ways: either a normal loss function trained by Gumbel-softmax distribution to approximate the discrete training (Jang, Gu, and Poole 2017), or a negative reward for reinforcement learning by policy gradient training (Williams 1992) as in Luo et al. (2019).”
Learning to Control the Fine-grained Sentiment for Story Ending Generation (2019, background): “To date several works of dialogue generation (Zhou et al., 2018; Huang et al., 2018; Zhou and Wang, 2018) and text sentiment transfer task (Li et al.; Luo et al., 2019) have studied on generating emotional or sentimental text.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, methodology): “We compare the output text that CAE-T5 generates with a selection of unpaired style-transfer models described in Section 2.2 (Shen et al., 2017; Li et al., 2018; Fu et al., 2018; Luo et al., 2019; Dai et al., 2019a).”