SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

By Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, Alexandros Potamianos. 2019

86 citations recorded by the source corpus.

Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder (SEQˆ3), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Baziotis et al. (2019) performed sentence compression by chaining two sequence-to-sequence (seq2seq) models as an auto-encoder.”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, methodology): “Unsupervised summarization approaches include autoencoders to mirror the information compression inherent in summarization (Baziotis et al., 2019; Chu and Liu, 2019; Bražinskas et al., 2020b) as well as large-scale pretraining for domain-”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, methodology): “For summarization tasks, the work of Baziotis et al. (2019) proposes an intermediate autoencoder latent representation.”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, background): “…Conventional approaches for unsupervised paraphrasing and summarization have focused on task-specific surrogates – e.g., back-translation (Wi-eting and Gimpel, 2018; Hu et al., 2019b) and autoencoding (Huang and Chang, 2021; Baziotis et al., 2019) – that guide the model toward desired output.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, methodology): “Consequently, as long as text is represented by discrete tokens, loss gradients computed with a classiﬁer cannot be back-propagated without tricks such as the REINFORCE algorithm (He et al., 2016) or the Gumbel-Softmax approximation (Baziotis et al., 2019) which can be slow and unstable.”
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020, background): “In summarization there is an additional task to select a subset of the original text (Baziotis et al., 2019; Schumann et al., 2020; West et al., 2019).”