Unsupervised Sentence Compression using Denoising Auto-Encoders

By Thibault Févry, Jason Phang. 2018

73 citations recorded by the source corpus.

In sentence compression, the task of shortening sentences while retaining the original meaning, models tend to be trained on large corpora containing pairs of verbose and compressed sentences. To remove the need for paired corpora, we emulate a summarization task and add noise to extend sentences and train a denoising auto-encoder to recover the original, constructing an end-to-end training regime without the need for any examples of compressed sentences. We conduct a human evaluation of our model on a standard text summarization dataset and show that it performs comparably to a supervised baseline based on grammatical correctness and retention of meaning. Despite being exposed to no target data, our unsupervised models learn to generate imperfect but reasonably readable sentence summaries. Although we underperform supervised models based on ROUGE scores, our models are competitive with a supervised baseline based on human evaluation for grammatical correctness and retention of meaning.

Subfields: Artificial Intelligence, Natural Language Processing, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Ximing Lu
3. Peter West
4. Faeze Brahman
5. Liwei Jiang
6. Ioannis Konstas
7. Jaehun Jung
8. Alexandros Potamianos

In Natural Language Processing

1. Yejin Choi
2. Ximing Lu
3. Peter West
4. Faeze Brahman
5. Liwei Jiang
6. Jaehun Jung
7. Ioannis Konstas
8. Ion Androutsopoulos

In Summarization

1. Yejin Choi
2. Ion Androutsopoulos
3. Alexandros Potamianos
4. Ioannis Konstas
5. Christos Baziotis

Citations in the corpus

• Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, methodology): “Common approaches include back-translation (Wieting and Gimpel, 2018) and auto-encoding (Févry and Phang, 2018), but are often limited in terms of corpus diversity and noisiness (Hu et al., 2019a,b).”
• SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (2019, methodology): “The re-sults of the extractive model of Fevry and Phang (2018) are also not comparable, as they were obtained on a different, not publicly available test set.”