ReadTwice: Reading Very Large Documents with Memories
By Yury Zemlyanskiy, J. Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha. 2021
18 citations recorded by the source corpus.
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose ReadTwice, a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books.
Subfields: Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Yuhuai Wu
- Markus Norman Rabe
- William W. Cohen
- Fei Sha
- Nicholas FitzGerald
- Sumit K. Sanghai
- Paul Christiano
- J. Ainslie
In Representation & Self-Supervised Learning
- Nicholas FitzGerald
- Fei Sha
- Paul Christiano
- Markus Norman Rabe
- Daniel M. Ziegler
- Ryan J. Lowe
- Christian Szegedy
- Jan Leike
Citations in the corpus
- Recursively Summarizing Books with Human Feedback (2021, methodology): “Our model’s summaries can help a human perform question answering quickly – see Appendix H – whereas the approach of Zemlyanskiy et al. (2021) produces hard-to-interpret latents.”
- LongT5: Efficient Text-To-Text Transformer for Long Sequences (2021, unknown): “For TriviaQA, we compare LongT5 with various top approaches on the leader board: BigBird-ETC (Zaheer et al., 2020a), Fusion-in-Decoder (Izacard and Grave, 2021), and ReadTwice (Zemlyanskiy et al., 2021).”
- FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2022, methodology): “We opt to introduce sparsity in attention layers as in ReadTwice (Zemlyanskiy et al., 2021) instead of attention patterns.”
- Training Language Models with Memory Augmentation (2022, unknown): “…de Masson D’Autume et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Zemlyanskiy et al., 2021; de Jong et al., 2022; Chen et al., 2022; Izacard and Grave, 2021; Singh et al., 2021), dialogue (Fan et al., 2021), and other knowledge-intensive NLP tasks (Lewis et al., 2020; Petroni et al., 2021”
- Memorizing Transformers (2022, background): “(Zemlyanskiy et al., 2021) process long sequences in two passes with different encoders.”