ListOps: A Diagnostic Dataset for Latent Tree Learning

By Nikita Nangia, Samuel R. Bowman. 2018

160 citations recorded by the source corpus.

Latent tree learning models learn to parse a sentence without syntactic supervision, and use that parse to build the sentence representation. Existing work on such models has shown that, while they perform well on tasks like sentence classification, they do not learn grammars that conform to any plausible semantic or syntactic formalism (Williams et al., 2018a). Studying the parsing ability of such models in natural language can be challenging due to the inherent complexities of natural language, like having several valid parses for a single sentence. In this paper we introduce ListOps, a toy dataset created to study the parsing ability of latent tree models. ListOps sequences are in the style of prefix arithmetic. The dataset is designed to have a single correct parsing strategy that a system needs to learn to succeed at the task. We show that the current leading latent tree models are unable to learn to parse and succeed at ListOps. These models achieve accuracies worse than purely sequential RNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Computational Linguistics & Semantics

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “Tasks used for comparison include: (1) a longer variation of the standard ListOps task proposed in (Nangia & Bowman, 2018), (2) byte-level text classiﬁcation using real-world data, (3) byte-level document retrieval, (4) image classiﬁcation on sequences of pixels, and (5) Pathﬁnder task (long-range…”
Pretrained Transformers as Universal Computation Engines (2021, methodology): “Taken from (Nangia and Bowman 2018; Tay et al. 2020), the model is shown a sequence of list operations (ex.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, methodology): “This benchmark task is a longer variation of the standard ListOps task proposed in (Nangia & Bowman, 2018), which was designed to investigate the parsing ability of neural models.”
Nyströmformer: A Nystöm-based Algorithm for Approximating Self-Attention (2021, methodology): “2020) with tasks of Listops (Nangia and Bowman 2018), byte-level IMDb reviews text classification (Maas et al.”
Random Feature Attention (2021, methodology): “We compare R FA against baselines on the following datasets: • ListOps ( LO ; Nangia & Bowman, 2018) aims to diagnose the capability of modelling hierarchically structured data.”
Luna: Linear Unified Nested Attention (2021, methodology): “We test its performance on three long-text modeling tasks in LRA (Tay et al., 2021), i.e., ListOps, Text and Retrieval and report results in Table 3.”
Long-Short Transformer: Efficient Transformers for Language and Vision (2021, methodology): “ListOps [47] is designed to measure the parsing ability of models through hierarchically structured data.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “We use the following tasks from this benchmark: • ListOps : parsing and evaluation of hierarchical expressions. a longer variant of (Nangia & Bowman, 2018); • Text : movie review sentiment analysis on the IMDB corpus (Maas et al., 2011); • Retrieval : article similarity classiﬁcation on the All…”
Analyzing Compositionality-Sensitivity of NLI Models (2018, methodology): “2 We give a brief description of each model below (see Table 1): RSE Residual Sentence Encoder Nie and Bansal (2017) is an encoding-based model that first uses multiple layers of residually-connected BiLSTM to encode the tokens in a sentence and then obtain the sentence’s fixed-length representation”
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (2020, background): “Naturally, the induced performance-based trees need not resemble linguistically plausible trees, and some studies (Williams et al., 2018a; Nangia & Bowman, 2018) examined the apparent fact that performance-based and lingusitically plausible trees bear little resemblance to one another.”
Gradient Estimation with Stochastic Softmax Tricks (2020, methodology): “We investigated the effect of X ’s structure and of the utility distribution in a latent parse tree task. the We used a simpliﬁed variant of ListOps dataset [61], which contains sequences of preﬁx arithmetic expressions, e.g., max[ 3 min[ 8 2 ]] , that evaluate to an integer in [0 , 9] .”
Revisiting Evaluation of Knowledge Base Completion Models (2020, methodology): “, 2019], semantic parsing [Nangia and Bowman, 2018] and NLI [Zellers et al.”
Simplified State Space Layers for Sequence Modeling (2022, background): “• ListOps: A lengthened version of the dataset presented by Nangia & Bowman [43].”
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation (2021, methodology): “We used ListOps [46], byte-level IMDb reviews text classiﬁcation [44], byte-level document retrieval [48], image classiﬁcation on sequences of pixels [37] and Pathﬁnder [40].”
cosFormer: Rethinking Softmax in Attention (2022, methodology): “ListOps contains some designed clever mathematical problem to clarify the parsing ability of neural models.”