Learning and Evaluating General Linguistic Intelligence

By Dani Yogatama, Cyprien de Masson d'Autume, Jerome T. Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer and 1 more. 2019

219 citations recorded by the source corpus.

We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Unlike Fisch et al. (2019); Yogatama et al. (2019), we measure OOD generalization by considering simple and natural distribution shifts, and we also evaluate more than question answering.”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “A possible reason is that the auxiliary task alleviates catastrophic forgetting (Yogatama et al., 2019) when ﬁne-tuning on augmented data.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, neutral): “Some of our experiments resemble those of Yogatama et al. (2019), who also empirically investigate transfer performance with limited amounts of data and ﬁnd similar evidence of catastrophic forgetting.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Nevertheless, there has been mounting evidence in recent work suggesting that the performance of these models as measured by the benchmark datasets may be overestimated, non-generalizable and at least partially driven by exploiting existing spurious correlations in training datasets [21, 22, 40, 47,”
The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “Given the plethora of question-answering datasets, Yogatama et al. (2019), Talmor & Berant (2019), and Sen & Saffari (2020) evaluate the extent to which models trained on SQuAD generalize to other question-answering datasets.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “…is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out-of-distribution generalisation (Chen et al., 2016; Weissenborn et al., 2017; Yogatama et al., 2019; McCoy et al., 2019).”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “We focus on zero-shot evaluation as it enables us to examine the representations without confounding our ﬁndings with the side-effects of ﬁne-tuning (Yogatama et al., 2019) or probing classi-ﬁers ( e.g. , Zhang and Bowman, 2018; Hewitt and Liang, 2019).”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “How do we know that incremental progress on today’s tasks will take us to our end goal, whether that is “General Linguistic Intelli-gence” (Yogatama et al., 2019) or a system that passes the Turing test or a system that captures the meaning of English, Arapaho, Thai, or Hausa to a linguist’s…”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “While we speculate that TAPT followed by DAPT would be susceptible to catastrophic forgetting of the task-relevant corpus (Yogatama et al., 2019), alternate methods of combining the procedures may result in better downstream performance.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “In retrospect, researchers have advocated for build- ing more human-like, general linguistic intelli- gence that can “reuse previously acquired knowl- edge about a language and adapt to a new task quickly” (Yogatama et al., 2019; Linzen, 2020).”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “We compare our proposed model with the following baselines: (i) Finetune (Yogatama et al., 2019): We ﬁnetuned GPT-2 model on several tasks sequentially.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, methodology): “• Finetune (Yogatama et al., 2019): finetune BERT model sequentially without the episodic memory module and any other loss.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “In the NLP domain, Yogatama et al. (2019) introduced a new metric, based on an online (prequential) encoding (Blier and Ollivier, 2018), which measures the adoption rate of an existing model to a new task.”