LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation

By Dong-Ho Lee, Rahul Khanna, Bill Yuchen Lin, Jamin Chen, Seyeon Lee, Qinyuan Ye, Elizabeth Boschee, Leonardo Neves, Xiang Ren. 2020

22 citations recorded by the source corpus.

Successfully training a deep neural network demands a huge corpus of labeled data. However, each label only provides limited information to learn from, and collecting the requisite number of labels involves massive human effort. In this work, we introduce LEAN-LIFE, a web-based, Label-Efficient AnnotatioN framework for sequence labeling and classification tasks, with an easy-to-use UI that not only allows an annotator to provide the needed labels for a task but also enables LearnIng From Explanations for each labeling decision. Such explanations enable us to generate useful additional labeled data from unlabeled instances, bolstering the pool of available training data. On three popular NLP tasks (named entity recognition, relation extraction, sentiment analysis), we find that using this enhanced supervision allows our models to surpass competitive baseline F1 scores by more than 5-10 percentage points, while using 2X times fewer labeled instances. Our framework is the first to utilize this enhanced supervision technique and does so for three important tasks – thus providing improved annotation recommendations to users and an ability to build datasets of (data, label, explanation) triples instead of the regular (data, label) pair.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alexander M. Rush
2. Colin Raffel
3. Victor Sanh
4. Thibault Févry
5. Mohit Bansal
6. Luke S. Zettlemoyer
7. Canwen Xu
8. Yonatan Belinkov

In Information Systems

1. Albert Webson
2. Colin Raffel
3. Alexander M. Rush
4. Stephen H. Bach
5. Canwen Xu
6. Tong Wang

In Evaluation & Benchmarks

1. Shanya Sharma Sharma
2. Han Wang
3. V. G. Vinod Vydiswaran
4. Manan Dey
5. Albert Webson
6. Xinyan Zhao
7. Colin Raffel
8. Alexander M. Rush

Citations in the corpus

• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, methodology): “More recently, many annotation systems have begun to incorporate learned models to improve workflow, using techniques such as ac-S1 tive learning (Lin et al., 2019; Li et al., 2021) and example recommendation (Lee et al., 2020; Kiela et al., 2021).”
• LIREx: Augmenting Language Inference with Relevant Explanation (2020, background): “Studies in this direction include (Li et al. 2016; Srivastava, Labutov, and Mitchell 2017; Wang et al. 2017; Hancock et al. 2018; Qin et al. 2020; Lee et al. 2020), in which the authors used semantic parsers to convert unstructured NLEs into structured feature-like logical forms.”