jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

By Yada Pruksachatkun, Philip Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman. 2020

97 citations recorded by the source corpus.

We introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. jiant enables modular and configuration driven experimentation with state-of-the-art models and a broad set of tasks for probing, transfer learning, and multitask training experiments. jiant implements over 50 NLU tasks, including all GLUE and SuperGLUE benchmark tasks. We demonstrate that jiant reproduces published performance on a variety of tasks and models, e.g., RoBERTa and BERT.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, methodology): “We use jiant [Wang et al., 2019] to compute the baseline.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “In this context, the GLUE benchmark (Wang et al., 2019a) has become a prominent evaluation framework for research towards general-purpose language understanding technologies.”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “We adjust the Jiant toolkit [42] to process our architectures.”
Transformers: State-of-the-Art Natural Language Processing (2020, unknown): “Their framework Jiant (Pruksachatkun et al., 2020) allows them to experiment with different ways of pretraining models and comparing their outputs.”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, methodology): “We follow Pruksachatkun et al. (2020) for training hyperparameters.”
Comparing Test Sets with Item Response Theory (2021, methodology): “We use the jiant (Pruksachatkun et al., 2020b) library which is based on”
Can neural networks acquire a structural bias from raw linguistic data? (2020, methodology): “We use Huggingface’s (Wolf et al., 2019) implementation of BERT-Large in PyTorch (Paszke et al., 2017), and carry out fine-tuning in jiant (Wang et al., 2019).”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “We then benchmark each conﬁguration with and without TISA for 5 runs on GLUE tasks (Wang et al., 2018), using jiant (Phang et al., 2020) and standard dataset splits to evaluate performance.”