Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices

By Vincent S. Chen, Sen Wu, Zhenzhen Weng, Alexander J. Ratner, Christopher R'e. 2019

56 citations recorded by the source corpus.

In real-world machine learning applications, data subsets correspond to especially critical outcomes: vulnerable cyclist detections are safety-critical in an autonomous driving task, and “question” sentences might be important to a dialogue agent’s language understanding for product purposes. While machine learning models can achieve high quality performance on coarse-grained metrics like F1-score and overall accuracy, they may underperform on critical subsets—we define these as slices, the key abstraction in our approach. To address slice-level performance, practitioners often train separate “expert” models on slice subsets or use multi-task hard parameter sharing. We propose Slice-based Learning, a new programming model in which the slicing function (SF), a programming interface, specifies critical data subsets for which the model should commit additional capacity. Any model can leverage SFs to learn slice expert representations, which are combined with an attention mechanism to make slice-aware predictions. We show that our approach maintains a parameter-efficient representation while improving over baselines by up to 19.0 F1 on slices and 4.6 F1 overall on datasets spanning language understanding (e.g. SuperGLUE), computer vision, and production-scale industrial systems.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “Thus, we use the slicing functions of Chen et al. (2019) to find patterns of interest (e.g., prepositions in NLI), and perturb those patterns by placing [BLANK]s on the matched spans.”
Relevance-guided Supervision for OpenQA with ColBERT (2020, background): “Across a number of simple “slices” (Chen et al., 2019) of NQ-dev, Table 4 compares ColBERTQA3 (C3 for short) against ColBERT-QA1 (C1), DPR, and BM25, revealing gains due to RGS (vs.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Several previous investigations have found relatively small disparities in algorithmic performance across demographic groups (Chen et al., 2019; Larrazabal et al., 2020); Seyyed-Kalantari et al. (2020) finds larger disparities in true positive rates across demographic groups, and future work should…”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, methodology): “[10] Vincent Chen, Sen Wu, Alexander J Ratner, Jen Weng, and Christopher Ré.”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, background): “Erru-dite [Wu et al., 2019] supports users in exploring subpopulations of their data, while CrossCheck [Arendt et al., 2020] and Manifold [Zhang et al., 2018] focus on visualization and analysis for model comparison.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…efficient110 (Pfeiffer et al., 2020a; Rücklé et al., 2021), these111 modular approaches allow models to be extended112 to new data settings (Chen et al., 2019; Rücklé113 et al., 2020), where newly learned knowledge can114 be combined (Stickland and Murray, 2019; Wang115 et al., 2021a;…”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “For images, we can create a library of pre-trained object detectors as in (Chen et al., 2019), or in some applications combine primitives of geometric properties of the images (Varma & Ré, 2018).”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging (2019, methodology): “lass performance, particularly given the failure of this method in the detection of clinically relevant subclasses in the MURA task. More advanced semi-supervised methods such as those of Chen et al. [8] may ultimately be required to tackle this problem, or it may be the case that both unsupervised ”