Few-shot Conformal Prediction with Auxiliary Tasks
By Adam Fisch, Tal Schuster, Tommi S. Jaakkola, Regina Barzilay. 2021
67 citations recorded by the source corpus.
We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tommi S. Jaakkola
- Tal Schuster
- Adam Fisch
- Regina Barzilay
- Percy Liang
- Luke S. Zettlemoyer
- Mohit Bansal
- Caiming Xiong
In Probabilistic & Bayesian Methods
- Tommi S. Jaakkola
- Regina Barzilay
- Stephen Bates
- Anastasios Nikolas Angelopoulos
- David M. Blei
- David Barber
- Martin J. Wainwright
- Michael I. Jordan
In Meta & Continual Learning
Citations in the corpus
- Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “Our work adds to several recent directions that explore CP in the context of risk-mitigating applications (Lei and Candès, 2020; Romano et al., 2020; Bates et al., 2020; Fisch et al., 2021a, inter alia ), or meta-learning settings (Fisch et al., 2021b).”
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, background): “The same researchers pooled information across multiple tasks in a meta-learning setup to form tight prediction sets for few-shot prediction [100].”