Calibrated Structured Prediction

By Volodymyr Kuleshov, Percy Liang. 2015

133 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Verified Uncertainty Calibration (2019, methodology): “Recent work in machine learning typically estimates each term in the calibration error directly from samples [7, 15, 16, 17].”
Deep Anomaly Detection with Outlier Exposure (2018, methodology): “Prior research (Guo et al., 2017; Nguyen & O’Connor, 2015; Kuleshov & Liang, 2015) considers calibrating systems where test-time queries are samples from Din, but systems also encounter samples from Dtestout and should also ascribe low confidence to these samples.”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, background): “They admit extensions to the multi-class setting (Zadrozny and Elkan, 2002) and to structured prediction (Kuleshov and Liang, 2015).”
Calibration for the (Computationally-Identifiable) Masses (2017, neutral): “on calibration in this context, see [SSV03,FH15] and the references therein. Calibration has also been studied in the context of structured predictions where the supported set of predictions is large [KL15]. Our algorithmic result for multicalibration bears similarity to works from the online learni”
Analyzing Uncertainty in Neural Machine Translation (2018, methodology): “Calibration (Guo et al., 2017; Kuleshov & Liang, 2015) veriﬁes whether the model probability estimates p m match the true data probabilities p d .”
Inherent Disagreements in Human Textual Inferences (2019, background): “, 2006; Bunescu, 2008) and related discussions of model calibration (Kuleshov and Liang, 2015), we argue that the best approach is to propagate uncertainty downstream, so that end tasks can decide if and how to handle inferences on which humans are likely to disagree.”
To Trust Or Not To Trust A Classifier (2018, methodology): “In recent work, [5] explore the structured prediction setting, and [12] obtain confidence estimates by using ensembles of networks.”
Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings (2018, background): “Evaluation Metrics We compare the methods on three different measures of calibration error (Candela et al., 2005) - Expected Calibration Error (ECE) (as described in Section 2), Brier Score ( 1|D| ∑ (xi,yi)∼D ∑ v(ci,v − ri,v)2), and NLL.”
Calibrated Model-Based Deep Reinforcement Learning (2019, background): “They can be extended from binary to multi-class classiﬁcation (Zadrozny & Elkan, 2002), to structured prediction (Kuleshov & Liang, 2015), and to regression (Kuleshov et al., 2018).”
Unsupervised Quality Estimation for Neural Machine Translation (2020, methodology): “However, the calibration error metrics they use are designed for binary classification tasks and cannot be easily transferred to NMT (Kuleshov and Liang, 2015). Ott et al. (2019) analyze uncertainty in NMT by comparing predictive probability distributions with the empirical distribution observed in ”
Multicalibration: Calibration for the (Computationally-Identifiable) Masses (2018, background): “Calibration has also been studied in the context of structured predictions where the supported set of predictions is large (Kuleshov & Liang, 2015).”
Teaching Models to Express Their Uncertainty in Words (2022, methodology): “The MSE combines calibration error with “sharpness” (Kuleshov & Liang, 2015), while the MAD (below) just measures the former.”
Mitigating bias in calibration error estimation (2020, background): “In addition, the mean squared prediction error of the classifier, or Brier score [3], can be decomposed into terms corresponding to the squared `2 calibration error and the variance of the model’s correctness likelihood [21, 22].”