Machine learning with operational costs

By Theja Tulabandhula, Cynthia Rudin. 2011

59 citations recorded by the source corpus.

This work proposes a way to align statistical modeling with decision making. We provide a method that propagates the uncertainty in predictive modeling to the uncertainty in operational cost, where operational cost is the amount spent by the practitioner in solving the problem. The method allows us to explore the range of operational costs associated with the set of reasonable statistical models, so as to provide a useful way for practitioners to understand uncertainty. To do this, the operational cost is cast as a regularization term in a learning algorithm's objective function, allowing either an optimistic or pessimistic view of possible costs, depending on the regularization parameter. From another perspective, if we have prior knowledge about the operational cost, for instance that it should be low, this knowledge can help to restrict the hypothesis space, and can help with generalization. We provide a theoretical generalization bound for this scenario. We also show that learning with operational costs is related to robust optimization.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Forward stability and model path selection (2021, neutral): “Within the recent Rashomon literature are methods for decision making (Tulabandhula and Rudin 2013, 2014), identifying ranges of variable importance within a model class (Fisher et al. 2019), and measuring model class simplicity (Semenova and Rudin 2019).”
Predictive Multiplicity in Classification (2019, neutral): “derlying distribution). Other approaches include choose a single model to “break ties" between equally good models on the basis of secondary criteria, such as simplicity [48] or operational cost [50]. 2Framework In this section, we introduce measures to describe the multiplicity of predictions ”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, methodology): “The work of Fisher et al. grew out of work by Tulabandhula and Rudin [42, 43] that uses the Rashomon set to assist with decision making, by finding the range of downstream operational costs associated with the Rashomon set.”
Smart "Predict, then Optimize" (2017, background): “The approach in Tulabandhula and Rudin (2013) relies on minimizing a loss function that combines the prediction error with the operational cost of the model on an unlabeled dataset.”
Distributionally Robust Optimization: A Review (2019, methodology): “Tulabandhula and Rudin [303] incorporate machine learning for the decision making.”
Robust Optimization using Machine Learning for Uncertainty Sets (2014, background): “In our previous work [Tulabandhula and Rudin, 2013, 2014] we considered statistical learning theory bounds also for cases when unlabeled points were available.”
Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models (2019, background): “ea that multiple models can explain the observed dataset well, namely the Rashomon eect (Breiman et al. (2001)), is discussed in other works that are not related to variable importance. For example, Tulabandhula and Rudin (2013) implement robust optimization to nd a decision rule that works well und”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, background): “The “Machine Learning with Operational Costs” framework (Tulabandhula and Rudin 2014a, 2013) computes a tethered hacking interval of the cost that a company might incur to enact an optimal policy in response to any good predictive model.”
On combining machine learning with decision making (2011, methodology): “These bounds are different than those of Tulabandhula and Rudin (2013) which use concentration of Rademacher complexity and Dudley’s entropy integral, and are not directly comparable.”