Runaway Feedback Loops in Predictive Policing

By D. Ensign, Sorelle A. Friedler, Scott Neville, Carlos Eduardo Scheidegger, Suresh Venkatasubramanian. 2017

401 citations recorded by the source corpus.

Predictive policing systems are increasingly used to determine how to allocate police across a city in order to best prevent crime. Discovered crime data (e.g., arrest counts) are used to help update the model, and the process is repeated. Such systems have been empirically shown to be susceptible to runaway feedback loops, where police are repeatedly sent back to the same neighborhoods regardless of the true crime rate. In response, we develop a mathematical model of predictive policing that proves why this feedback loop occurs, show empirically that this model exhibits such problems, and demonstrate how to change the inputs to a predictive policing system (in a black-box manner) so the runaway feedback loop does not occur, allowing the true crime rate to be learned. Our results are quantitative: we can establish a link (in our model) between the degree to which runaway feedback causes problems and the disparity in crime rates between areas. Moreover, we can also demonstrate the way in which \emph{reported} incidents of crime (those reported by residents) and \emph{discovered} incidents of crime (i.e. those directly observed by police officers dispatched as a result of the predictive policing algorithm) interact: in brief, while reported incidents can attenuate the degree of runaway feedback, they cannot entirely remove it without the interventions we suggest.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “, 2020) and feedback loops (Ensign et al., 2018; Milli et al., 2019; Martin Jr. et al., 2020).”
Efficient candidate screening under multiple tests and implications for fairness (2019, background): “More recently, a number of papers have drawn inspiration from economic modeling, extending the literature on fairness in classification to consider longer-term dynamics, equilibria, and the emergence of feedback loops [10, 9, 7].”
50 Years of Test (Un)fairness: Lessons for Machine Learning (2018, background): “The rise of interest in fairness today has corresponded with public interest in the use of machine learning in criminal sentencing and predictive policing, including discussions around compas [16, 20, 42] and PredPol [25, 49].”
The Price of Fair PCA: One Extra Dimension (2018, background): “Much of the other work on fairness for learning algorithms focuses on fairness in classiﬁcation or scoring [Dwork et al., 2012; Hardt et al., 2016; Kleinberg et al., 2016; Chouldechova, 2017], or online learning settings [Joseph et al., 2016; Kannan et al., 2017; Ensign et al., 2017b,a].”
Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, background): “By considering historical examples across industries, we make the case that such audits can be leveraged to anticipate potential negative consequences before they occur, in addition to providing decision support to design mitigations, more clearly defining and monitoring potentially adverse outcomes”
Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics (2020, background): “…on (i.e., classiﬁes or regresses over) data that are static snap-shots of the problem domain, and consequently often ignores the non-linear and dynamically complex nature of society that involves feedback loops and time-delays between actions and impacts (Ensign et al., 2017; Liu et al., 2018).”
Algorithmic Fairness (2020, background): “For example, sending more police officers to an area that was predicted to be at high risk for crime will inevitably cause the arrest of more individuals in this area, and then, the prediction model will eventually further increase the risk prediction for the area [53].”
Residual Unfairness in Fair Machine Learning from Prejudiced Data (2018, methodology): “Ensign et al. (2017) consider a discrete model of how beliefs of crime rates in different areas adjust after observing arrest rates, and propose implementing the Horvitz-Thompson estimator via rejection sampling of arrest observations in an “online” fashion.”
A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and Dynamics (2018, background): “…records) are used to predict the location of new crimes and determine police deployment, runaway feedback loops can cause increasing surveillance of particular neighborhoods regardless of the true crime rate (Ensign et al., 2018), leading to over-policing of “high-risk” individuals (Stroud, 2016).”
What About Applied Fairness? (2018, background): “The attention and critical thought is well deserved as we see applications emerge which can dramatically impact people’s lives and families, such as predictive policing (Ensign et al., 2018) and sentencing (Choulde-chova, 2017; Barabas et al., 2018).”
Fairness and Transparency of Machine Learning for Trustworthy Cloud Services (2018, background): “Even though feedback loops rise due to user actions, it is possible to employ recent techniques as to detect their presence [27].”
A Framework for Understanding Unintended Consequences of Machine Learning (2019, background): “This can also lead to a feedback loop wherein the group is subject to further monitoring because of the apparent higher rate of mistakes (Barocas and Selbst, 2016; Ensign et al., 2017).”
Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018, background): “In the worst cases, the model can have a polarising effect: directing police resources disproportionately to areas with slightly higher crime risk will, without corrections, skew future training data collection in those area, which might be demographically or socioeconomically disproportionate [21].”
Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare (2022, background): “…requirements. example arises in the setting of predictive policing, in which an algorithm tries to allocate police across a city to prevent crimes: 73 showed how continual retraining of the algorithm on observed crime data, along with a naïve performance metric, can lead to runaway feedback loops…”
Why Is My Classifier Discriminatory? (2018, background): “Finally, additional data collection to improve the model may cause unexpected delayed impacts (Liu et al., 2018) and negative feedback loops (Ensign et al., 2017) as a result of distributional shifts in the data.”
The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning (2018, background): “Predictive algorithms can also create feedback loops, leading to unintended consequences (Ensign et al., 2017).”