Assessing Risk Assessment in Action

By M. Stevenson. 2018

Recent years have seen a rush toward evidence-based tools in criminal justice. As part of this movement, many jurisdictions have adopted actuarial risk assessment to supplement or replace the ad-hoc decisions of judges. Proponents of risk assessment tools claim that they can dramatically reduce incarceration without harming public safety. Critics claim that risk assessment will exacerbate racial disparities. Despite extensive and heated rhetoric, there is virtually no evidence on how use of this “evidence-based” tool affects key outcomes such as incarceration rates, crime, or racial disparities. The research discussing what “should” happen as a result of risk assessment is hypothetical and largely ignores the complexities of implementation. This Article is one of the first studies to document the impacts of risk assessment in practice. It evaluates pretrial risk assessment in Kentucky, a state that was an early adopter of risk assessment and is often cited as an example of best-practices in the pretrial area. Using rich data on more than one million criminal cases, this Article shows that a 2011 law making risk assessment a mandatory part of the bail decision led to a significant change in bail setting practice, but only a small increase in pretrial release. These changes eroded over time as judges returned to their previous habits. Furthermore, the increase in releases was not cost-free: failures-to-appear and pretrial crime increased as well. Risk assessment had no effect on racial disparities in pretrial detention once differing regional trends were accounted for. Kentucky’s experience does not mean we should abandon risk assessment, but it should temper the hyperbolic hopes (and fears) about its effects. Risk assessment in practice is different from risk assessment in the abstract, and its impacts depend on context and details of implementation. If indeed risk assessment is capable of producing large benefits, it will take research and experimentation to learn how to achieve them. Such a process would be evidence-based criminal justice at its best: not a flocking toward methods that bear the glossy veneer of science, but a careful and iterative evaluation of what works and what does not.

Subfields: Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Social Sciences & Humanities

Citations in the corpus

Measurement and Fairness (2019, background): “Indeed, interacting with measurements can create room for more individual judgments, automation bias, and (lack of) intuition and interpretability [6, 16, 28, 38, 55, 60].”
Fairness and Abstraction in Sociotechnical Systems (2019, background): “How scores are initially used might differ from what happens when judges see them frequently [65].”
A Causal Bayesian Networks Viewpoint on Fairness (2018, neutral): “This uneven distribution of police attention, as well as other factors such as funding for pretrial services [30, 43], means that diﬀerences in base rates be-tween racial groups are not reﬂective of ground truth rates.”
The false promise of risk assessments: epistemic reform and the limits of fairness (2020, background): “In Kentucky, for instance, the mandatory use of a pretrial risk assessment led to increases in release for low and medium risk defendants and a decrease in release for high risk defendants [152].”
The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, background): “This lack of evidence is particularly troubling in light of research which suggests that people struggle to interpret machine learning models and to incorporate algorithmic predictions into their decisions, often leading machine learning systems to generate unexpected and unfair outcomes [37, 67].”
A Framework for Understanding Unintended Consequences of Machine Learning (2019, background): “Stevenson (2018) builds on this idea, and through an in-depth study of the deployment of risk assessment tools in Kentucky, demonstrates how evaluating the system in isolation created highly unrealistic notions of what the real ben-eﬁts and consequences actually were.”
It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks (2021, background): “] The important question is whether the use of actuarial risk assessment tools results in more disparate outcomes than the status quo, or other viable alternatives” [12].”
Simple rules to guide expert classifications (2020, background): “In the criminal justice setting, past work indeed suggests that judges may not apply the recommendations of risk assessments in a consistent manner (Christin, 2017; DeMichele et al., 2018; Stevenson, 2018).”
This Thing Called Fairness (2019, background): “However, even setting aside concerns about biases in the data, how such tools interact with existing work practices is unclear and an area of active research [52, 145].”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, background): “Stevenson [41] builds on this idea, and through an in-depth study of the deployment of risk assessment tools in Kentucky, demonstrates how evaluating the system in isolation created unrealistic notions of its benefits and consequences.”
Simplicity Creates Inequity: Implications for Fairness, Stereotypes, and Interpretability (2018, unknown): “We are already seeing algorithms play this role in domains including hiring, education, lending, medicine, and criminal justice [8, 22, 34, 36].”
Human-AI Interactions in Public Sector Decision-Making:"Automation Bias"and"Selective Adherence"to Algorithmic Advice (2021, background): “Stevenson (2018) uses archival data of criminal cases from the state of Kentucky to compare observationally detention rates before and after a reform in 2011 that made risk assessment mandatory in pretrial procedures.”
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments (2019, background): “This is an important factor to consider when deploying algorithmic systems, especially given research demonstrating that the changes instigated by risk assessments are short-lived [59].”
Fairness in Machine Learning (2020, background): “As previous research has shown [81, 113, 160], modern policing tactics center around targeting a small number of neighborhoods—often disproportionately populated by non-white and low income residents—with recurring patrols and stops.”