Build It, Break It, Fix It: Contesting Secure Development

By Andrew Ruef, Michael W. Hicks, James Parker, Dave Levin, Michelle L. Mazurek, Piotr (Peter) Mardziel. 2016

78 citations recorded by the source corpus.

Typical security contests focus on breaking or mitigating the impact of buggy systems. We present the Build-it, Break-it, Fix-it (BIBIFI) contest, which aims to assess the ability to securely build software, not just break it. In BIBIFI, teams build specified software with the goal of maximizing correctness, performance, and security. The latter is tested when teams attempt to break other teams' submissions. Winners are chosen from among the best builders and the best breakers. BIBIFI was designed to be open-ended-teams can use any language, tool, process, etc. that they like. As such, contest outcomes shed light on factors that correlate with successfully building secure software and breaking insecure software. During 2015, we ran three contests involving a total of 116 teams and two different programming problems. Quantitative analysis from these contests found that the most efficient build-it submissions used C/C++, but submissions coded in other statically-typed languages were less likely to have a security flaw; build-it teams with diverse programming-language knowledge also produced more secure code. Shorter programs correlated with better scores. Break-it teams that were also successful build-it teams were significantly better at finding security bugs.

Subfields: Artificial Intelligence, Information Systems, Privacy & Security, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Privacy & Security

Jordan L. Boyd-Graber

In Signal Processing

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “The “Build it Break it Fix it” strategy in the security domain (Ruef et al., 2016) has been adapted to NLP (Ettinger et al., 2017) as well as dialogue safety (Dinan et al., 2019).”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, methodology): “As mentioned in the introduction, our approach takes inspiration from “Build it Break it” approaches which have been successfully tried in other domains (Ruef et al., 2016; Ettinger et al., 2017).”
What Question Answering can Learn from Trivia Nerds (2019, methodology): “This improves not only the quality of evaluation (we can have real-time human judging) but also lets the test set reflect the build it/break it cycle (Ruef et al., 2016), as attempted by the 2019 iteration of FEVER (Thorne et al., 2019).”
Understanding security mistakes developers make: Qualitative analysis from Build It, Break It, Fix It (2020, background): “The original BIBIFI paper [66] explored how different quantitative factors influenced the performance and security of contest submissions.”