Fuzz4ALL: Universal Fuzzing with Large Language Models

By Chun Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, Lingming Zhang. International Conference on Software Engineering, 2023

326 citations recorded by the source corpus.

Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4ALL, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4ALL is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are well- suited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4ALL on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java, and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4ALL has identified 98 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 64 bugs already confirmed by developers as previously unknown.

Subfields: Computer Networks and Communications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/9a68e0b4dbc81d8ee1d24c367f555623758cec00

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

Citations in the corpus

Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Xia et al. [103] recently demonstrate that LLMs can serve as a universal fuzzer for systems across different application domains and programing languages, including C/C++ compilers, JDK, SMT solvers, and even quantum computing systems.”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, background): “(2018), testing (Kang et al., 2023; Xia et al., 2023; Wang et al., 2023), and program repair (Monperrus, 2018; Gupta et al.”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “LLMs have demonstrated state-of-the-art performance on various code-related tasks, including code generation [11, 33, 25], program repair [69, 27, 68, 65], automated testing [15, 14, 67, 35, 71], code translation [31, 55] and code summarization [1, 37].”
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair (2024, unknown): “…for code generation and code editing: Beyond program repair, LLMs have been applied to a variety of other code generation and code editing tasks, including code completion [35], [59], fuzzing [60], generating and improving unit tests [61], [62], [63], [64], [65], [66], multi-step code editing [67].”