On Learning Meaningful Assert Statements for Unit Test Cases

By Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, Denys Poshyvanyk. 2020

196 citations recorded by the source corpus.

Software testing is an essential part of the software lifecycle and requires a substantial amount of time and effort. It has been estimated that software developers spend close to 50% of their time on testing the code they write. For these reasons, a long standing goal within the research community is to (partially) automate software testing. While several techniques and tools have been proposed to automatically generate test methods, recent work has criticized the quality and usefulness of the assert statements they generate. Therefore, we employ a Neural Machine Translation (NMT) based approach called Atlas (AuTomatic Learning of Assert Statements) to automatically generate meaningful assert statements for test methods. Given a test method and a focal method (i.e., the main method under test), Atlas can predict a meaningful assert statement to assess the correctness of the focal method. We applied Atlas to thousands of test methods from GitHub projects and it was able to predict the exact assert statement manually written by developers in 31% of the cases when only considering the top-1 predicted assert. When considering the top-5 predicted assert statements, Atlas is able to predict exact matches in 50% of the cases. These promising results hint to the potential usefulness of our approach as (i) a complement to automatic test case generation techniques, and (ii) a code completion support for developers, who can benefit from the recommended assert statements while writing test code.

Subfields: Information Systems, Software, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Code Generation & Program Synthesis

Citations in the corpus

TypeWriter: neural type prediction with search-based validation (2019, background): “Beyond type prediction, others have proposed learning-based techniques to find programming errors [25, 28], predict variable and method names [1, 29, 35], suggest how to improve names [19], search code [10, 31], detect clones [38, 42], classify code [24, 41], predict code edits [33, 40, 43], predict”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Test generation using LLMs has been explored in (Yuan et al., 2023; Sch¨afer et al., 2024; Tufano et al., 2022; Watson et al., 2020).”
Unit Test Case Generation with Transformers (2020, background): “Deep learning techniques have shown the potential of learning from real-world examples, and have been employed in several software engineering tasks, such as code completion [17], automated patch generation [18], [19], comment generation [20], and many others [21].”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “Actually, before using LLMs, researchers have proposed RNN-based approaches that aim at learning from thousands of unit test methods to generate meaningful assert statements [137], yet only 17% of the generated asserts can exactly match with the ground truth asserts.”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “We are aware of several recent research efforts in which LLMs are used for test generation [23], [24], [25], [26], [27], [28], [29].”
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “To address these issues, recent work [16], [17], [18], [19], [20] has leveraged advanced deep learning (DL) techniques, especially large language models (LLMs), to generate unit tests.”