Unit Test Case Generation with Transformers

By Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan. 2020

292 citations recorded by the source corpus.

Automated Unit Test Case generation has been the focus of extensive literature within the research community. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult to read or understand for developers. In this paper we propose AthenaTest, an approach that aims at generating unit test cases by learning from real-world, developer-written test cases. Our approach relies on a state-of-the-art sequence-to-sequence transformer model which is able to write useful test cases for a given method under test (i.e., focal method). We also introduce methods2test - the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 630k test cases mined from 70k open-source repositories hosted on GitHub. We use this dataset to train a transformer model to translate focal methods into the corresponding test cases. We evaluate the ability of our model in generating test cases using natural language processing as well as code-specific criteria. First, we assess the quality of the translation compared to the target test case, then we analyze properties of the test case such as syntactic correctness and number and variety of testing APIs (e.g., asserts). We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated test cases.

Subfields: Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, methodology): “Tufano et al. (2020) use Transformers to generate unit tests for code which outperformed commercial offerings.”
Code Llama: Open Foundation Models for Code (2023, methodology): “…fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a), as well as solving math problems as demonstrated by PaLM (Chowdhery et al., 2022) or Codex…”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
SantaCoder: don't reach for the stars! (2023, neutral): “Evaluating other tasks Code generation models have also been used to solve a variety of tasks (Tufano et al., 2020; Feng et al., 2020b; Ahmed & Devanbu, 2022; Hellendoorn et al., 2018; Pradel et al., 2020).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “These ﬁndings suggest that common practices to improve LLMs may not address the complexities inherent in open-domain code generation problems.”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Future works can consider using models to help humans write useful tests (Tufano et al., 2020), or formally verify the correctness of a predicted solution (Chu et al., 2017).”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, neutral): “The test case generation task [19, 70] is to generate unit test cases automatically.”
CodeT: Code Generation with Generated Tests (2022, methodology): “To combat these limitations, recently proposed approaches (Tufano et al., 2020; Li et al., 2022b) leveraged pre-trained language models like BART (Lewis et al., 2019) and T5 (Raffel et al., 2020) ﬁne-tuned on labelled data for test case generation.”
A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages (2022, background): “Code-aware language models have been applied to a variety of tasks, including test generation (Tufano et al. 2020), doc-string generation (Lu et al. 2021), code search (Feng et al. 2020; Ahmed and Devanbu 2022), type inference (Wei et al. 2020; Hellendoorn et al. 2018; Pradel et al. 2020), and more…”
METHODS2TEST: A dataset of focal methods mapped to test cases (2022, methodology): “We initially build this dataset in conjunction with our previous work [26], where we proposed AthenaTest, an EncoderDecoder Transformer model trained to translate a focal method (with focal context) into the corresponding test case.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “Tufano et al. [25] utilized a similar schema by pre-training the LLM on a large unsupervised Java corpus, and supervised ﬁne-tuning a downstream translation task for generating unit tests.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, methodology): “Code generation models have been applied to a variety of tasks, including test generation [19], docstring generation [20], code search [17, 21], type inference [22, 23, 24], and more [25].”
Planning with Large Language Models for Code Generation (2023, methodology): “Tufano et al. (2020a;b) propose to generate test cases and corresponding accurate assert statements with Transformer models (Vaswani et al., 2017a Planning and reinforcement learning for code generation.”
The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development (2023, methodology): “Codefluent large language models are capable of generating code from natural language descriptions [105], translating code from one language to another [75], generating unit tests [92], and even generating documentation for code [36, 38, 97, 98].”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, unknown): “Similarly, Tufano et al.’s A THENA T EST [26] generates tests using a BART transformer model [44] ﬁne-tuned on a training set of functions and their corresponding tests.”
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction (2022, background): “We find that providing constructors (à la AthenaTest [28]) does not help significantly, but adding stack traces does help reproduce crash bugs, indicating that LIBRO can benefit from using the stack information to replicate issues more accurately.”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation (2023, unknown): “Previous research on unit test generation has employed evolutionary algorithms, leading to tools such as EvoSuite [15] and Randoop [27], and several notable machine learning models [47, 3, 19, 13, 40, 39, 49, 51, 12, 28, 14, 34, 18].”
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “Since AthenaTest has not released its pre-trained BART-based model or its ﬁne-tuned model, we reproduce it according to its paper by ﬁne-tuning the widely-used LLM CodeT5 on the same ﬁne-tuning dataset used in AthenaTest.”