Leveraging Automated Unit Tests for Unsupervised Code Translation

By Baptiste Rozière, Jie M. Zhang, François Charton, Mark Harman, Gabriel Synnaeve, Guillaume Lample. 2021

168 citations recorded by the source corpus.

With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine translation approaches rely on back-translation, a method developed in the context of natural language translation and one that inherently involves training on noisy inputs. Unfortunately, source code is highly sensitive to small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence. To address this issue, we propose to leverage an automated unit-testing system to filter out invalid translations, thereby creating a fully tested parallel corpus. We found that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated, comfortably outperforming the state-of-the-art for all language pairs studied. In particular, for Java $\to$ Python and Python $\to$ C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.

Subfields: Information Systems, Software, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Code Generation & Program Synthesis

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, methodology): “These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
CodeT: Code Generation with Generated Tests (2022, methodology): “This execution-guided approach has been widely adopted in various code-related tasks, such as code generation (Chen et al., 2021; Li et al., 2022b; Shi et al., 2022), code translation (Roziere et al., 2021), and program synthesis (Chen et al., 2018; Ellis et al., 2019).”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We refer to this as the repository structure format , which begins with the root folder of the repository and organizes code files or folder names.”
Language Models Can Teach Themselves to Program Better (2022, background): “Data augmentation is not new to code generation as multiple works have synthesized tests for human-authored code [e.g. Li et al., 2022; Roziere et al., 2021].”
Planning with Large Language Models for Code Generation (2023, methodology): “Transformer has achieved significant improvements on the benchmarking tasks of code translation (Roziere et al., 2022), code completion (Chen et al.”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “LLMs have demonstrated state-ofthe-art performance on various code-related tasks, including code generation/synthesis [11, 28, 24], program repair [58, 59, 57], code translation [27, 45] and code summarization [1, 30].”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “…the level of a statement (Huang et al., 2022) or function (Husain et al., 2019), focusing on only one or two speciﬁc tasks such as code translation (Roziere et al., 2021a) and summarization (Lu et al., 2021), and in many cases without proper ﬁne-tuning or training data (Austin et al., 2021) or in…”
Code Translation with Compiler Representations (2022, methodology): “Later, DOBF (Lachaux et al., 2021) improved the model pre-training method used in TransCoder, and Roziere et al. (2022) used automatically generated unit tests to improve translation performance between Java, C++ and Python.”
Teaching Large Language Models to Self-Debug (2023, background): “, 2023), and performing selection based on the consensus on unit test execution outputs among samples (Chen et al., 2019; 2021b; Roziere et al., 2022; Shi et al., 2022; Li et al., 2022; Chen et al., 2023b).”
LLMs Corrupt Your Documents When You Delegate (2026, unknown): “…applied in other domains, for instance to the code domain, where it was leveraged to train unsupervised code translation models across programming languages (Lachaux et al., 2020; Rozière et al., 2021), and to jointly train code generation and summarization as dual tasks (Wei et al., 2019).”