UniXcoder: Unified Cross-Modal Pre-training for Code Representation

By Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, Jian Yin. Annual Meeting of the Association for Computational Linguistics, 2022

1,023 citations recorded by the source corpus.

Pre-trained models for programming languages have recently demonstrated great success on code intelligence. To support both code-related understanding and generation tasks, recent works attempt to pre-train unified encoder-decoder models. However, such encoder-decoder framework is sub-optimal for auto-regressive tasks, especially code completion that requires a decoder-only manner for efficient inference. In this paper, we present UniXcoder, a unified cross-modal pre-trained model for programming language. The model utilizes mask attention matrices with prefix adapters to control the behavior of the model and leverages cross-modal contents like AST and code comment to enhance code representation. To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task. We evaluate UniXcoder on five code-related tasks over nine datasets. To further evaluate the performance of code fragment representation, we also construct a dataset for a new task, called zero-shot code-to-code search. Results show that our model achieves state-of-the-art performance on most tasks and analysis reveals that comment and AST can both enhance UniXcoder.

View Paper: https://www.semanticscholar.org/paper/4b27f18bff43d605805c92696a979714ced0b805

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yue Wang
2. Steven C. H. Hoi
3. Hung Le
4. Junnan Li
5. Yichen Xu
6. Daya Guo
7. Canwen Xu
8. Yanqiao Zhu

Citations in the corpus

• Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, result): “In line with recent work (Guo et al., 2021, 2022 (Husain et al., 2019) with CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2021), UniXcoder (Guo et al., 2022) and CodeRetriever (Li et al., 2022).”
• A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “…be specific, it is possible to extract the Abstract Syntax Tree (AST), Control-Flow Graphs (CFGs), and Data-Flow Graphs (DFGs) in the preprocessing phase and utilize these structures in code modeling (Peng et al., 2021b; Shiv and Quirk, 2019; Guo et al., 2021; Wang et al., 2021a; Guo et al., 2022).”
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (2023, methodology): “UniXCoder [18] and CugLM [24] incorporates multi-task learning strategies, and leverages code structures to enhance pretraining.”
• A Survey on Large Language Models for Code Generation (2024, unknown): “GitHub Copilot [62] Code Llama [63] ToolGen [64] B. Advanced Code”
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, methodology): “From the results listed in by UniXcoder (Guo et al., 2022) (detailed in Appendix B) and find that the simple sparse retriever achieves equivalent performance, highlighting the robustness of RepoCoder across different code retrieval and generation models.”
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “More related to our work is UniXcoder [Guo et al., 2022], which adopts a UniLM-style design [Dong et al., 2019] and supports various tasks by manipulating input attention masks.”
• LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023, methodology): “To ensure fair comparison with other models, we pre-train LongCoder on the CodeSearchNet dataset using the same next token prediction objective and pre-training setting as baselines (Lu et al., 2021; Guo et al., 2022).”
• Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study (2026, unknown): “For all, we use as a baseline model a UnixCoder (Guo et al., 2022) (125M parameters).”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “Typically, [193] introduced a text encoder enhanced with RoPE [148] and unpadding [125], which is pre-trained by masked language model based on much longer (e.g. 8K tokens) and multilingual context with a two-stage curriculum.”
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval (2024, unknown): “To pursue a more general training approach, we chose to train using a generation model rather than a text retrieval model.”
• RepoNav: From Snippet Retrieval to File-Centered Repository Navigation for Code Agents (2026, unknown): “Snip-pet Search and RepoNav use the same UniXcoder index, GPT-OSS-120B backbone, agent loop, and interaction budget.”