VerilogEval: Evaluating Large Language Models for Verilog Code Generation
By Mingjie Liu, Nathaniel Ross Pinckney, Brucek Khailany, Haoxing Ren. arXiv.org, 2023
411 citations recorded by the source corpus.
The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering
View Paper: https://www.semanticscholar.org/paper/cbd81507197e2d32b6f6c8ac99039f3a607ee8f1
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Siddharth Garg
- Brendan Dolan-Gavitt
- Bryan Catanzaro
- Benjamin Tan
- Ramesh Karri
- Robert Kirby
- Baleegh Ahmad
- Brucek Khailany
In Electrical and Electronic Engineering
Citations in the corpus
- ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “• Code: We report average pass-rate of coding benchmarks with greedy decoding, including HumanEval [23], VerilogEval-Machine [12], and VerilogEval-Human [12].”
- VeriGen: A Large Language Model for Verilog Code Generation (2023, methodology): “Problem Set II significantly expands on this initial set by integrating an extensive range of problems from HDLBits [19, 34].”
- VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validation (2025, unknown): “5-14B-Instruct using our curated dataset and evaluate it on two established RTL code generation benchmarks: VerilogEval [9] and RTLLM [10].”