Benchmarking Meaning Representations in Neural Semantic Parsing
By Jiaqi Guo, Qian Liu, Jian-Guang Lou, Zhenwen Li, Xueqing Liu, Tao Xie, Ting Liu. 2020
29 citations recorded by the source corpus.
Meaning representation is an important component of semantic parsing. Although researchers have designed a lot of meaning representations, recent work focuses on only a few of them. Thus, the impact of meaning representation on semantic parsing is less understood. Furthermore, existing work’s performance is often not comprehensively evaluated due to the lack of readily-available execution engines. Upon identifying these gaps, we propose , a new unified benchmark on meaning representations, by integrating existing semantic parsing datasets, completing the missing logical forms, and implementing the missing execution engines. The resulting unified benchmark contains the complete enumeration of logical forms and execution engines over three datasets × four meaning representations. A thorough experimental study on Unimer reveals that neural semantic parsing approaches exhibit notably different performance when they are trained to generate different meaning representations. Also, program alias and grammar rules heavily impact the performance of different meaning representations. Our benchmark, execution engines and implementation can be found on: https://github.com/JasperGuo/Unimer.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Percy Liang
- Caiming Xiong
- Mohit Bansal
- Jonathan Berant
- Graham Neubig
- Hannaneh Hajishirzi
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luke S. Zettlemoyer
- Caiming Xiong
- Mohit Bansal
- Hannaneh Hajishirzi
- Luc Van Gool
- Panupong Pasupat
- Jonathan Berant
- Percy Liang
In Evaluation & Benchmarks
- Caiming Xiong
- Luke S. Zettlemoyer
- Ming-Wei Chang
- Dragomir R. Radev
- Mari Ostendorf
- Yuan Zhang
- Kelvin Guu
- Yushi Hu
In Information Extraction
- Jonathan Herzig
- Panupong Pasupat
- Peter Shaw
- Yuan Zhang
- Ming-Wei Chang
- Luke S. Zettlemoyer
- Kelvin Guu
- Caiming Xiong
Citations in the corpus
- Binding Language Models in Symbolic Languages (2022, background): “The gap between Python and SQL performance on W IKI TQ may lie in the properties of each programming language, as suggested by Guo et al. (2020).”
- Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, background): “Guo et al. (2020a) compares performance of semantic parsers across several such formalisms as well as SQL.”