Natural- to formal-language generation using Tensor Product Representations

By Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, Jianfeng Gao. 2019

3 citations recorded by the source corpus.

Generating formal-language represented by relational tuples, such as Lisp programs or mathematical expressions, from a natural-language input is an extremely challenging task because it requires to explicitly capture discrete symbolic structural information from the input to generate the output. Most state-of-the-art neural sequence models do not explicitly capture such structure information, and thus do not perform well on these tasks. In this paper, we propose a new encoder-decoder model based on Tensor Product Representations (TPRs) for Natural- to Formal-language generation, called TP-N2F. The encoder of TP-N2F employs TPR 'binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR 'unbinding' to generate a sequence of relational tuples, each consisting of a relation (or operation) and a number of arguments, in symbolic space. TP-N2F considerably outperforms LSTM-based Seq2Seq models, creating a new state of the art results on two benchmarks: the MathQA dataset for math problem solving, and the AlgoList dataset for program synthesis. Ablation studies show that improvements are mainly attributed to the use of TPRs in both the encoder and decoder to explicitly capture relational structure information for symbolic reasoning.

Subfields: Hardware and Architecture, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Xiaodong Liu
2. Jianfeng Gao
3. Weizhu Chen
4. Lingfei Wu
5. Pengcheng He
6. Xiaodong He
7. Li Deng
8. Wen-tau Yih

In Software

1. Jianfeng Gao

In Natural Language Processing

1. Jianfeng Gao
2. Xiaodong Liu
3. Weizhu Chen
4. Pengcheng He
5. Lingfei Wu
6. Caiming Xiong
7. Richard Socher
8. Fengyuan Xu

Citations in the corpus

• DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “(1)In this sense, our model shares some similarity to Tensor Product Representation [23, 24, 25] where a word is represented using a tensor product of its filler (content) vector and its role (position) vector.”
• Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem (2020, methodology): “Researchers recently attempted to apply the Graph Neural Networks (GNNs) in various NLP tasks, including the neural machine translation (Bastings et al., 2017; Beck et al., 2018), conversational machine reading comprehension (Chen et al., 2019), and AMR-to-text (Song et al., 2018).”