Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations

By Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, Yuan Zhang. 2021

72 citations recorded by the source corpus.

Sequence-to-sequence (seq2seq) models are prevalent in semantic parsing, but have been found to struggle at out-of-distribution compositional generalization. While specialized model architectures and pre-training of seq2seq models have been proposed to address this issue, the former often comes at the cost of generality and the latter only shows limited success. In this paper, we study the impact of intermediate representations on compositional generalization in pre-trained seq2seq models, without changing the model architecture at all, and identify key aspects for designing effective representations. Instead of training to directly map natural language to an executable form, we map to a reversible or lossy intermediate representation that has stronger structural correspondence with natural language. The combination of our proposed intermediate representations and pre-trained models is surprisingly effective, where the best combinations obtain a new state-of-the-art on CFQ (+14.8 accuracy points) and on the template-splits of three text-to-SQL datasets (+15.0 to +19.4 accuracy points). This work highlights that intermediate representations provide an important and potentially overlooked degree of freedom for improving the compositional generalization abilities of pre-trained seq2seq models.

Subfields: Artificial Intelligence, Molecular Biology, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Large Language Models

In Natural Language Processing

Citations in the corpus

Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, background): “Herzig et al. (2021) solve text-to-SQL tasks with intermediate text representations.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, background): “Herzig et al. (2021) predict intermediate semantic representations with stronger structural correspondence to natural language than m , replacing the role of c in our approach with a modiﬁed meaning representation m (cid:48) .”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “Prior work has shown the potential of leveraging alternative output representations to improve semantic parsers (Herzig et al., 2021; Shin et al., 2021).”
Compositional Semantic Parsing with Large Language Models (2022, methodology): “Yet others exploit correlations between the input and output tokens, e.g. through loss functions with attention supervision (Yin et al., 2021), using a lexicon (Akyürek & Andreas, 2021), and reformulating the label space (Herzig et al., 2021).”