BERTese: Learning to Speak to BERT

By Adi Haviv, Jonathan Berant, Amir Globerson. 2021

139 citations recorded by the source corpus.

Large pre-trained language models have been shown to encode large amounts of world and commonsense knowledge in their parameters, leading to substantial interest in methods for extracting that knowledge. In past work, knowledge was extracted by taking manually-authored queries and gathering paraphrases for them using a separate pipeline. In this work, we propose a method for automatically rewriting queries into “BERTese”, a paraphrase query that is directly optimized towards better knowledge extraction. To encourage meaningful rewrites, we add auxiliary loss functions that encourage the query to correspond to actual language tokens. We empirically show our approach outperforms competing baselines, obviating the need for complex pipelines. Moreover, BERTese provides some insight into the type of language that helps language models perform knowledge extraction.

Subfields: Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, background): “Concurrently with this work, other authors explore continuous prompt optimization: Haviv et al. (2021) use an encoder to map a manually written prompt to a sequence of continuous vectors, which are then replaced with the discrete tokens that are nearby in embedding space; Li and Liang (2021) propose”
Learning To Retrieve Prompts for In-Context Learning (2021, methodology): “…et al., 2019; Schick and Schütze, 2021), decoding meth-ods (Min et al., 2021; Zhao et al., 2021; Holtzman et al., 2021), and methods for extracting either hard (Shin et al., 2020; Haviv et al., 2021) or soft (Li and Liang, 2021; Zhong et al., 2021; Qin and Eisner, 2021) prompts automatically.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “…handcrafting for knowledge extraction in LMs (Liu et al., 2021a), as prompts engineered using discrete (Jiang et al., 2020b; Shin et al., 2020; Haviv et al., 2021) and continuous (Zhong et al., 2021; Qin and Eisner, 2021; Liu et al., 2021b) optimization have improved LMs’ lower-bound…”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Common programmatic approaches include mining the templates from an external corpus [108] and paraphrasing with a seed prompt [89, 251].”
Black-box Prompt Learning for Pre-trained Language Models (2022, background): “…learning can be categorized into two kinds: discrete prompt (Wallace et al., 2019; Shin et al., 2020; Jiang et al., 2020; Yuan et al., 2021; Haviv et al., 2021; Gao et al., 2021; Ben-David et al., 2021) and continuous prompt (Zhong et al., 2021; Qin and Eisner, 2021; Hambardzumyan et al.,…”