BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
By Nora Kassner, Hinrich Schütze. 2020
72 citations recorded by the source corpus.
Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance. We show that this idea is beneficial for open-domain question answering (QA). To improve the recall of facts encountered during training, we combine BERT (Devlin et al., 2019) with a traditional information retrieval step (IR) and a kNN search over a large datastore of an embedded text collection. Our contributions are as follows: i) BERT-kNN outperforms BERT on cloze-style QA by large margins without any further training. ii) We show that BERT often identifies the correct response category (e.g., US city), but only kNN recovers the factually correct answer (e.g.,“Miami”). iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN can easily handle facts not covered by BERT’s training set, e.g., recent events.
Subfields: Artificial Intelligence, Information Retrieval & Search, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lawrence Carin
- Dinghan Shen
- Jianfeng Gao
- Yizhe Zhang
- Peter Clark
- Dani Yogatama
- Chunyuan Li
- Weizhu Chen
In Information Retrieval & Search
- Cyprien de Masson d'Autume
- Lawrence Carin
- Dani Yogatama
- Dinghan Shen
- Weizhu Chen
- Bhavana Dalvi
- Lingpeng Kong
- Peter Clark
In Large Language Models
- Cyprien de Masson d'Autume
- Lawrence Carin
- Dani Yogatama
- Dinghan Shen
- Weizhu Chen
- Yizhe Zhang
- Lingpeng Kong
- Peter Clark
Citations in the corpus
- Adaptive Semiparametric Language Models (2021, methodology): “In contrast to previous language models that either interpolate output probabilities (Merity et al., 2017; Grave et al., 2017c; Khandelwal et al., 2020; Kassner and Schutze, 2020) or use input concatenation (Guu et al.”
- What Makes Good In-Context Examples for GPT-3? (2021, background): “, 2020), and BERT-kNN (Kassner and Schütze, 2020) generate the next token by retrieving the nearest k neighbors from the datastore.”
- Modifying Memories in Transformer Models (2020, neutral): “This line of work is further supported by the recent efforts on adding the k -NN component to language models to improve their generalization via memorization (Kassner and Sch¨utze, 2020; Khandelwal et al., 2020).”
- DREAM: Uncovering Mental Models behind Language Models (2021, background): “These additional, preliminary results suggest additional exciting opportunities for building interpretable nearest-neighbor models (Khandelwal et al., 2019; Kassner and Schütze, 2020) that can use and adapt old experiences (mental models) to understand and solve new problems.”