Beyond [CLS] through Ranking by Generation

By C. D. Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, Bing Xiang. 2020

33 citations recorded by the source corpus.

Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document's language model, were very successful in various IR tasks in the past. However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead. Recently, deep generative models such as GPT2 and BART have been shown to be excellent text generators, but their effectiveness as rankers have not been demonstrated yet. In this work, we revisit the generative framework for information retrieval and show that our generative approaches are as effective as state-of-the-art semantic similarity-based discriminative models for the answer selection task. Additionally, we demonstrate the effectiveness of unlikelihood losses for IR.

Subfields: Information Systems, Information Retrieval & Search, Large Language Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Wen-tau Yih
2. Luke S. Zettlemoyer
3. Kyunghyun Cho
4. Hannaneh Hajishirzi
5. Mandar Joshi
6. Joelle Pineau
7. Yejin Choi
8. Sean Welleck

In Information Retrieval & Search

1. Hannaneh Hajishirzi
2. Wen-tau Yih
3. Luke S. Zettlemoyer
4. Michael Lewis
5. Sebastian Riedel
6. Danqi Chen
7. Kyunghyun Cho
8. Armen Aghajanyan

In Large Language Models

1. Yejin Choi
2. Luke S. Zettlemoyer
3. Hannaneh Hajishirzi
4. Wen-tau Yih
5. Armen Aghajanyan
6. Michael Lewis
7. Ronan Le Bras
8. Kyunghyun Cho

In Signal Processing

1. Wen-tau Yih
2. Joelle Pineau
3. Yejin Choi
4. Kyunghyun Cho
5. Luke S. Zettlemoyer
6. Hannaneh Hajishirzi
7. Michael Lewis
8. Devendra Singh Sachan

Citations in the corpus

• Improving Passage Retrieval with Zero-Shot Question Generation (2022, methodology): “More recent work on re-rankers have fine-tuned PLMs on question-passage pairs to generate relevance labels (Nogueira et al., 2020), some-times to jointly generate question and relevance labels (Nogueira dos Santos et al., 2020; Ju et al., 2021).”
• NaturalProofs: Mathematical Theorem Proving in Natural Language (2021, background): “Recent work suggests that these models store knowledge in their parameters [29], are capable of reasoning in mathematical [31, 47] and language [6, 38] domains, and are effective for information retrieval tasks [27, 28].”