MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval
By Lila Boualili, Jose G. Moreno, Mohand Boughanem. 2020
28 citations recorded by the source corpus.
The Information Retrieval (IR) community has witnessed a flourishing development of deep neural networks, however, only a few managed to beat strong baselines. Among them, models like DRMM and DUET were able to achieve better results thanks to the proper handling of exact match signals. Nowadays, the application of pre-trained language models to IR tasks has achieved impressive results exceeding all previous work. In this paper, we assume that established IR cues like exact term-matching, proven to be valuable for deep neural models, can be used to augment the direct supervision from labeled data for training these pre-trained models. To study the effectiveness of this assumption, we propose MarkedBERT a modified version of one of the most popular pre-trained models via language modeling tasks, BERT. MarkedBERT integrates exact match signals using a marking technique that locates and highlights Exact Matched query-document terms using marker tokens. Experiments on MS MARCO Passage Ranking task show that our rather simple approach is actually effective. We find that augmenting the input with marker tokens allows the model to focus on valuable text sequences for IR.
Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jimmy J. Lin
- Andrew Yates
- Rodrigo Nogueira
- Zeynep Akkalyoncu Yilmaz
- Sebastian Hofstätter
- Allan Hanbury
- Zhuyun Dai
- Jamie Callan
In Information Systems
- Rodrigo Nogueira
- Andrew Yates
- Jimmy J. Lin
- Sebastian Hofstätter
- Allan Hanbury
- Sean MacAvaney
- Jamie Callan
- Nazli Goharian
In Information Retrieval & Search
- Jimmy J. Lin
- Rodrigo Nogueira
- Andrew Yates
- Zeynep Akkalyoncu Yilmaz
- Sebastian Hofstätter
- Allan Hanbury
- Zhuyun Dai
- Wei Yang
In Large Language Models
- Jimmy J. Lin
- Andrew Yates
- Rodrigo Nogueira
- Zhuyun Dai
- Jamie Callan
- Zeynep Akkalyoncu Yilmaz
- Sebastian Hofstätter
- Allan Hanbury
Citations in the corpus
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “their relative positions, if the input sequence lacks this signal, then 77There is, however, recent work suggesting that BERT might benefit from explicit exact match cues conveyed using marker tokens [Boualili et al., 2020]. However, the authors reported results only with BERT base (with MRR@10 score”