Simple and Efficient ways to Improve REALM

By Vidhisha Balachandran, Ashish Vaswani, Yulia Tsvetkov, Niki Parmar. 2021

5 citations recorded by the source corpus.

Dense retrieval has been shown to be effective for Open Domain Question Answering, surpassing sparse retrieval methods like BM25. One such model, REALM, (Guu et al., 2020) is an end-to-end dense retrieval system that uses MLM based pretraining for improved downstream QA performance. However, the current REALM setup uses limited resources and is not comparable in scale to more recent systems, contributing to its lower performance. Additionally, it relies on noisy supervision for retrieval during fine-tuning. We propose REALM++, where we improve upon the training and inference setups and introduce better supervision signal for improving performance, without any architectural changes. REALM++ achieves ~5.5% absolute accuracy gains over the baseline while being faster to train. It also matches the performance of large models which have 3x more parameters demonstrating the efficiency of our setup.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

Citations in the corpus

• You Only Need One Model for Open-domain Question Answering (2021, neutral): “REALM++ (Balachandran et al., 2021) added a re-ranker to REALM.”