Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs

By Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, Xavier Amatriain. 2020

85 citations recorded by the source corpus.

People increasingly search online for answers to their medical questions but the rate at which medical questions are asked online significantly exceeds the capacity of qualified people to answer them. This leaves many questions unanswered or inadequately answered. Many of these questions are not unique, and reliable identification of similar questions would enable more efficient and effective question answering schema. COVID-19 has only exacerbated this problem. Almost every government agency and healthcare organization has tried to meet the informational need of users by building online FAQs, but there is no way for people to ask their question and know if it is answered on one of these pages. While many research efforts have focused on the problem of general question similarity, these approaches do not generalize well to domains that require expert knowledge to determine semantic similarity, such as the medical domain. In this paper, we show how a double fine-tuning approach of pretraining a neural network on medical question-answer pairs followed by fine-tuning on medical question-question pairs is a particularly useful intermediate task for the ultimate goal of determining medical question similarity. While other pretraining tasks yield an accuracy below 78.7% on this task, our model achieves an accuracy of 82.6% with the same number of training examples, an accuracy of 80.0% with a much smaller training set, and an accuracy of 84.5% when the full corpus of medical question-answer data is used. We also describe a currently live system that uses the trained model to match user questions to COVID-related FAQs.

Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiang Ren
2. Bill Yuchen Lin
3. Qinyuan Ye
4. Sang-goo Lee
5. Kang Min Yoo
6. Sang-Woo Lee
7. Taeuk Kim
8. Mohit Bansal

In Information Systems

1. Xiang Ren
2. Sang-goo Lee
3. Kang Min Yoo
4. Bill Yuchen Lin

In Question Answering & Reading Comprehension

1. Xiang Ren
2. Bill Yuchen Lin

Citations in the corpus

• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “We include 26 datasets as follows: fi-nancial_phrasebank (Malo et al., 2014), poem_sentiment (Sheng and Uthus, 2020), medical_questions_pairs (McCreery et al., 2020), glue-mrpc (Dolan and Brockett, 2005), glue-wnli (Levesque et al., 2012), climate_fever (Diggel-mann et al., 2020), glue-rte (Dagan et…”
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…other Manotas et al. 2020 math_qa qa/multiple-choice qa Amini et al. 2019 mc_taco qa/binary Zhou et al. 2019 medical_questions_pairs cls/paraphrase McCreery et al. 2020 mocha other/regression Chen et al. 2020a multi_news cg/summarization Fabbri et al. 2019 numer_sense qa/closed-book qa Lin et al.…”
• Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022, methodology): “…Tice, 2000) 5,452 500 5 financial_phrasebank (Malo et al., 2014) 1,181 453 3 poem_sentiment (Sheng and Uthus, 2020) 843 105 3 medical_questions_pairs (McCreery et al., 2020) 2,438 610 2 sick (Marelli et al., 2014) 4,439 495 3 hate_speech18 (de Gibert et al., 2018) 8,562 2,141 4…”