Read before Generate! Faithful Long Form Question Answering with Machine Reading

By Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung. 2022

77 citations recorded by the source corpus.

Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models answer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient information which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed analysis further proves the competency of our methods in generating fluent, relevant, and more faithful answers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Bryan Catanzaro
3. Andrea Madotto
4. Dan Su
5. Wei Ping
6. Wenliang Dai
7. Mohammad Shoeybi
8. Zihan Liu

In Computer Vision and Pattern Recognition

1. Pascale Fung
2. Wenliang Dai
3. Ziwei Ji
4. Dan Su
5. Mohammad Shoeybi
6. M. Patwary
7. Bryan Catanzaro
8. Zihan Liu

In Retrieval-Augmented Generation

1. Daniel Khashabi

In Natural Language Processing

1. Pascale Fung
2. Andrea Madotto
3. Nayeon Lee
4. Daniel Khashabi
5. Zihan Liu
6. Bryan Catanzaro
7. Etsuko Ishii
8. Yejin Bang

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, methodology): “Normally GQA system involves searching an external knowledge source for information relevant to the question.”
• “According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, methodology): “For retrieval-augmented SOTA, we show Izacard et al. (2022) for NQ, TriviaQA and Hot-potQA, and Su et al. (2022) for ELI5.”
• Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, background): “Su et al. (2022) augment the generation model with fine-grained, answer-related salient information predicted by a machine reading comprehension module, to reduce hallucination in generative question answer task.”
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
• Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “…GPT-3 In previous studies, different metrics and methods have been proposed to measure and improve the factual accuracy of language generation within different tasks [11], including text summarization [e.g., 12–15], question answering [e.g., 16–18], and table-to-text generation [e.g., 19, 20].”