CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models

By Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, Mykola Pechenizkiy. 2023

36 citations recorded by the source corpus.

redWarning: This paper contains content that may be offensive or upsetting.Pretrained conversational agents have been exposed to safety issues, exhibiting a range of stereotypical human biases such as gender bias. However, there are still limited bias categories in current research, and most of them only focus on English. In this paper, we introduce a new Chinese dataset, CHBias, for bias evaluation and mitigation of Chinese conversational language models.Apart from those previous well-explored bias categories, CHBias includes under-explored bias categories, such as ageism and appearance biases, which received less attention. We evaluate two popular pretrained Chinese conversational models, CDial-GPT and EVA2.0, using CHBias. Furthermore, to mitigate different biases, we apply several debiasing methods to the Chinese pretrained models. Experimental results show that these Chinese pretrained models are potentially risky for generating texts that contain social biases, and debiasing methods using the proposed dataset can make response generation less biased while preserving the models’ conversational capabilities.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Zhexin Zhang
2. Hao Sun
3. Jindong Wang
4. Minlie Huang
5. Jiawen Deng
6. Yidong Wang
7. Jiale Cheng
8. Linyi Yang

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Jindong Wang
4. Minlie Huang
5. Jiawen Deng
6. Jiale Cheng

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Similarly, Zhao et al. (2023) gathered data from social media platforms and performed strict data preprocessing.”
• A Survey on Evaluation of Large Language Models (2023, background): “[242] proposes a Chinese conversational bias evaluation dataset CHBias, discovers bias risks in pre-trained models, and explores debiasing methods.”