COLD: A Benchmark for Chinese Offensive Language Detection

By Deng Jiawen, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Minlie Huang. 2022

170 citations recorded by the source corpus.

Offensive language detection is increasingly crucial for maintaining a civilized social media platform and deploying pre-trained language models. However, this task in Chinese is still under exploration due to the scarcity of reliable datasets. To this end, we propose a benchmark –COLD for Chinese offensive language analysis, including a Chinese Offensive Language Dataset –COLDATASET and a baseline detector –COLDETECTOR which is trained on the dataset. We show that the COLD benchmark contributes to Chinese offensive language detection which is challenging for existing resources. We then deploy the COLDETECTOR and conduct detailed analyses on popular Chinese pre-trained language models. We first analyze the offensiveness of existing generative models and show that these models inevitably expose varying degrees of offensive issues. Furthermore, we investigate the factors that influence the offensive generations, and we find that anti-bias contents and keywords referring to certain groups or revealing negative attitudes trigger offensive outputs easier.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minlie Huang
2. Xiaoyan Zhu
3. Chujie Zheng
4. Pei Ke
5. Jie Tang
6. Fei Mi
7. Yuxian Gu
8. Hao Sun

In Natural Language Processing

1. Minlie Huang
2. Xiaoyan Zhu
3. Chujie Zheng
4. Zhexin Zhang
5. Hao Sun
6. Pei Ke
7. Jiawen Deng
8. Jiale Cheng

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Minlie Huang
4. Xinwei Wu
5. Tianhao Shen
6. Jiawen Deng
7. Jiale Cheng
8. Jie Tang

Citations in the corpus

• Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “…et al., 2019; Bang et al., 2021; Nadeem et al., 2021), extracted real-world data (Gehman et al., 2020; Sheng et al., 2021; Schick et al., 2021; Deng et al., 2022; Zhou et al., 2022), crowdsourcing workers (Xu et al., 2020), or automatic generation based on pretrained language models (Perez et…”
• Large Language Model Alignment: A Survey (2023, background): “This observation accentuates the importance of rigorous data scrutiny for LLMs. Shifting focus to the Chinese context, COLD (Deng et al., 2022) explores the detection of offensive language in Chinese.”
• EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training (2022, background): “0 is also likely to give offensive, toxic, and biased answers when faced with some “trap contexts” (Deng et al., 2022), which hinders its application.”
• Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, unknown): “…2018; Zhao et al., 2019; Basta et al., 2019; Zhang et al., 2020; Nadeem et al., 2021; Zhou et al., 2021b; Du et al., 2021; Smith et al., 2022), detecting bias in language generation (Gehman et al., 2020; Deng et al., 2022), mitigating the generated bias (Schick et al., 2021; Barikeri et al., 2021).”