CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility

By Guohai Xu, Jiayi Liu, Mingshi Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang and 4 more. 2023

108 citations recorded by the source corpus.

With the rapid evolution of large language models (LLMs), there is a growing concern that they may pose risks or have negative social impacts. Therefore, evaluation of human values alignment is becoming increasingly important. Previous work mainly focuses on assessing the performance of LLMs on certain knowledge and reasoning abilities, while neglecting the alignment to human values, especially in a Chinese context. In this paper, we present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs in terms of both safety and responsibility criteria. As a result, we have manually collected adversarial safety prompts across 10 scenarios and induced responsibility prompts from 8 domains by professional experts. To provide a comprehensive values evaluation of Chinese LLMs, we not only conduct human evaluation for reliable comparison, but also construct multi-choice prompts for automatic evaluation. Our findings suggest that while most Chinese LLMs perform well in terms of safety, there is considerable room for improvement in terms of responsibility. Moreover, both the automatic and human evaluation are important for assessing the human values alignment in different aspects. The benchmark and code is available on ModelScope and Github.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hao Sun
2. Zhexin Zhang
3. Jindong Wang
4. Xingxu Xie
5. Minlie Huang
6. Hao Chen
7. Jiawen Deng
8. Yidong Wang

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Jindong Wang
4. Minlie Huang
5. Jiawen Deng
6. Jiale Cheng

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “We examine the main meth-ods employed in LMs safety evaluation, including model preference-based safety testing (Nadeem et al., 2021; Xu et al., 2023a), adversarial safety attacks (Perez et al., 2022; Ganguli et al., 2022), safety risk detection (Dinan et al., 2019; Sun et al., 2022), and other…”
• Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “For example, CValues (Xu et al. 2023a) focuses on the safety and responsibility of LLMs, M3Exam (Zhang et al. 2023) focuses on human exam and ToolBench (Qin et al. 2023) evaluates how well LLMs use external tools.”
• A Survey on Evaluation of Large Language Models (2023, background): “This study emphasizes the importance of [21] Social knowledge Specific downstream task Social language understanding MME [43] Multimodal LLMs Multi-modal task Ability of perception and cognition Xiezhi [55] Comprehensive domain knowledge General language task Overall performance across multiple benc”