Identifying the Risks of LM Agents with an LM-Emulated Sandbox
By Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, Tatsunori B. Hashimoto. International Conference on Learning Representations, 2023
440 citations recorded by the source corpus.
Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, setting up the environment for each test scenario manually, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios, without manual instantiation. Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks. We test both the tool emulator and evaluator through human evaluation and find that 68.8% of failures identified with ToolEmu would be valid real-world agent failures. Using our curated initial benchmark consisting of 36 high-stakes tools and 144 test cases, we provide a quantitative risk analysis of current LM agents and identify numerous failures with potentially severe outcomes. Notably, even the safest LM agent exhibits such failures 23.9% of the time according to our evaluator, underscoring the need to develop safer LM agents for real-world deployment.
View Paper: https://www.semanticscholar.org/paper/0a893657e00fe8ecfadcc65c63bf293e70cb1564
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Christopher Rawles
- Sarah Clinckemaillie
- Daniel Toyama
- Yifan Chang
- Noah Shinn
- J. Waltz
- G. Lau
- Timothy P. Lillicrap
Citations in the corpus
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “In our experiments, we adopt the ReAct prompt as employed by Ruan et al. (2023) prompt includes a requirement for the safety and security of tool calls, instructing the agent to refrain from executing tools that could be harmful to users.”
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “Effective evaluation of autonomous agents requires environments that not only mimic real-world scenarios but also provide immediate reward signals upon successful task completion [41, 9, 3, 42, 6].”
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024, methodology): “Existing benchmarks [27, 29, 12, 14, 16] for language agents often feature simplified instruction-following setups, where the agent autonomously interacts with an environment (web, code terminal, or APIs) given all the information upfront, without any human-in-the-loop interaction and without the…”
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “The ToolEmu benchmark (Ruan et al., 2023) focuses on scenarios where the underlying user intent is assumed to be benign rather than malicious and there is no intention to direct the LM agent towards causing harm.”
- Tool learning with large language models: a survey (2024, unknown): “1 A detailed list of different benchmarks and their specific configurations.”
- τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025, unknown): “Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”