UNC Charlotte student Andrei Vince presents AI agent reliability benchmark at international conference in South Korea

Andrei Vince, a fourth-year computer engineering major, was the sole author of ACID-Bench and presented a poster on the benchmark at KDD Workshop on Evaluation and Trustworthiness of Agentic AI in Jeju, South Korea. The benchmark rigorously tests AI agents that use tools to change information inside software systems.

As AI agents are integrated into software systems across industries, ACID-Bench — a system Vince developed independently — seeks to audit the agent beyond its output. It created and tested 52 failure scenarios across retail and airline environments, including misleading errors after successful actions, outdated information, incomplete updates and crash recovery. The benchmark then evaluates both the agent’s actions and the state of the system.

“AI agents can finish a task and still leave a mess behind,” Vince said. “Most benchmarks check whether the agent reached the requested result. ACID-Bench also checks what it did along the way and what state it left the system in.”

In 285 of 780 evaluations, ACID-Bench’s stricter trace-and-state evaluation and the usual task score reached different conclusions on whether the task succeeded. This reinforced the thesis that checking only the final result can miss failures that happened during the process.

Vince spent months building and testing ACID-Bench on his laptop. Going to South Korea allowed him not just to showcase the benchmark but to get feedback from researchers from across the world.

“In Jeju I got the chance to talk about it with researchers from Mexico, China and Russia; it was very different from sitting by myself and debugging it,” Vince said.

Written by: Alex Reynolds