Papers by Qi GU
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation (2026.findings-acl)
Copied to clipboard
Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi GU, Hui Su, Xunliang Cai, Xiangnan He
| Challenge: | Existing approaches to verify agent behaviors in complex environments rely on rule-based verifiers or LLM-as-a-Judge models. |
| Approach: | They propose a benchmark to evaluate Agent-as-a-Judge across three domains . the benchmark covers search, data systems, and graphical user interfaces - with 155 tasks and 516 trajectories . |
| Outcome: | The proposed benchmark outperforms existing benchmarks in search, data systems, and GUI domains while revealing open challenges in agent-based verification. |