JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice (2026.acl-long)
Copied to clipboard
Ziang Chen, Guannan Li, Fanlin Ji, Yipeng Kang, Jiaqi Li, Muhan Zhang, Yangtao Zhang, Li Tianjiao, Jiannan Wang, Xin Guo, Song-Chun Zhu, Bin Ling
| Challenge: | Existing legal benchmarks evaluate isolated tasks or exam-style questions, failing to capture the procedural interdependencies and adjudicative rigor inherent in professional practice. |
| Approach: | They propose a vertical, depth-oriented, domain-specific benchmark to evaluate Large Language Models (LLMs) in Chinese civil litigation. |
| Outcome: | The proposed benchmarks show that large language models exhibit an "illusion of competence" the results highlight a critical gap between fluent linguistic output and judicial reliability . |
Similar Papers
LawBench: Benchmarking Legal Knowledge of Large Language Models (2024.emnlp-main)
Copied to clipboard
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, Vincent Ng
| Challenge: | LegalBench evaluated 20 LLMs in 162 legal tasks in 20 countries and jurisdictions. |
| Approach: | They present a comprehensive evaluation of 21 popular Large Language Models and the first comparative analysis of the empirical results. |
| Outcome: | The proposed benchmarks are based on the Bloom’s cognitive taxonomy and are compared to 21 popular LLMs. |
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice (2026.acl-long)
Copied to clipboard
Yuzhen Shi, Huanghai Liu, Yiran HU, Song Gaojie, Xu Xinran, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Feng Di, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Qiu Yuanyang, Yuemeng Qi, Zhang Jingwen, Sui Xiaoyu, Yifan Chen, Zhang Yi, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, HU Wei
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice (2026.findings-acl)
Copied to clipboard
Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin, Min Yang
| Challenge: | Existing datasets and benchmarks focus only on patents or cover limited aspects of the IP field, lacking alignment with real-world scenarios. |
| Approach: | They propose a bilingual IP task taxonomy and a large-scale bilingual benchmark to evaluate LLMs in real-world IP practice. |
| Outcome: | The proposed model achieves only 75.8% accuracy, indicating room for improvement . open-source IP and law-oriented models lag behind closed-source general-purpose models . |
LegalAgentBench: Evaluating LLM Agents in Legal Domain (2025.acl-long)
Copied to clipboard
Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, Wuyue Wang, Yiqun Liu, Minlie Huang
| Challenge: | Existing general-domain benchmarks do not capture complexity of real-world judicial cognition and decision-making. |
| Approach: | They propose a benchmark specifically designed to evaluate LLM Agents in the legal domain. |
| Outcome: | The proposed benchmark includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. |
Pub-LawBench: Public-Oriented Benchmarking for LegalAI (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on legal professionals, not legal professionals. |
| Approach: | They propose a public-oriented LegalAI benchmark grounded in legal functionalism and genre analysis to address this gap. |
| Outcome: | The proposed model evaluates 17 large language models on Pub-LawBench using simple prompts and Chain-of-Thought under a vanilla inference setting. |
UCL-Bench: A Chinese User-Centric Legal Benchmark for Large Language Models (2025.findings-naacl)
Copied to clipboard
Ruoli Gan, Duanyu Feng, Chen Zhang, Zhihang Lin, Haochen Jia, Hao Wang, Zhenyang Cai, Lei Cui, Qianqian Xie, Jimin Huang, Benyou Wang
| Challenge: | Existing legal benchmarks focusing on knowledge and logic evaluate LLMs on various tasks in legal domain, but few have explored the practical application of LLM by actual users. |
| Approach: | They propose a Chinese user-centric legal benchmark that aims to assess the practical application of LLMs by real users. |
| Outcome: | The proposed model outperforms existing models on various tasks in legal domain but does not outperfect ChatGPT. |
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022.acl-long)
Copied to clipboard
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras
| Challenge: | Laws and their interpretations, legal arguments and agreements are typically expressed in writing. |
| Approach: | They propose a benchmark to evaluate model performance across legal NLU tasks . they also evaluate several generic and legal-oriented models . |
| Outcome: | The proposed model performs better across multiple tasks than previous models. |
LAiW: A Chinese Legal Large Language Models Benchmark (2025.coling-main)
Copied to clipboard
Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, Hao Wang
| Challenge: | Xie et al., 2023) show that large language models (LLMs) can generate legal text, but lack the legal syllogism . legal experts are cautious about their practical application due to the opaque nature of the LLMs. |
| Approach: | They propose a Chinese legal LLM benchmark structured around the legal syllogism . they evaluate LLMs across three levels of capability, each reflecting a more complex stage of legal . |
| Outcome: | The proposed benchmark identifies that LLMs lack the legal syllogism, which hinders trust and understanding from legal experts. |
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence (2026.findings-acl)
Copied to clipboard
Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, Erik Cambria
| Challenge: | Existing benchmarks for legal general intelligence (GI) are result-oriented and do not evaluate the legal intelligence of large language models (LLMs). |
| Approach: | They propose a Chinese legal benchmark for evaluating legal GI in large language models . they use recent legal cases and exam questions to create multiple-choice questions . |
| Outcome: | The proposed benchmarks lack a systematic evaluation of the legal intelligence of large language models (LLMs) the results show that even the best LLMs lagging behind human legal professionals. |