A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences (2025.findings-acl)
Copied to clipboard
Jiaxin Shen, Jinan Xu, Huiqi Hu, Luyi Lin, Guoyang Ma, Fei Zheng, Fandong Meng, Jie Zhou, Wenjuan Han
| Challenge: | a recent study focuses on generating impartial and interpretable judicial judgments based on established criminal fact. |
| Approach: | They propose a law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience that enables public scrutiny and preventing bias. |
| Outcome: | The proposed schema enables public scrutiny and prevents bias in the "Intelligent Court" it employs a suite of legal analysis tools to address the challenge task. |
Similar Papers
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that Legal-R1 delivers competitive performance across diverse tasks. |
| Approach: | They propose to evaluate 12 large language models across 17 legal tasks across statutory and case-law traditions to determine their general reasoning performance. |
| Outcome: | The proposed model performs well across 17 legal tasks across statutory and case-law traditions. |
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice (2026.acl-long)
Copied to clipboard
Yuzhen Shi, Huanghai Liu, Yiran HU, Song Gaojie, Xu Xinran, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Feng Di, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Qiu Yuanyang, Yuemeng Qi, Zhang Jingwen, Sui Xiaoyu, Yifan Chen, Zhang Yi, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, HU Wei
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice (2026.acl-long)
Copied to clipboard
Ziang Chen, Guannan Li, Fanlin Ji, Yipeng Kang, Jiaqi Li, Muhan Zhang, Yangtao Zhang, Li Tianjiao, Jiannan Wang, Xin Guo, Song-Chun Zhu, Bin Ling
| Challenge: | Existing legal benchmarks evaluate isolated tasks or exam-style questions, failing to capture the procedural interdependencies and adjudicative rigor inherent in professional practice. |
| Approach: | They propose a vertical, depth-oriented, domain-specific benchmark to evaluate Large Language Models (LLMs) in Chinese civil litigation. |
| Outcome: | The proposed benchmarks show that large language models exhibit an "illusion of competence" the results highlight a critical gap between fluent linguistic output and judicial reliability . |
LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning (2026.acl-long)
Copied to clipboard
Zerui Chen, Qinggang Zhang, Zhishang Xiang, Zhimin Wei, Linfeng Gao, Xiao Huang, Zhihong Zhang, Jinsong Su
| Challenge: | Graph-based Retrieval-Augmented Generation (GraphRAG) is a new approach to document retrieval, but it is not suitable for legal reasoning. |
| Approach: | They propose a framework for reliable legal reasoning that structures knowledge as relational graphs and uses a multi-agent system to verify validity. |
| Outcome: | The proposed framework outperforms existing GraphRAG models in accurate and trustworthy legal analysis. |
IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Legal systems worldwide struggle with exponentially growing legal cases in various courts. |
| Approach: | They propose a benchmark for Indian legal text understanding and reasoning task that includes domain-specific tasks that address different aspects of the legal system. |
| Outcome: | The proposed benchmark for Indian legal text understanding and reasoning aims to address the gap between models and the ground truth. |
Pub-LawBench: Public-Oriented Benchmarking for LegalAI (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on legal professionals, not legal professionals. |
| Approach: | They propose a public-oriented LegalAI benchmark grounded in legal functionalism and genre analysis to address this gap. |
| Outcome: | The proposed model evaluates 17 large language models on Pub-LawBench using simple prompts and Chain-of-Thought under a vanilla inference setting. |
Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to large language models focus on semantic similarity, neglecting the intricate logical structures and reasoning essential for addressing complex legal issues. |
| Approach: | They propose a Logical-Semantic Integration Model (LSIM) that bridges semantic and logical coherence and a supervised framework that integrates semantic features with in-context learning. |
| Outcome: | The proposed framework significantly improves accuracy and reliability on a real-world legal QA dataset. |
GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GreekBarBench evaluates LLMs on legal questions across five different legal areas from the Greek Bar exams. |
| Approach: | They propose a three-dimensional scoring system and an LLM-as-a-judge approach to tackle the challenges of free-text evaluation. |
| Outcome: | The proposed system uses an LLM-as-a-judge approach to evaluate LLMs on legal questions across five legal areas from the Greek Bar exams. |
ART: Adaptive Reasoning Trees for Explainable Claim Verification (2026.findings-eacl)
Copied to clipboard
Sahil Wadhwa, Himanshu Kumar, Guanqun Yang, Abbaas Alif Mohamed Nishar, Pranab Mohanty, Swapnil Shinde, Yue Wu
| Challenge: | Large Language Models (LLMs) are powerful candidates for complex decision-making, leveraging vast encoded knowledge and remarkable zero-shot abilities. |
| Approach: | They propose a hierarchical method for claim verification that uses a root claim and a pairwise tournament of its children to determine an argument's strength. |
| Outcome: | The proposed method outperforms baseline methods on multiple datasets and shows that it is more reliable and clearer than existing methods. |
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022.acl-long)
Copied to clipboard
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras
| Challenge: | Laws and their interpretations, legal arguments and agreements are typically expressed in writing. |
| Approach: | They propose a benchmark to evaluate model performance across legal NLU tasks . they also evaluate several generic and legal-oriented models . |
| Outcome: | The proposed model performs better across multiple tasks than previous models. |