LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation (2026.acl-long)
Copied to clipboard
Weikang Yuan, Kaisong Song, Zhuoren Jiang, Junjie Cao, Yujie Zhang, Jun Lin, Kun Kuang, Ji Zhang, Xiaozhong Liu
| Challenge: | Current systems for legal consultation are insufficient to handle the knowledge-intensive nature of real-world consultations. |
| Approach: | They propose a multi-turn benchmark dataset to evaluate LLMs in legal consultation settings. |
| Outcome: | The proposed framework assesses LLMs’ consultation capabilities in terms of (1) clarification capability and (2) professional advice quality. |
Similar Papers
UCL-Bench: A Chinese User-Centric Legal Benchmark for Large Language Models (2025.findings-naacl)
Copied to clipboard
Ruoli Gan, Duanyu Feng, Chen Zhang, Zhihang Lin, Haochen Jia, Hao Wang, Zhenyang Cai, Lei Cui, Qianqian Xie, Jimin Huang, Benyou Wang
| Challenge: | Existing legal benchmarks focusing on knowledge and logic evaluate LLMs on various tasks in legal domain, but few have explored the practical application of LLM by actual users. |
| Approach: | They propose a Chinese user-centric legal benchmark that aims to assess the practical application of LLMs by real users. |
| Outcome: | The proposed model outperforms existing models on various tasks in legal domain but does not outperfect ChatGPT. |
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022.acl-long)
Copied to clipboard
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras
| Challenge: | Laws and their interpretations, legal arguments and agreements are typically expressed in writing. |
| Approach: | They propose a benchmark to evaluate model performance across legal NLU tasks . they also evaluate several generic and legal-oriented models . |
| Outcome: | The proposed model performs better across multiple tasks than previous models. |
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice (2026.acl-long)
Copied to clipboard
Yuzhen Shi, Huanghai Liu, Yiran HU, Song Gaojie, Xu Xinran, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Feng Di, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Qiu Yuanyang, Yuemeng Qi, Zhang Jingwen, Sui Xiaoyu, Yifan Chen, Zhang Yi, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, HU Wei
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice (2026.acl-long)
Copied to clipboard
Ziang Chen, Guannan Li, Fanlin Ji, Yipeng Kang, Jiaqi Li, Muhan Zhang, Yangtao Zhang, Li Tianjiao, Jiannan Wang, Xin Guo, Song-Chun Zhu, Bin Ling
| Challenge: | Existing legal benchmarks evaluate isolated tasks or exam-style questions, failing to capture the procedural interdependencies and adjudicative rigor inherent in professional practice. |
| Approach: | They propose a vertical, depth-oriented, domain-specific benchmark to evaluate Large Language Models (LLMs) in Chinese civil litigation. |
| Outcome: | The proposed benchmarks show that large language models exhibit an "illusion of competence" the results highlight a critical gap between fluent linguistic output and judicial reliability . |
Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed a substantial increase in the demand for legal services, especially for individuals with modest means. |
| Approach: | They propose a diagnostic legal large language model which uses adaptive lawyer-like diagnostic questions to collect additional case information and then provides high-quality feedback. |
| Outcome: | The proposed model surpasses classical LLMs by providing outstanding performance and a remarkable user experience in the legal domain. |
A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. |
| Approach: | They propose a framework to investigate LLMs' competence in the law domain by using similar cases and multi-choice options. |
| Outcome: | The proposed solutions can be extended to other domains to facilitate evaluations in other domain. |
-Stance: A Large-Scale Real World Dataset of Stances in Legal Argumentation (2025.acl-long)
Copied to clipboard
| Challenge: | Current tools for legal argument reasoning do not support this task. |
| Approach: | They propose to use a large-scale dataset to facilitate work on the legal argument stance classification task by evaluating whether a case summary strengthens or weakens a legal argument. |
| Outcome: | The proposed dataset is used to facilitate work on the legal argument stance classification task, which involves assessing whether a case summary strengthens or weakens a legal argument (polarity) and to what extent (intensity). |
A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on specific predefined model abilities, such as world knowledge, reasoning, etc., making it difficult for users to determine which LLM best suits their particular needs. |
| Approach: | They propose to evaluate large language models from a user-centric perspective and use real-world use cases to identify their effectiveness under distinct intents. |
| Outcome: | The proposed benchmarks achieve a correlation between human preference and the user-reported scenarios and human intents. |
LawBench: Benchmarking Legal Knowledge of Large Language Models (2024.emnlp-main)
Copied to clipboard
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, Vincent Ng
| Challenge: | LegalBench evaluated 20 LLMs in 162 legal tasks in 20 countries and jurisdictions. |
| Approach: | They present a comprehensive evaluation of 21 popular Large Language Models and the first comparative analysis of the empirical results. |
| Outcome: | The proposed benchmarks are based on the Bloom’s cognitive taxonomy and are compared to 21 popular LLMs. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |