From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work in explainable COC has been limited to annotations by a single expert. |
| Approach: | They construct a two-level task-independent taxonomy from a dataset obtained from two experts in the domain of international human rights law . they find disagreements stem from underspecification of the legal context . |
| Outcome: | The proposed dataset is the first in legal NLP that focuses on human label variation. |
Similar Papers
The Craft of Selective Prediction: Towards Reliable Case Outcome Classification - An Empirical Study on European Court of Human Rights Cases (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing COC tasks prioritize high task performance over model reliability . however, large models exhibit overconfidence and Monte Carlo dropout methods produce reliable confidence estimates . |
| Approach: | They conduct an empirical investigation into how various design choices affect the reliability of COC models within the framework of selective prediction. |
| Outcome: | The proposed model is able to predict the outcome of a legal case based on the text of the case facts and is compared with other models using a pre-training corpus. |
Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for NLP calibration ignore inherent human label variation (HLV) split votes are a problem in high-stakes domains such as legal and medical decisions . |
| Approach: | They present a case outcome classification dataset with judges' vote distributions and build a taxonomy of disagreement with SV-specific subcategories. |
| Outcome: | The proposed model is compared against a judge vote distribution and assesses the alignment of perceived difficulty between models and humans. |
Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals. |
| Approach: | They propose to use domain expertise to identify statistically predictive but legally irrelevant information and adopt adversarial training to prevent it from relying on it. |
| Outcome: | The proposed model aligns better with expert rationales than baseline models . the results are compared with an existing benchmark dataset of human rights cases . |
Towards Explainability in Legal Outcome Prediction Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Current legal outcome prediction models do not explain their reasoning in the real world, but human legal actors need to understand the model’s decisions. |
| Approach: | They propose a method for identifying the precedent employed by legal outcome prediction models and a taxonomy of legal precedent to compare human judges and neural models. |
| Outcome: | The proposed model learns to predict outcomes reasonably well, but its use of precedent is unlike that of human judges. |
LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues (2026.acl-long)
Copied to clipboard
Fanyu Wang, Xiaoxi Kang, Paul Burgess, Aashish Srivastava, Chetan Arora, Adnan Trakic, Lay-Ki Soon, Md Khalid Hossain, Lizhen Qu
| Challenge: | Large language models (LLMs) have impressive reasoning capabilities, but their precision remains inadequate. |
| Approach: | They propose a framework that integrates neural generation with statistical reasoning to improve the accuracy of large language models. |
| Outcome: | The proposed framework achieves interpretability through transparent feature weighting while maintaining data efficiency through correlation-based statistical classification. |
Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies fail to distinguish different classification errors with a standard cross-entropy classification loss and ignore the numbers in the fact description for predicting the term of penalty. |
| Approach: | They propose to extract crime amounts from fact description and use them to learn distinguishable representations to exploit the numbers in the fact description for predicting the term of penalty. |
| Outcome: | The proposed method achieves state-of-the-art results on real-world datasets and ablation studies demonstrate the effectiveness of each component. |
Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Using Swiss Judgement Prediction, we evaluate the explainability of state-of-the-art monolingual and multilingual LJP models. |
| Approach: | They propose an occlusion-based approach to evaluate the explainability performance of legal judgement prediction models using Swiss Judgement Prediction, the only available multilingual LJP dataset. |
| Outcome: | The proposed framework allows us to quantify the influence of lower court information on model predictions, exposing current models’ biases. |
Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases (2021.naacl-main)
Copied to clipboard
Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, Prodromos Malakasiotis
| Challenge: | Interpretability or explainability is an emerging field of research in NLP . experimental results indicate that the newly introduced task is very challenging . |
| Approach: | They propose to extract rationales as paragraphs in multi-paragraph structured court cases . they also propose a constraint that allows models to be more specific . |
| Outcome: | The proposed task is very challenging and there is a large scope for further research. |
Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings (2022.lrec-1)
Copied to clipboard
Thibault Charmet, Inès Cherichi, Matthieu Allain, Urszula Czerwinska, Amaury Fouret, Benoît Sagot, Rachel Bawden
| Challenge: | Detecting divergences in the applications of the law is an important task . divergencies can occur at three levels: within the Cour de Cassation, between trial courts and, more rarely, between a trial court and the Cour of Cassion. |
| Approach: | They propose to provide automatic tools to facilitate the search for similar rulings . they provide automatic keyword sequence generation models and predict keyword sequences based on available texts . |
| Outcome: | The proposed tools improve correlations between the obtained similarities and human judgments of similarity. |
Courtroom-LLM: A Legal-Inspired Multi-LLM Framework for Resolving Ambiguous Text Classifications (2025.coling-main)
Copied to clipboard
| Challenge: | Using a multi-LLM structure inspired by legal courtroom processes, we demonstrate that it can improve decision-making accuracy in ambiguous text classification scenarios. |
| Approach: | They propose a legal-inspired multi-LLM structure that simulates a courtroom setting within LLMs and assigns roles similar to those of prosecutors, defense attorneys, and judges. |
| Outcome: | The proposed model outperforms both single-LLM classifiers and simpler multi-LLMS setups in ambiguous text classification tasks. |