JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text simplification focus on only one dimension: fluency, simplicity and meaning preservation. |
| Approach: | They introduce a dataset to assess legal meaning preservation between two legal texts . they also introduce sanity checks for two identical sentences . |
| Outcome: | The proposed metric shows superior correlation with human judgment compared to existing metrics. |
Similar Papers
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022.acl-long)
Copied to clipboard
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras
| Challenge: | Laws and their interpretations, legal arguments and agreements are typically expressed in writing. |
| Approach: | They propose a benchmark to evaluate model performance across legal NLU tasks . they also evaluate several generic and legal-oriented models . |
| Outcome: | The proposed model performs better across multiple tasks than previous models. |
Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings (2022.lrec-1)
Copied to clipboard
Thibault Charmet, Inès Cherichi, Matthieu Allain, Urszula Czerwinska, Amaury Fouret, Benoît Sagot, Rachel Bawden
| Challenge: | Detecting divergences in the applications of the law is an important task . divergencies can occur at three levels: within the Cour de Cassation, between trial courts and, more rarely, between a trial court and the Cour of Cassion. |
| Approach: | They propose to provide automatic tools to facilitate the search for similar rulings . they provide automatic keyword sequence generation models and predict keyword sequences based on available texts . |
| Outcome: | The proposed tools improve correlations between the obtained similarities and human judgments of similarity. |
Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals. |
| Approach: | They propose to use domain expertise to identify statistically predictive but legally irrelevant information and adopt adversarial training to prevent it from relying on it. |
| Outcome: | The proposed model aligns better with expert rationales than baseline models . the results are compared with an existing benchmark dataset of human rights cases . |
Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Using Swiss Judgement Prediction, we evaluate the explainability of state-of-the-art monolingual and multilingual LJP models. |
| Approach: | They propose an occlusion-based approach to evaluate the explainability performance of legal judgement prediction models using Swiss Judgement Prediction, the only available multilingual LJP dataset. |
| Outcome: | The proposed framework allows us to quantify the influence of lower court information on model predictions, exposing current models’ biases. |
Validity Assessment of Legal Will Statements as Natural Language Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | This study introduces a dataset that focuses on the validity of statements in legal wills. |
| Approach: | They propose a dataset that focuses on the validity of statements in legal wills. |
| Outcome: | The proposed model achieves 80% macro F1 and accuracy, but group accuracy is in mid 80s at best, suggesting that the models’ understanding of the task remains superficial. |
On the Ethical Limits of Natural Language Processing on Legal Text (2021.findings-acl)
Copied to clipboard
| Challenge: | Legal NLP holds the promise of improving access to justice and offers tools for empirical analysis of law on a large scale. |
| Approach: | They propose ways to think systematically about ethical limits of NLP . they place emphasis on three crucial normative parameters that have been underestimated . |
| Outcome: | The proposed methods are based on a real-life scenario that has prompted debate in the legal NLP community. |
DETECT: Determining Ease and Textual Clarity of German Text Simplifications (2026.eacl-long)
Copied to clipboard
| Challenge: | Current evaluation of German automatic text simplification relies on general-purpose metrics such as SARI, BLEU, and BERTScore. |
| Approach: | They propose a German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency. |
| Outcome: | The proposed metric achieves higher correlations with human judgments than widely used ATS metrics. |
Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerland (2025.findings-emnlp)
Copied to clipboard
Luca Rolshoven, Vishvaksenan Rasiah, Srinanda Brügger Bose, Sarah Hostettler, Lara Burkhalter, Matthias Stürmer, Joel Niklaus
| Challenge: | a dataset of 20K rulings from the Swiss Federal Supreme Court is lacking in legal headnotes due to the high cost of manual annotation. |
| Approach: | They propose a dataset that contains 20K rulings from the Swiss Federal Supreme Court . they fine-tune open models and compare them to larger general-purpose and reasoning-tunned LLMs . |
| Outcome: | The proposed dataset contains 20K rulings from the Swiss Federal Supreme Court with headnotes in German, French, and Italian. |
JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice (2026.acl-long)
Copied to clipboard
Ziang Chen, Guannan Li, Fanlin Ji, Yipeng Kang, Jiaqi Li, Muhan Zhang, Yangtao Zhang, Li Tianjiao, Jiannan Wang, Xin Guo, Song-Chun Zhu, Bin Ling
| Challenge: | Existing legal benchmarks evaluate isolated tasks or exam-style questions, failing to capture the procedural interdependencies and adjudicative rigor inherent in professional practice. |
| Approach: | They propose a vertical, depth-oriented, domain-specific benchmark to evaluate Large Language Models (LLMs) in Chinese civil litigation. |
| Outcome: | The proposed benchmarks show that large language models exhibit an "illusion of competence" the results highlight a critical gap between fluent linguistic output and judicial reliability . |
A large-scale computational study of content preservation measures for text style transfer and paraphrase generation (2022.acl-srw)
Copied to clipboard
| Challenge: | Text style transfer and paraphrases generation are growing areas of NLP . many researchers still use BLEU-like measures to evaluate content preservation . |
| Approach: | They compare 57 different measures based on different principles on 19 annotated datasets . they find that measures relying on cross-encoder models outperform alternative approaches . |
| Outcome: | The proposed methods outperform traditional methods on 19 datasets. |