| Challenge: | Existing guidelines for pre-training and fine-tuning do not always generalize well in the legal domain. |
| Approach: | They propose to use BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domains. |
| Outcome: | The proposed strategies are: use the original BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domain specific corpors. |
Similar Papers
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research examines simple classification tasks, but ability of LMs to classify on complex tasks is less well understood. |
| Approach: | They analyze a Supreme Court opinion annotated by a team of domain experts . they find generative models perform poorly when given instructions equal to human annotators . |
| Outcome: | The proposed model performs poorly when given instructions equal to instructions given to human annotations . strongest results derive from fine-tuning models on the annotated dataset . |
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent years have witnessed the widespread use of transfer learning techniques in Natural Language Processing (NLP) |
| Approach: | They train BERT models from scratch using many configurations involving general and medical corpora. |
| Outcome: | The initial corpus only has a weak influence when these are further pre-trained on a medical corpus. |
A Legal Perspective on Training Models for Natural Language Processing (L18-1)
Copied to clipboard
| Challenge: | a significant concern in processing natural language data is the unclear legal status of the input and output data/resources. |
| Approach: | They examine which legal rules apply at relevant steps and how they affect the legal status of the results. |
| Outcome: | The proposed model training process is based on three scenarios . the analysis focuses on which legal rules apply and how they affect the legal status of the results . |
LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing large language models are not designed for semantic retrieval and PDF-based legislative sources introduce substantial noise due to imperfect text extraction. |
| Approach: | They propose a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages. |
| Outcome: | The proposed model improves Top-k retrieval accuracy in monolingual and bilingual settings . it also improves accuracy in low- and high-resource languages . |
LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain (2025.findings-naacl)
Copied to clipboard
Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, Christopher D Manning
| Challenge: | In general, instruction tuning is important for direct user interaction, but the legal domain is underrepresented in typical instruction datasets. |
| Approach: | They aggregate 58 annotated legal datasets and write instructions for each to create LawInstruct. |
| Outcome: | The proposed model improves on LegalBench across all model sizes, but no drop in MMLU. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text (2024.eacl-long)
Copied to clipboard
Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, Kyryl Truskovskyi
| Challenge: | a recent study focused on detecting legal violations within unstructured textual data . a similar study focused only on associating violations with potentially affected individuals . |
| Approach: | They constructed two datasets using Large Language Models (LLMs) they publicize the results to advance legal natural language processing research . |
| Outcome: | The proposed datasets and the code used for the experiments have been released to advance legal natural language processing (NLP) |
Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models require huge training corpora, which is unobtainable for most NLP practitioners. |
| Approach: | They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget. |
| Outcome: | The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics. |
LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development (2023.acl-long)
Copied to clipboard
| Challenge: | In this study, we examine the performance of legal-oriented pre-trained language models. |
| Approach: | They conduct a detailed analysis on the performance of legal-oriented pre-trained language models by examining their original objective, acquired knowledge, and legal language understanding capacities. |
| Outcome: | The results show that the models' size and pre-training corpora are important for the development of domain-specific models. |
Unveiling the Potential of BERT-family: A New Recipe for Building Scalable, General and Competitive Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) have significantly influenced various aspects of society, reshaping how we access and interact with information and knowledge. |
| Approach: | They propose a pre-training task that helps BERT-family excel in wider applications . they also explore the integration of cutting-edge technologies into their models to further enhance their capabilities. |
| Outcome: | The proposed model exhibits performance levels comparable to current SOTA LLMs across a spectrum of tasks. |