Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining

Challenge:	Pretraining large language models has resulted in tremendous performance improvement for many natural language processing tasks.
Approach:	They propose to incorporate pretraining objectives that explicitly exploit domain specific language characteristics into the model.
Outcome:	The proposed objectives target token-level feature representation and incorporate sentence level semantics.

Similar Papers

Overview

Challenge

Approach

Outcome

Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)

Copied to clipboard

Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, Nikolaos Aletras

Language Models

Challenge:	Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations.
Approach:	They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective.
Outcome:	Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture.

Cross-domain Analysis on Japanese Legal Pretrained Language Models (2022.findings-aacl)

Copied to clipboard

Keisuke Miyazaki, Hiroaki Yamada, Takenobu Tokunaga

Semantic Text Processing

Challenge:	Existing studies do not care the performance of domain-adapted PLMs for a generic domain.
Approach:	They propose to use pretraining strategies to build pretrained language models specialised in the legal domain to improve their performance.
Outcome:	The pretrained language models can learn domain-specific and general word meanings simultaneously and can distinguish them.

mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model (2021.findings-emnlp)

Copied to clipboard

Rasmus Kær Jørgensen, Mareike Hartmann, Xiang Dai, Desmond Elliott

Language Models

Challenge:	Existing domain-specific multilingual pretraining data is difficult to obtain due to regulations, legislation, or simply a lack of language- and domain- specific text.
Approach:	They propose to continue pretraining a language model on domain-specific unlabelled text . this allows for better modelling of text for downstream tasks within the domain .
Outcome:	The proposed approach outperforms the general multilingual model and performs close to its monolingual counterpart.

To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks (2020.acl-main)

Copied to clipboard

Sinong Wang, Madian Khabsa, Hao Ma

Language Models

Challenge:	Existing studies on pretraining NLP models with variants of Masked Language Model (MLM) objectives have shown that the number of training samples used in the downstream task is limited.
Approach:	They propose to use MLM objectives to pretrain NLP models with variants of Masked Language Model (MLM) objectives to improve accuracy on downstream tasks.
Outcome:	The proposed model can reach a diminishing return point as the supervised data size increases significantly.

How does the pre-training objective affect what large language models learn about linguistic properties? (2022.acl-short)

Copied to clipboard

Ahmed Alajrami, Nikolaos Aletras

Language Models

Challenge:	Several pre-training objectives have been proposed to pre-train language models . but, to our knowledge, no studies have investigated how different pre- training objectives affect what BERT learns about linguistic properties.
Approach:	They propose to use masked language modeling to pre-train language models . they propose to optimize a mangled language modeling objective to learn linguistic information .
Outcome:	The proposed objectives improve BERT's learning of linguistic properties compared to non-linguistically motivated objectives.

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)

Copied to clipboard

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith

Language Models

Challenge:	Language models prerained on text from a wide variety of sources form the foundation of today’s NLP.
Approach:	They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain.
Outcome:	The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings.

Efficient Vision-Language pre-training via domain-specific learning for human activities (2024.emnlp-main)

Copied to clipboard

Adrian Bulat, Yassine Ouali, Ricardo Guerrero, Brais Martinez, Georgios Tzimiropoulos

Multimodality

Challenge:	Current vision-language models owe their success to large-scale pretraining on web-collected data.
Approach:	They propose a domain-aligned pretraining strategy that aligns the downstream tasks to the downstream domain without additional data collection.
Outcome:	The proposed method outperforms existing models on large-scale vision-language training datasets while preserving generalist knowledge.

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)

Copied to clipboard

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

Language Models

Challenge:	Large pre-trained models suffer from domain shift and are not optimal for specific domains.
Approach:	They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains.
Outcome:	The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE.

NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)

Copied to clipboard

Nikolay Arefyev, Dmitrii Kharchev, Artem Shelmanov

Responsible & Trustworthy NLP

Challenge:	Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding.
Approach:	They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.
Outcome:	The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.

Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains (2024.lrec-main)

Copied to clipboard

Vincent Segonne, Aidan Mannion, Laura Cristina Alonzo Canul, Alexandre Daniel Audibert, Xingyu Liu, Cécile Macaire, Adrien Pupier, Yongxin Zhou, Mathilde Aguiar, Felix E. Herron, Magali Norré, Massih R Amini, Pierrette Bouillon, Iris Eshkol-Taravella, Emmanuelle Esperança-Rodier, Thomas François, Lorraine Goeuriot, Jérôme Goulian, Mathieu Lafourcade, Benjamin Lecouteux, François Portet, Fabien Ringeval, Vincent Vandeghinste, Maximin Coavoux, Marco Dinarelli, Didier Schwab

Language Models

Challenge:	Pretrained language models are the de facto backbone of most state-of-the-art NLP systems.
Approach:	They propose a family of domain-specific pretrained PLMs for French focusing on three important domains: transcribed speech, medicine, and law.
Outcome:	The proposed models perform better on transcribed speech, medicine, and law domains than state-of-the-art models on a diverse set of tasks and datasets.