A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT (2026.findings-acl)
Copied to clipboard
| Challenge: | Diversity has been gaining interest in the NLP community in recent years. |
| Approach: | They propose to use diversity-driven sampling to pre-train models on French with a fixed compute budget. |
| Outcome: | The diversity-driven sampling reduces the pre-training dataset by 94% and the pretraining time by 73% while maintaining comparable performance. |
Similar Papers
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
On the Impact of Cross-Domain Data on German Language Models (2023.findings-emnlp)
Copied to clipboard
Amin Dada, Aokun Chen, Cheng Peng, Kaleb Smith, Ahmad Idrissi-Yaghir, Constantin Seibold, Jianning Li, Lars Heiliger, Christoph Friedrich, Daniel Truhn, Jan Egger, Jiang Bian, Jens Kleesiek, Yonghui Wu
| Challenge: | Traditionally, large language models have been trained on general web crawls or domain-specific data. |
| Approach: | They present a German dataset and a dataset aimed at containing high-quality data to examine the importance of data diversity over quality. |
| Outcome: | The proposed model outperforms models trained on quality data on multiple downstream tasks. |
SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity (2025.findings-emnlp)
Copied to clipboard
Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye
| Challenge: | Existing methods for pretraining data mixing for large language models neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. |
| Approach: | They propose a sample-wise data mixture approach that performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample. |
| Outcome: | The proposed method exceeds existing domain-based methods in multiple downstream tasks and perplexity assessments. |
Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent study has not examined their generalizability between formats, cultures, and genders. |
| Approach: | They evaluate large language models (LLMs) and small LLMs at clinical de-identification . they show that smaller models achieve comparable performance while substantially reducing inference cost . |
| Outcome: | The proposed models outperform larger models in de-identification tasks with limited data . the models can be fine-tuned with limited datasets to outperformed larger models . |
Generating Diverse Training Samples for Relation Extraction with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models for Relation Extraction (RE) have good results on many benchmarks, but data scarcity is a common problem. |
| Approach: | They propose to use Large Language Models to generate training data for Relation Extraction . they propose to make LLMs produce dissimilar samples by direct instruction . |
| Outcome: | The proposed approach improves the diversity of training samples generated with LLMs while maintaining correctness. |
Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality (2021.naacl-main)
Copied to clipboard
| Challenge: | In human-level NLP tasks, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within transformer-based language models. |
| Approach: | They propose to use dimension reduction methods to fine-tune large models with limited data and to use pre-trained dimension reduction regimes to improve model performance. |
| Outcome: | The proposed model outperforms other models in human-level NLP tasks with a pre-trained dimension reduction regime. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Diversity-oriented Data Augmentation with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing data augmentation methods focus on increasing sample numbers while neglecting sample distribution diversity, which can lead to model overfitting. |
| Approach: | They propose a data augmentation framework that focuses on sample distribution diversity and trains a large language model as a diverse paraphraser. |
| Outcome: | The proposed framework achieves an average performance gain of 10.52% surpassing the runner-up baseline with more than three percentage points. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
When Do You Need Billions of Words of Pretraining Data? (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are dominated by models that can encode billions of words. |
| Approach: | They use classifier probing, information-theoretic probing and unsupervised relative acceptability judgments to evaluate model ability. |
| Outcome: | The proposed models require only about 10M to 100M words to learn to encode most syntactic and semantic features. |