Papers with Pre-training
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent LLM development trends involve pre-training models with a vast number of parameters on massive datasets. |
| Approach: | They propose a method that integrates parameter-efficient tuning techniques with model growth to reduce memory requirements while maintaining equivalent performance. |
| Outcome: | The proposed method reduces memory requirements by 53.9% while maintaining equivalent performance to vanilla pre-trained models on downstream tasks. |
Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages. |
| Approach: | They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences . |
| Outcome: | The proposed method can be used on four sentence pair modeling tasks in English and Japanese. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for reducing computational costs during pre-training have been studied, but they often degrade performance under fair conditions. |
| Approach: | They propose a method that combines parameter-efficient tuning and staged training to reduce memory requirements while maintaining comparable performance. |
| Outcome: | The proposed method reduces memory requirements by 40.4% while maintaining comparable performance. |
XDoc: Unified Pre-training for Cross-Format Document Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained models target one document format at a time, making it difficult to combine knowledge from multiple document formats. |
| Approach: | They propose a unified pre-trained model which deals with different document formats in a single model. |
| Outcome: | The proposed model achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models. |
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to build initial unsupervised machine translation models with cross-lingual n-gram embeddings are inexplicit and limited. |
| Approach: | They propose a cross-lingual pre-training method that incorporates cross-linguistic training signals into pre-trained models by randomly choosing source n-grams in the input text stream. |
| Outcome: | The proposed method significantly improves the performance of unsupervised machine translation. |
Analyzing Effects of Learning Downstream Tasks on Moral Bias in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models replicate and perpetuate social biases . pre-existing moral bias may be mitigated or amplified even when presented with opposing views . |
| Approach: | They develop methods to assess the agreement of LMs to explicit codified norms . they find that introducing downstream tasks may lead to unexpected inconsistencies . |
| Outcome: | The proposed model can be used to improve morality in data-scarce tasks. |
Tree Transformer: Integrating Tree Structures into Self-Attention (D19-1)
Copied to clipboard
| Challenge: | Existing work on hierarchical structure in neural networks has not captured human intuitions about hierarchic structures. |
| Approach: | They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder to encourage attention heads to follow tree structures. |
| Outcome: | The proposed model improves language modeling and learning more explainable attention scores. |
SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse (2025.findings-naacl)
Copied to clipboard
| Challenge: | Sparse mixture of experts (SMoE) is an effective approach for scaling large language models with a constant computational cost. |
| Approach: | They propose a similarity-based sparse mixture of experts (SimSMoE) that guarantees a solution to address the representation collapse problem between experts given a fixed FLOPs budget. |
| Outcome: | The proposed method outperforms existing methods and outperformed other SMoE routing methods in performance. |
ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions to reduce the cost of pretraining Transformer-based models are expensive especially for large-scale models. |
| Approach: | They propose to reduce the cost of pre-training Transformer-based models by compressing the sequence of hidden states inside Transformer architecture. |
| Outcome: | The proposed model achieves state-of-the-art on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models. |
Domain-matched Pre-training Tasks for Dense Retrieval (2022.findings-naacl)
Copied to clipboard
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, Yashar Mehdad
| Challenge: | Existing approaches to improve performance of pre-training tasks are needed. |
| Approach: | They propose to pre-train large bi-encoder models on a recently released set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting reddit conversation dataset. |
| Outcome: | The proposed model can be pre-trained on a set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting dataset of Reddit conversations. |
Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation (2023.acl-short)
Copied to clipboard
| Challenge: | End-to-end speech translation models have limited training data and are often inefficient due to the inconsistency of length and representation between speech and text. |
| Approach: | They find that the "modality gap" between speech and text data is not a major problem in E2E ST . they decouple the encoder to speech encoder and text encoder, and they find that there is a 'capacity gap' |
| Outcome: | The proposed model achieves 29.0 for en-de and 40.3 for fr on the MuST-C dataset. |
UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)
Copied to clipboard
Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu
| Challenge: | Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data. |
| Approach: | They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts. |
| Outcome: | The proposed model breaks through performance upper bounds of experts without additional annotated data. |
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. |
| Approach: | They propose a new approach to learn generic representations adapted to spoken dialog using a hierarchical encoder based on transformer architectures. |
| Outcome: | The proposed model is model-agnostic and contains 10 datasets of various sizes. |
On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that PT and BT are nicely complementary to each other. |
| Approach: | They introduce two probing tasks for PT and BT respectively and investigate their complementarity. |
| Outcome: | The proposed methods establish state-of-the-art on the WMT16 English-Romanian and English-Russian benchmarks. |
UnitCoder: Scalable Code Synthesis from Pre-training Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at code understanding and generation, yet code generation remains a challenge. |
| Approach: | They propose a model that supervises pre-training data quality through automatically generated unit tests while ensuring correctness via an iterative fix and refine flow. |
| Outcome: | The proposed model improves performance on a large dataset with high quality pre-training data. |
An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training large language models is a standard practice in the natural language processing community. |
| Approach: | They propose to use elastic weight consolidation to mitigate catastrophic forgetting when pre-trained large language models are evaluated on generic benchmarks. |
| Outcome: | The proposed model achieves state-of-the-art on out-of domain tasks with minimal pre-training . elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. |
Semi-supervised URL Segmentation with Recurrent Neural Networks Pre-trained on Knowledge Graph Entities (2020.coling-main)
Copied to clipboard
| Challenge: | Domain names such as openresearch are being added to a growing set of tokens that an NLP system may need to deal with. |
| Approach: | They propose a tagging model that uses characters as input to break domain names into component words . they propose taagging methods that use concatenated entity names in a large knowledge database . |
| Outcome: | The proposed model improves on concatenated entity names in a knowledge database by 33% . the proposed model can be used for a wide range of languages, including Chinese and Japanese . |
The Thieves on Sesame Street are Polyglots - Extracting Multilingual Models from Monolingual APIs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated that deployed NLP models can be stolen by adversaries by querying victim models with gibberish input data that consists of random sequences of words. |
| Approach: | They propose to extract a local copy of a monolingual victim model from an API and query it with gibberish input data paired with the victim's labels. |
| Outcome: | The extracted model learns the task from the monolingual victim, but it generalizes far better than the victim to several other languages. |
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls (2025.emnlp-main)
Copied to clipboard
Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, Carole-Jean Wu
| Challenge: | a large-scale empirical study compares natural web data, diverse synthetic types, and mixtures of natural and synthetic data. |
| Approach: | They conduct a large-scale empirical study on large-volume LLMs using a unified protocol and scaling laws. |
| Outcome: | The proposed method is faster than pre-training on natural web data, the authors show . their results are consistent with previous studies on rephrased text and textbooks . |
Re3Dial: Retrieve, Reorganize and Rescale Conversations for Long-Turn Open-Domain Dialogue Pre-training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-training models lack long-turn dialogue sessions due to the scarcity of long-term sessions. |
| Approach: | They propose a framework that can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
| Outcome: | The proposed framework can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on multi-modal dialogue pre-training is limited due to limited availability of multi-dimensional data . a recent emergence of chatGPT 1 has increased confidence in the potential for this goal . |
| Approach: | They propose a framework for multi-modal dialogue pre-training that integrates experts to accommodate multi-faceted tasks. |
| Outcome: | The proposed framework achieves state-of-the-art on eight multi-modal dialog benchmarks. |
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, Quanzeng You
| Challenge: | InfiMM-WebMath-40B is a dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Approach: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it contains 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Outcome: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |
Block Pruning For Faster Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pruning methods have proven to be effective at reducing model size, while distillation methods are proven for speeding up inference. |
| Approach: | They propose a block pruning approach that integrates structured pruning methods with the movement pruning paradigm for fine-tuning. |
| Outcome: | The proposed model is 2.4x faster, 74% smaller and faster than distilled models on classification and generation tasks. |