Papers with regularizer
Topic Balancing with Additive Regularization of Topic Models (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for topic modelling on unbalanced data contain topics in various proportions and documents of the relatively small theme become distributed all over the larger topics instead of being grouped into one topic. |
| Approach: | They propose a new regularizer for topic models on unbalanced data collections . they make sure this regularizer increases the quality of topic models, trained on unstructured data . |
| Outcome: | The proposed method improves the quality of topic models trained on unbalanced datasets. |
Adaptive Weighted Proxy Tuning: Efficient Gray-Box Steering for Image Captioning. (2026.acl-industry)
Copied to clipboard
| Challenge: | Proxy tuning is a decoding-time approach that fails to account for instance-specific variations in model certainty and domain shift. |
| Approach: | They propose a gray-box steering framework that dynamically modulates the logit contributions of a large base model, a fine-tuned expert, and an untune . |
| Outcome: | Adaptive Weighted Proxy Tuning achieves performance parity with fine-tuned models while remaining parameter-free. |
K-hop neighbourhood regularization for few-shot learning on graphs: A case study of text classification (2023.eacl-main)
Copied to clipboard
| Challenge: | We show that few-sample word-document graphs can be used for improved learning in low-resource settings. |
| Approach: | They propose a method to utilize word-document graph properties for improved learning in low-resource settings by using a regularizer for heterogeneous graphs. |
| Outcome: | The proposed method outperforms a baseline TextGCN with 17% accuracy over eight languages while performing on par with the state-of-the-art models. |
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart
| Challenge: | Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping. |
| Approach: | They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models. |
| Outcome: | The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets. |
SelectNoise: Unsupervised Noise Injection to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, MT systems for low-resource languages lack parallel data and monolingual data. |
| Approach: | They propose an unsupervised approach to generate noisy HRLs training data by selective candidate extraction and noise injection. |
| Outcome: | The proposed model outperforms strong baselines on 12 ELRLs in a zero-shot setting . |
Disentangling the Roles of Target-side Transfer and Regularization in Multilingual Machine Translation (2024.eacl-long)
Copied to clipboard
| Challenge: | Multilingual Machine Translation (MMT) benefits from knowledge transfer across different language pairs, but performance differences between one-to-many and many-to-1 translation are negligible. |
| Approach: | They conduct a large-scale study that varies the auxiliary target-side languages along two dimensions to show the dynamic impact of knowledge transfer on the main language pairs. |
| Outcome: | The proposed model can translate between multiple languages with minimal positive transfer ability. |
Uncertainty-Aware Bootstrap Learning for Joint Extraction on Distantly-Supervised Data (2023.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to extract entity pairs and their relations from labeled data are noisy and expensive. |
| Approach: | They propose a bootstrap learning approach that is motivated by intuition that the higher the uncertainty of an instance, the more likely the model confidence is inconsistent with the ground truths. |
| Outcome: | The proposed method outperforms baselines and related methods on two large datasets. |
Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in language modeling have enabled applications across multiple domains such as education, jurisprudence, and healthcare. |
| Approach: | They propose a method to use knowledge to identify which rare words are important and uplift their conditional probability. |
| Outcome: | The proposed approach reduces the uncertainty of the model and improves factuality and coherence without negatively impacting fluency. |
Graph Enhanced Dual Attention Network for Document-Level Relation Extraction (2020.coling-main)
Copied to clipboard
| Challenge: | Document-level relation extraction requires inter-sentence reasoning capabilities to capture local and global contextual information for multiple relation facts. |
| Approach: | They propose to characterize the interaction between sentences and potential relation instances via a Graph Enhanced Dual Attention network (GEDA) . they also propose a simple yet effective regularizer based on the natural duality of the S2R and R2S attentions, whose weights are also supervised by the supporting evidence of relation instances during training. |
| Outcome: | The proposed model achieves competitive performance on an existing large-scale dataset while the predictions can be interpretable and easily observed. |
Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in Pretrained Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods to adapt pretrained embeddings to a large corpus are limited and do not provide sufficient quality. |
| Approach: | They propose to use a small corpus D_T to pretrain embeddings that accurately capture the sense of words in a limited set of focused topics. |
| Outcome: | The proposed embeddings capture the sense of words in a topic in spite of the limited size of the corpus D_T. |
Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experiments on five text classification benchmarks and five backbone models have shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource conditions (upto 17.5%). |
| Approach: | They propose to add a small adversarial perturbation to the mixing coefficients rather than the examples to relax locally linear constraints. |
| Outcome: | Experiments on five text classification benchmarks and five backbone models show that the proposed methods reduce the error rate over Mixup variants by 31.3%, especially in low-resource conditions. |
Text Generation Model Enhanced with Semantic Information in Aspect Category Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for ACSA fail to model relations of target words and opinion words in a sentence including multiple aspects. |
| Approach: | They propose to incorporate AMR into a text generation model to model relations of target words and opinion words in a sentence including multiple aspects. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three datasets. |
Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models (N19-1)
Copied to clipboard
| Challenge: | a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus is effective, and more time-efficient than training on ngrams. |
| Approach: | They propose a technique that uses large corpus n-gram statistics as a regularizer for training on a smaller corpus. |
| Outcome: | The proposed technique is effective and more time-efficient than training on a larger corpus. |
QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization (D19-1)
Copied to clipboard
| Challenge: | Existing models are not good at distinguishing distractor sentences which look related but do not answer the question. |
| Approach: | They propose a method to regularize question answering models by maximizing mutual information among passages, questions, and answers. |
| Outcome: | The proposed model achieves state-of-the-art on the Adversarial-SQuAD dataset. |
Affect-Driven Dialog Generation (N19-1)
Copied to clipboard
| Challenge: | Existing systems for end-to-end dialog generation focus on response quality without explicit control over affective content of the responses. |
| Approach: | They propose an affect-driven dialog system which generates emotional responses using a continuous representation of emotions. |
| Outcome: | The proposed system outperforms existing systems in terms of BLEU score and response diversity, and qualitative measures. |
Penalizing Divergence: Multi-Parallel Translation for Low-Resource Languages of North America (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies show that multi-parallel translation models can overfit when training data are limited. |
| Approach: | They introduce a regularizer which penalizes translation models when they represent source sentences with identical target translations in divergent ways. |
| Outcome: | The proposed model improves when the target data for all language pairs are identical. |
The Role of n-gram Smoothing in the Age of Neural Networks (2024.naacl-long)
Copied to clipboard
| Challenge: | n-gram smoothing techniques were used to overcome overfitting problems in neural language models for decades. |
| Approach: | They propose to convert any n-gram smoothing technique into a regularizer compatible with neural language models. |
| Outcome: | The proposed regularizers outperform label smoothing on language modeling and machine translation. |
A Cognitive Regularizer for Language Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | a uniform information density hypothesis is used to explain certain linguistic phenomena . a regularizer that encodes the UID hypothesis can be used for language training . |
| Approach: | They propose to augment the canonical MLE objective with a regularizer that encodes UID . they find that regularization consistently improves perplexity in language models . |
| Outcome: | The proposed hypothesis can be operationalized as an inductive bias for language modeling. |
Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Overconfidence in model generalization and calibration has been shown to impair model generalisation and calibration. |
| Approach: | They propose a regularization scheme that takes model probability into account and takes it into account . they use a prior label distribution to smooth target labels . |
| Outcome: | The proposed model improves model generalization and calibration by taking model probability into account. |
Efficient Training of Language Models with Compact and Consistent Next Token Distributions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train language models have focused on maximizing the likelihood of the next token . however, the construction and querying of such n-grams can be costly and impede training speed. |
| Approach: | They propose a method to train language models faster by pre-aggregating corpus with collapsed n-gram distribution. |
| Outcome: | The proposed model improves model quality and convergence rate while reducing variance across mini-batches compared to the standard next-token loss method. |
PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Image translation is a task that translates an image containing text in the source language to the target language. |
| Approach: | They propose an end-to-end image translation framework that bridges the modality gap between visual inputs and textual inputs/outputs of machine translation (MT). |
| Outcome: | The proposed framework outperforms existing models on a large-scale image translation corpus . it significantly outperformed both cascaded and strong models on the e-commerce domain . |
Language Model Priors and Data Augmentation Strategies for Low-resource Machine Translation: A Case Study Using Finnish to Northern Sámi (2024.findings-acl)
Copied to clipboard
| Challenge: | a new study examines the use of monolingual data for improving low-resource machine translation. |
| Approach: | They investigate ways of using monolingual data for improving low-resource machine translation. |
| Outcome: | The proposed model can perform better on the target-side data without augmentation of parallel data. |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |
Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve security of LLM generated code are ineffective and lack localized regions of code. |
| Approach: | They propose a method for distilling a preference dataset of insecure and secure code pairs from frontier LLMs and a security reasoning that explains the issues and the fix. |
| Outcome: | The proposed method reduces code insecurity while improving overall code quality. |
LoRE: Enhancing Search Relevance with Progressive Chain-of-Thought and Preference Alignment (2026.findings-acl)
Copied to clipboard
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, null Lihaoran, Songyan Liu, Pengjie Wang, Chuan Yu, Jian Xu, Bo Zheng
| Challenge: | E-commerce search relevance is a critical component of retrieval systems. |
| Approach: | They propose a large-generative model for search relevance that trains reasoning knowledge, multi-modal understanding and rule awareness into three core competencies. |
| Outcome: | The proposed model outperforms GPT-5 in Macro-F1 and achieves 27% online gain. |
Octopus: Gated Selective Attention for Memory-Bounded Long-Context Inference in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Subquadratic architectures rely on aggressive state compression that degrades performance on complex reasoning tasks. |
| Approach: | They propose a framework that confers fixed-memory inference onto pretrained Transformers . they use a learnable module that enforces an adaptive sparsity policy over the context history . |
| Outcome: | The proposed framework outperforms state-of-the-art linearized baselines on the GSM8K benchmark by over 36 points under identical memory constraints. |