Learning Is Not A Race: Improving Retrieval in Language Models via Equal Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Overparametrized models trained on cross-entropy loss can overfit on noise . Fitting some tokens early reduces gradient signals in later iterations . |
| Approach: | They propose to overfit models trained on cross-entropy loss on noise . fitting some tokens early reduces gradient signals in later iterations . |
| Outcome: | The proposed approaches can be applied to large language models with longer contexts or larger embedding sizes. |
Similar Papers
EqualizeIR: Mitigating Linguistic Biases in Retrieval Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing information retrieval models show significant linguistic biases based on the linguistic complexity of queries. |
| Approach: | They propose a framework to mitigate linguistic biases in IR models by using a linguistically biased weak learner to capture biased queries and then train a robust model by regularizing and refining its predictions. |
| Outcome: | The proposed framework reduces performance disparities across simple and complex queries while improving overall retrieval performance. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing generative language models neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. |
| Approach: | They propose a function to mitigate the imbalance between frequent and infrequent tokens . authors propose 'MiLe Loss' function to assess learning difficulty of tokens during training . |
| Outcome: | Experiments show that models with proposed model can improve on downstream benchmarks. |
Exploring Robust Overfitting for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent literature has revealed their vulnerability to crafted adversarial examples on a wide range of NLP tasks. |
| Approach: | They propose to combine regularization methods with adversarial training to mitigate robust overfitting for pre-trained language models by scaling the model's loss. |
| Outcome: | The proposed methods mitigate robust overfitting upon three top adversarial training methods and further promote adversarially robustness. |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pretrained large language models (LLMs) can bridge the performance gap for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
| Approach: | They propose to use pretrained large language models to bridge this gap by automating and evaluating data-to-text generation in under-resourced languages. |
| Outcome: | The proposed model can set the state of the art for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to generate text data for training and evaluating other models. |
| Approach: | They propose to use logit suppression and temperature sampling to diversify text generation but at the cost of data accuracy. |
| Outcome: | The proposed approach can increase diversity but at the cost of data accuracy. |
Reverse Modeling in Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Using pre-trained LLMs with reversed text inputs can improve their performance across multiple languages. |
| Approach: | They propose a way to determine whether LLMs can understand reversed text inputs by reversing entire paragraphs or documents at the token level. |
| Outcome: | The proposed model can be used to improve understanding across multiple languages. |
Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible (2026.eacl-long)
Copied to clipboard
| Challenge: | linguists have discovered patterns which hold across virtually all known natural languages . lingulists are able to learn languages by comparing their learning curves to those of humans . |
| Approach: | They compare LLM learning curves on existing and "impossible" datasets . they find that GPT-2 learns each language and its impossible counterpart equally easily . |
| Outcome: | The proposed model learns each language and its impossible counterpart equally easily, the study shows . the study also shows that the proposed model does not provide any kind of separation between the possible and the impossible . |
Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest using large language models to make tabular classifications . however, LLMs have been shown to exhibit harmful social biases based on stereotypes and inequalities present in society. |
| Approach: | They propose to use large language models to make tabular classifications . they show that LLMs inherit biases from their training data . |
| Outcome: | The proposed models exhibit harmful biases that reflect stereotypes and inequalities in society. |