Papers with LMs
What do Large Language Models Learn about Scripts? (2022.starsem-1)
Copied to clipboard
| Challenge: | Script Knowledge is important for language understanding but expensive to produce manually and difficult to induce from text due to reporting bias. |
| Approach: | They propose a pipeline-based script induction framework which can generate good quality ESDs for unseen scenarios. |
| Outcome: | The proposed framework produces good quality ESDs for unseen scenarios, but manual evaluation shows there is room for improvement. |
Benchmarking Distributional Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group remains uncertain. |
| Approach: | They construct a dataset expanding beyond political values and create human baselines for this task and evaluate the extent to which an LM can align with a particular group’s opinion distribution. |
| Outcome: | The proposed model can better describe opinion distributions than simulate demographic groups. |
LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) are opaque and unexplained, causing problems for endusers and developers who wish to debug or fix their behaviour. |
| Approach: | They propose an interactive debugger tool for transformer-based LMs that provides a fine-grained interpretation of the model's internal prediction process and a powerful framework for intervening in LM behavior. |
| Outcome: | The proposed tool provides a fine-grained interpretation of the model's internal prediction construction process, and a powerful framework for intervening in LM behavior. |
Complex Reasoning in Natural Language (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Recent research shows that pretrained language models are often brittle for complex reasoning tasks. |
| Approach: | They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks . |
| Outcome: | This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness . |
Can Transformer Language Models Predict Psychometric Properties? (2021.starsem-1)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) are gaining popularity on many NLP benchmark tasks. |
| Approach: | They use human responses to calculate psychometric properties of test items . they find transformer-based LMs predict psychometric property consistently well . |
| Outcome: | The transformer-based language models are able to predict psychometric properties of test items . the models can predict psychometries well in certain categories but poorly in others . |
Investigating Critical Period Effects in Language Acquisition through Neural Language Models (2025.tacl-1)
Copied to clipboard
| Challenge: | Scholars of human development have long debated whether these phenomena are predetermined by innately encoded developmental changes in the maturing brain or natural consequences of increased experience. |
| Approach: | They use language models to test whether CP effects are peculiar to humans . they find that LMs do not show CP when L2 exposure is delayed . scholars have long debated whether innate maturation changes predetermine CP . |
| Outcome: | The proposed model does not show CP effects when the age of exposure of L2 is delayed. |
Retrieval-based Language Models and Applications (2023.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will provide a comprehensive overview of retrieval-based language models. |
| Approach: | This tutorial will provide a comprehensive overview of recent advances in retrieval-based language models. |
| Outcome: | This tutorial will provide a comprehensive overview of recent advances in retrieval-based language models. |
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process (2025.acl-long)
Copied to clipboard
| Challenge: | Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are key processes for aligning Language Models with human preferences post pre-training. |
| Approach: | They propose to combine Supervised Fine-Tuning and Preference Optimization (PO) with two sub-processes defined at token level within the Markov Decision Process (MDP) |
| Outcome: | The proposed process performs comparably or even superiorly to SFT and some typical PO methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. |
Distribution Shifts Are Bottlenecks: Extensive Evaluation for Grounding Language Models to Knowledge Bases (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing benchmarks fail to reflect robustness challenges and fairly evaluate models. |
| Approach: | They propose to ground language models to knowledge bases to investigate distribution shifts in language and linguistic aspects of distribution shift. |
| Outcome: | The proposed method fails to evaluate language models in large and small datasets . the proposed model fails to cope with unseen schemas and language variations . |
GOLFer: Smaller LMs-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs)-based query expansion for information retrieval necessitates larger, more advanced LLMs. |
| Approach: | They propose a method leveraging smaller open-source LMs for query expansion that augments queries with generated hypothetical documents with LLMs. |
| Outcome: | The proposed method outperforms existing methods on three web search and ten low-resource datasets and maintains competitive performance against larger LLMs. |
Hierarchical Text Classification with LLM-Refined Taxonomies (2026.eacl-long)
Copied to clipboard
| Challenge: | Hierarchical text classification (HTC) relies on taxonomies that organize labels into structured hierarchies, but many real-world taxonomies introduce ambiguities, such as identical leaf names under similar parent nodes, which prevent language models from learning clear decision boundaries. |
| Approach: | They propose a framework that uses large language models to transform entire taxonomies through operations such as renaming, merging, splitting, and reordering to better match the semantics encoded by LMs. |
| Outcome: | The proposed framework outperforms human-curated taxonomies in three HTC benchmarks and shows that it aligns better with the model's actual confusion patterns. |
Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent work has observed an inverse scaling relationship between Transformers’ per-word estimated probability and the predictive power of their surprisal estimates on reading times. |
| Approach: | They conducted a more comprehensive evaluation using surprisal estimates from 17 pre-trained LMs on two functional magnetic resonance imaging datasets. |
| Outcome: | Recent work shows that surprisal from larger Transformer-based models is less predictive of reading times, resolving the inconclusive results and indicating that this trend is not specific to latency-based measures. |
Thesis proposal: COGNILENS: Analyzing Cognitive Decline in Language Models for Alzheimer’s Monitoring (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods to detect AD and Mild Cognitive Impairment (MCI) are not effective in early stages. |
| Approach: | They propose to develop digital twins of Alzheimer's Disease using language models to mimic functional deficits observed in AD patients. |
| Outcome: | The proposed models will mimic the functional deficits observed in AD patients and evaluate their effects on brain score against the state-of-the-art models. |
Large Language Models for Psycholinguistic Plausibility Pretesting (2024.findings-eacl)
Copied to clipboard
| Challenge: | Psycholinguists typically use language models to create controlled materials . plausibility judgments are often based on coarse-grained judgements, but fine-grounded ones do not . |
| Approach: | They investigate whether Language Models can be used to generate plausibility judgments . they find that plausible judgements from LMs are highly related to human judgements - whereas other LM models are not . |
| Outcome: | The proposed language models can generate plausibility judgments from human evaluators . the proposed models do not provide satisfactory discriminative power . |
Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms. |
| Approach: | They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity. |
| Outcome: | The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics . |
Transformers for Tabular Data Representation: A Survey of Models and Applications (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent research efforts extend LMs by developing neural representations for structured data. |
| Approach: | They propose to extend transformer-based language models to tabular data by analyzing inputs, model training, and supported downstream tasks. |
| Outcome: | The proposed models are compared against existing models and are based on a traditional pipeline. |
Aligned Probing: Relating Toxic Behavior and Model Internals (2026.tacl-1)
Copied to clipboard
| Challenge: | Language models (LMs) may produce toxic text that contains hate speech, insults, or vulgarity, even when prompted with innocuous text. |
| Approach: | They propose an interpretability framework that aligns the behavior of language models based on their outputs and internal representations. |
| Outcome: | The proposed framework bridges behavioral and internal perspectives for toxicity for the first time. |
Hey, wait a minute: on at-issue sensitivity in Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing methods to evaluate dialogue naturalness are limited. |
| Approach: | They propose a method to assess dialogue naturalness using linguistic notion of at-issueness. |
| Outcome: | The proposed method mitigates bias in linguistic analyses of LMs and tests discourse-sensitive behavior. |
Time-Aware Language Models as Temporal Knowledge Bases (2022.tacl-1)
Copied to clipboard
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
| Challenge: | Existing language models are trained on snapshots of data collected at a specific moment in time. |
| Approach: | They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. |
| Outcome: | The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods. |
Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing labeled datasets are heavily imbalanced, limiting the QA performance in this domain. |
| Approach: | They propose a question answering task that captures relevant text segments from unlabeled policy documents and expands the positive examples in the training set. |
| Outcome: | The proposed framework elevates the baseline by a large margin (10% F1) and achieves a new state-of-the-art F1 score of 50%. |
Efficient Long-Text Understanding with Short-Text Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing transformer-based pretrained language models cannot be applied to long sequences due to their quadratic complexity. |
| Approach: | They propose a simple approach to long sequences that re-uses battle-tested short-text pretrained LMs. |
| Outcome: | The proposed approach is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. |
Monotonic Representation of Numeric Attributes in Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902, but how this information is encoded in the model’s internal representations is not understood well. |
| Approach: | They propose a method for finding and editing representations of numeric properties such as Karl Popper’s birth year. |
| Outcome: | The proposed method can express an increasingly late birthyear by patching activations along a “birthyear” direction. |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information (2024.starsem-1)
Copied to clipboard
| Challenge: | Contextualized word representations from pre-trained language models encode more information than is necessary for the identification of word senses and some of this information affect performance negatively in unsupervised settings. |
| Approach: | They propose to use a framework to erase specific information from pre-trained word models and create feature-invariant representations that are invariant to these ‘nuisance features’. |
| Outcome: | The proposed framework erases information from the representations of pre-trained language models, thereby creating feature-invariant representations. |
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts (2023.acl-short)
Copied to clipboard
| Challenge: | Text detoxification can mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle. |
| Approach: | They propose a text detoxification algorithm that combines controllable generation and text rewriting methods using a Product of Experts and autoencoder language models to find candidate words to mask and potentially replace. |
| Outcome: | The proposed method outperforms baselines on automatic metrics and is preferred 2.1 times more in human evaluation. |
Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024.eacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are statistical models trained to assign probability to human-generated text. |
| Approach: | They evaluate language models' ability to reproduce variability that humans exhibit in the ‘next word prediction’ task. |
| Outcome: | The language models are trained to assign probability to human-generated text . they exhibit low calibration to human uncertainty, and advise against it . |
Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Linguistic steganography studies how to hide secret messages in natural language cover texts. |
| Approach: | They propose a method which encodes secret messages using self-adjusting arithmetic coding based on a neural language model. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on four datasets by 15.3% and 38.9% in terms of bits/word and KL metrics. |
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2020.tacl-1)
Copied to clipboard
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman
| Challenge: | Recent studies have examined how linguistic knowledge of language models (LMs) varies across English phenomena. |
| Approach: | They propose a benchmark to evaluate linguistic knowledge of language models on major grammatical phenomena in English. |
| Outcome: | The proposed benchmark evaluates the linguistic knowledge of language models on major grammatical phenomena in English. |
Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2022.findings-naacl)
Copied to clipboard
| Challenge: | Our approach pairs an LM with a growing memory of cases where the user identified an output error and provided general feedback on how to correct it. |
| Approach: | They propose to use an existing script generator to train a model to repair output errors without retraining. |
| Outcome: | The proposed model learns to apply user feedback to repair output errors while avoiding similar past mistakes on new, unseen examples. |
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour (2022.aacl-short)
Copied to clipboard
| Challenge: | Language models trained on text-only corpora have no direct access to the physical world and thus suffer from reporting bias. |
| Approach: | They investigate reporting bias from the perspective of colour in larger language models such as PaLM and GPT-3. |
| Outcome: | The proposed models outperform smaller models on the basis of colour and more closely track human judgements than smaller models. |
Explore Spurious Correlations at the Concept Level in Language Models for Text Classification (2024.acl-long)
Copied to clipboard
| Challenge: | Language models have demonstrated remarkable performance in numerous NLP tasks, employing both fine-tuning and in-context learning (ICL) methods. |
| Approach: | They propose a method to assess concept bias in models during fine-tuning and in-context learning using ChatGPT. |
| Outcome: | The proposed method outperforms token removal approaches and is validated through extensive testing. |
Selective Perception: Learning Concise State Descriptions for Language Model Actors (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent large language models support longer contexts, but requiring them to process redundant or irrelevant data increases inference time and cost. |
| Approach: | They propose a method that leverages a small finetuned LM to sample the minimal set of input features that maximizes the performance of a downstream LM. |
| Outcome: | The proposed method reduces the length of LM actor input by 87% and 99% while improving task success rates by 158% and 54% on NetHack and robot planning. |
HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing methods do not examine social groups categorised by geographical information, leaving the region-related biases in pre-trained LMs unexplored. |
| Approach: | They propose a hierarchical regional bias evaluation method to quantify regional bias in pre-trained language models. |
| Outcome: | The proposed method evaluates regional bias with regard to comprehensive topics and measures potential regional bias that can be propagated to downstream tasks. |
Humans and transformer LMs: Abstraction drives language learning (2026.eacl-long)
Copied to clipboard
| Challenge: | lexical semantic and syntactic categories emerge using novel divergence-based metrics . |
| Approach: | They compare transformer-based language model's linguistic categories learning to exemplar-based accounts of human language acquisition. |
| Outcome: | The proposed model can be used as an existence proof for human language acquisition. |
Literature-Augmented Clinical Outcome Prediction (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to clinical outcome prediction use only clinical notes and general biomedical literature. |
| Approach: | They propose to retrieve patient-specific medical literature and incorporate it into predictive models by combining clinical notes with language models. |
| Outcome: | The proposed approach boosts predictive performance on three important clinical tasks in comparison to strong LM baselines, increasing F1 by up to 5 points and precision@Top-K by a large margin of over 25%. |
Finding a Needle in the Adversarial Haystack: A Targeted Paraphrasing Approach For Uncovering Edge Cases with Minimal Distribution Distortion (2024.eacl-long)
Copied to clipboard
| Challenge: | Adversarial attacks against Language models (LMs) are a significant concern. |
| Approach: | They propose an approach to automatically learn a policy to generate challenging examples that improve the model’s performance. |
| Outcome: | The proposed approach outperforms baselines and exhibits generalizability across classifiers and datasets. |
DECAF: A Dynamically Extensible Corpus Analysis Framework (2025.acl-demo)
Copied to clipboard
| Challenge: | DeCAF is an open-source Python library that enables the analysis and filtering of linguistically-annotated datasets down to the character level. |
| Approach: | They propose a framework that enables the analysis and filtering of linguistically-annotated datasets down to the character level. |
| Outcome: | The proposed framework analyzes a parsed version of the 115M-word BabyLM corpus and generates highly controlled and reproducible experimental settings targeting specific research questions. |
A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese (2026.tacl-1)
Copied to clipboard
Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, Hai Hu
| Challenge: | Using sub-linear length normalized log-probabilities (SLLN-LP), we find unequal lengths of sentences in minimal pairs difficult for LMs even up to 32B parameters. |
| Approach: | They propose to use ZhoBLiMP as a linguistic minimal pair benchmark for Chinese language models to mitigate biases. |
| Outcome: | The proposed metric mitigates biases in Chinese language models with over 100 paradigms . Anaphor, Quantifiers, and Ellipsis are difficult for LMs even up to 32B parameters . |
Plug-and-Play Adaptation for Continuously-updated QA (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing tasks to assess LMs’ efficacy as KBs do not adequately consider multiple large-scale updates. |
| Approach: | They propose a task where multiple large-scale updates are made to language models and plug-in modules are used to handle the updates. |
| Outcome: | The proposed method outperforms existing methods on zsRE QA and NQ datasets and is 4x more effective in terms of updates/forgets ratio compared to a fine-tuning baseline. |
Can language models learn from explanations in context? (2022.findings-emnlp)
Copied to clipboard
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, Felix Hill
| Challenge: | Language Models can adapt to a few in-context examples, but without training. |
| Approach: | They examine how explanations of few-shot examples can help Language Models (LMs) explanations can improve performance even without tuning, they find . |
| Outcome: | The proposed explanations outperform hand-tuned explanations on small validation sets. |
BeamR: Beam Reweighing with Attribute Discriminators for Controllable Text Generation (2022.findings-aacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have led to the availability of large pre-trained language models with rich generative capabilities. |
| Approach: | They propose a method to combine generative LMs with attribute discriminators to control different attributes of text generation. |
| Outcome: | The proposed method performs better than existing state-of-the-art approaches in sentiment steering and machine translation formality tasks. |
Task-driven Layerwise Additive Activation Intervention (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing approaches to task adaptation rely heavily on heuristic rules or prompt inputs. |
| Approach: | They propose a layer-wise additive activation intervention framework that steers the LMs’ generation process by identifying and manipulating the activations. |
| Outcome: | The proposed framework improves the accuracy of pretrained LMs and competing baselines on various datasets, demonstrating improvements in the accuracy and sample efficiency of the proposed framework. |
A Family of Pretrained Transformer Language Models for Russian (2024.lrec-main)
Copied to clipboard
Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, Alena Fenogenova
| Challenge: | Developing Transformer language models for the Russian language has received little attention . most of these LMs are developed for English, which imposes substantial constraints on the potential of the language technologies. |
| Approach: | They propose to release 13 Russian Transformer language models that span three languages . they aim to broaden the scope of NLP research directions and develop industrial solutions for the Russian language. |
| Outcome: | The proposed models are based on Russian language datasets and benchmarks. |
oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent success of pre-trained language models has spurred widespread interest in their capabilities. |
| Approach: | They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls . |
| Outcome: | The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls. |
Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses (2021.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. |
| Approach: | They propose a setup for analyzing lexical polysemy knowledge in pre-trained language models and multilingual BERT models by analyzing different sense distributions and controlling for parameters that are highly correlated with polysyntax. |
| Outcome: | The proposed model can be used to analyze lexical polysemy in English, French, Spanish, and Greek and in multilingual BERT. |
Reflect-RL: Two-Player Online RL Fine-Tuning for LMs (2024.acl-long)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. |
| Approach: | They propose a two-player system to fine-tune an LM using SFT and online RL . they use negative example generation to enhance error-correction ability of the reflection model . |
| Outcome: | The proposed system outperforms SFT and online RL without reflection on a GPT-2 XL 1.56B model. |
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. |
| Approach: | They investigate how well pretrained language models reason with terms less frequent in pretraining data. |
| Outcome: | The models are more accurate on instances whose terms are more prevalent, in some cases above 70% more accurate than the bottom 10%. |
Federated Learning of Gboard Language Models with Differential Privacy (2023.acl-industry)
Copied to clipboard
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan Mcmahan, Jesse Rosenstock, Yuanbo Zhang
| Challenge: | Using federated learning and differential privacy, we train and deploy language models with federation and DP in Google Keyboard. |
| Approach: | They train and deploy language models with federated learning and differential privacy in Google Keyboard . |
| Outcome: | The proposed algorithm achieves meaningfully formal DP guarantees without uniform sampling of clients. |
Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods that decompose multi-hop questions into single hop sub-questions are difficult to implement. |
| Approach: | They propose to use random-walks to guide pre-trained language models to map multi-hop questions to random-walked paths that lead to the answer. |
| Outcome: | The proposed methods improve on two T5 LMs. |
Do Language Models Know When They’re Hallucinating References? (2024.findings-eacl)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. |
| Approach: | They propose to use hallucinated book and article references as "model organism" of hallucinism research . authors propose queries to the language model to identify hallucinous references . |
| Outcome: | The authors show that language models can identify hallucinated references without external resources . they show that LMs often produce inconsistent author lists for hallucinos, but also accurately recall the authors of real references . |
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Approach: | They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Outcome: | The proposed approach significantly improves over a baseline approach. |
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
Learning Rich Representation of Keyphrases from Text (2022.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has referred to extractive (part of document) or abstractive (not part of document). |
| Approach: | They propose to use a new pre-training objective to introduce keyphrases into transformer language models in discriminative and generative settings. |
| Outcome: | The proposed model improves performance in discriminative and generative settings and also improves on named entity recognition, question answering, relation extraction and abstractive summarization tasks. |
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations. |
| Approach: | They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context. |
| Outcome: | The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks. |
Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models (2024.findings-acl)
Copied to clipboard
Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
| Challenge: | Existing methods to expand internal memory boundaries of language models by providing external context can often conflict, leading to knowledge conflicts. |
| Approach: | They propose a method that prunes conflicting attention heads without updating model parameters. |
| Outcome: | The proposed method can flexibly control eight LMs to use internal memory or external context without updating model parameters. |
Can Machine Unlearning Reduce Social Bias in Language Models? (2024.emnlp-industry)
Copied to clipboard
Omkar Dige, Diljot Arneja, Tsz Fung Yau, Qixuan Zhang, Mohammad Bolandraftar, Xiaodan Zhu, Faiza Khattak
| Challenge: | Existing methods for mitigating bias in language models are expensive and time-consuming . comparative studies have not evaluated their respective advantages and disadvantages . |
| Approach: | They propose to use Partitioned Contrastive Gradient Unlearning and Negation via Task Vector to reduce social biases in open-source language models. |
| Outcome: | The proposed methods outperform PCGU and DPO in debiasing models . the proposed methods can be easily tuned to balance the trade-off between bias reduction and generation quality . |
Topicalization in Language Models: A Case Study on Japanese (2022.coling-1)
Copied to clipboard
| Challenge: | a recent study has shown that neural language models can capture discourse-level preferences in text generation . a particular aspect of discourse is the topic-comment structure . |
| Approach: | They analyze whether neural language models can capture discourse-level preferences in text generation . they use Japanese language and crowdsourced human topicalization judgment data . |
| Outcome: | The proposed model can capture human-like generalizations in discourse-level linguistic aspects. |
Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking (2021.acl-short)
Copied to clipboard
| Challenge: | Existing work on transferring domain-specific knowledge from a pretraining model to a resource-poor language is limited to English . a novel cross-lingual biomedical entity linking task is proposed to improve this capability. |
| Approach: | They propose a cross-lingual biomedical entity linking task and establish a new benchmark spanning 10 typologically diverse languages. |
| Outcome: | The proposed methods yield consistent gains across all target languages, sometimes up to 20 Precision@1 points, without any in-domain knowledge in the target language and without any parallel data. |
Condenser: a Pre-training Architecture for Dense Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work fine-tunes deep LMs to encode text sequences into single dense vector representations, but dense encoders require a lot of data and sophisticated techniques to train and suffer in low data situations. |
| Approach: | They propose to pre-train Transformer language models (LMs) with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. |
| Outcome: | The proposed model improves on various text retrieval and similarity tasks by large margins over standard LMs. |
Evaluating Language Model Character Traits (2024.findings-emnlp)
Copied to clipboard
Francis Ward, Zejia Yang, Alex Jackson, Randy Brown, Chandler Smith, Grace Colverd, Louis Thomson, Raymond Douglas, Patrik Bartak, Andrew Rowan
| Challenge: | Language models (LMs) exhibit human-like behaviour, but it is unclear how to describe this behaviour without undue anthropomorphism. |
| Approach: | They formalise a behaviourist view of LM character traits and infer belief and intent from LM behaviour, finding consistency varies with model size, fine-tuning, and prompting. |
| Outcome: | The proposed model enables us to describe LM behaviour precisely and without undue anthropomorphism. |
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. |
| Approach: | They propose a method where a reward model is trained on preference data in one source language and applied to other target languages. |
| Outcome: | The proposed approach is effective under comprehensive evaluation settings, including human evaluation. |
Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new metric is developed to pinpoint the moment of invocation when hallucinations arise in small LMs. |
| Approach: | They propose a metric that measures hallucinations during the generation process of small LMs. |
| Outcome: | The proposed metric outperforms baselines in hallucination detection across multiple QA datasets. |
Morphology Without Borders: Clause-Level Morphology (2022.tacl-1)
Copied to clipboard
| Challenge: | Morphological tasks use large multi-lingual datasets that organize words into inflection tables . lack of a clear linguistic and operational definition of what is a word impairs universality of tasks . |
| Approach: | They propose to view morphology as a clause-level phenomenon, rather than word-level . they propose to use a dataset for clause- level morphological tasks in 4 different languages . |
| Outcome: | The proposed dataset for clause-level morphology covers 4 typologically different languages: English, German, Turkish, and Hebrew. |
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (LMs) are an integral part of natural language processing but their usability is constrained by computational and time complexity and their increasing size. |
| Approach: | They propose a technique for converting knowledge of fully parameterised LMs into a compact recursive student. |
| Outcome: | The proposed models match the performance of bloated models with negligible performance losses. |
Identifying Linear Relational Concepts in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a technique for finding concept directions for human-interpretable concepts is needed to find their direction in the latent space . a linear relational concept (LRC) can be used to locate concepts in hidden activations . |
| Approach: | They propose a method for finding human-interpretable concepts by inverting a linear relational embedding and using earlier object layers. |
| Outcome: | The proposed method outperforms standard probing classifiers on performance as concept classifier and ability to causally change model output. |
Injecting Numerical Reasoning Skills into Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for numerical reasoning are limited by their flexibility and require specialized architectures to capture high-level skills. |
| Approach: | They propose to inject numerical reasoning skills into pre-trained LMs by generating large amounts of data and training in a multi-task setup. |
| Outcome: | The proposed model performs better on DROP than other models of comparable size while maintaining high performance on standard RC tasks. |
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models encode correlations between social groups and traits, like associating the group with the group. |
| Approach: | They adapt the Agency-Belief-Communion (ABC) stereotype model to a language model and introduce the sensitivity test (SeT) to measure stereotypical associations. |
| Outcome: | The proposed framework is used to measure stereotyping of intersectional identities in language models. |
Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification require numerous LMs’ calls to search optimal prompts, thus resulting in overfitting performance and increasing computational cost. |
| Approach: | They propose a multi-scale knowledge prompt-based memory model that extracts instance-level and class-level knowledge and stores them in memory banks during training. |
| Outcome: | Experiments on different benchmarks and parameter analysis demonstrate the effectiveness and efficiency of MuSKPrompt in black-box few-shot text classification tasks. |
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using positional embeddings, Causal transformer language models learn an implicit notion of absolute positions. |
| Approach: | They propose to use positional embeddings to encode positional information in transformer language models. |
| Outcome: | The proposed model learns an implicit notion of absolute positions across datasets, model sizes, and sequence lengths. |
Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communities (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies treat ideology as a liberal/conservative binary and fail to capture the spectrum of ideologies that may organically emerge in interconnected online communities. |
| Approach: | They propose a method that uses finetuning language models to probe nuanced ideologies of online communities by analyzing discussions of the 2020 election on Twitter. |
| Outcome: | The proposed approach shows higher alignment than baselines for the proposed approach. |
Learning Multi-Step Reasoning by Solving Arithmetic Tasks (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies have demonstrated large LMs’ impressive performance in solving math problems, but such ability seems only to emerge from models with abundant parameters. |
| Approach: | They propose to continuously pre-train LMs with the capabilities of multi-step reasoning by continuously pretraining them on a synthetic dataset MsAT. |
| Outcome: | The proposed method improves LMs' multi-step reasoning abilities on four math word problem datasets. |
ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Performance prediction is a method to estimate the performance of Language Models (LMs) on various Natural Language Processing (NLP) tasks. |
| Approach: | They propose a task- and language-agnostic framework to predict the performance of Language Models (LMs) using proxy models. |
| Outcome: | The proposed framework outperforms the state-of-the-art in root-mean-square error (RMSE) and other robustness tests on multilingual NLP tasks. |
Language Models for Lexical Inference in Context (2021.eacl-main)
Copied to clipboard
| Challenge: | Lexical inference in context (LIiC) is a variant of the natural language inference task focused on lexical semantics. |
| Approach: | They propose three approaches based on pretrained language models for LIiC . they propose a few-shot NLI classifier and a relation induction approach based upon handcrafted patterns expressing the semantics of lexical inference. |
| Outcome: | The proposed approaches outperform the previous state of the art and show their potential for LIiC. |
What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work on LM and acceptability judgments treat these effects uniformly across models, making a strong assumption that models require the same degree of adjustment to control for length and unigram frequency effects. |
| Approach: | They propose a linking theory where the optimal level of adjustment is estimated from data via learned parameters for length and unigram frequency. |
| Outcome: | The proposed theory outperforms a commonly used linking theory for acceptability—SLOR—across two families of transformer LMs. |
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating text are unsupervised and require supervision. |
| Approach: | They propose an unsupervised method that uses two off-the-shelf pretrained LMs in opposite directions to apply them to non-sequential tasks. |
| Outcome: | The proposed method outperforms strong unsupervised baselines on paraphrasing and abductive text infilling. |
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on how to encode compositional task structure has been limited by semantic parsing and multihop reasoning for the purpose of Q&A. |
| Approach: | They propose an approach to decomposing a target task into component tasks and fine-tuning smaller LMs on a curriculum of such component tasks. |
| Outcome: | The proposed approach outperforms end-to-end learning even with equal data, and gets better as more component tasks are modeled. |
Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches to building monolingual models for low-resource languages require a full model tuning process. |
| Approach: | They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language. |
| Outcome: | The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language. |
Self-Influence Guided Data Reweighting for Language Model Pre-training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) pre-trained with selfsupervision on large text data are the default starting point for developing models for various downstream tasks. |
| Approach: | They propose a method for jointly reweighting samples by leveraging self-influence scores as an indicator of sample importance and pre-training. |
| Outcome: | The proposed method promotes novelty and stability for model pre-training. |
Does Vision Accelerate Hierarchical Generalization in Neural Language Learners? (2025.coling-main)
Copied to clipboard
| Challenge: | Neural language models (LMs) are arguably less data-efficient than humans from a language acquisition perspective. |
| Approach: | They investigate the advantage of grounded language acquisition over visual input to improve syntactic generalization. |
| Outcome: | The proposed model is less efficient than humans in language acquisition . it shows that visual input helps syntactic generalization, but not vision . |
Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge (2024.eacl-long)
Copied to clipboard
| Challenge: | Using low-resource languages, multilingual language models (ML-LMs) have been developed to transfer factual knowledge across languages. |
| Approach: | They ask how ML-LMs acquire and represent factual knowledge . they use a multilingual factual information probing dataset to investigate ML . |
| Outcome: | The findings highlight the challenge of maintaining consistency factual knowledge across languages. |
Ethos: Rectifying Language Models in Orthogonal Parameter Space (2024.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) generate toxic, biased content and reveal private training records. |
| Approach: | They propose an efficient approach that rectifies LMs to mitigate toxicity and bias . Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors . |
| Outcome: | The proposed approach mitigates toxicity and bias in outputs and avoids privacy leakage. |
Nonparametric Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. |
| Approach: | They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it. |
| Outcome: | The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering. |
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that asking for direct user feedback can be disruptive . we examine whether incorporating the contents of user feedback improves model performance . |
| Approach: | They analyze user feedback in the user-LLM conversation logs and harvest learning signals from it. |
| Outcome: | The proposed approach can lead to model degradation on two user-LM interaction datasets. |
Efficient Benchmarking (of Language Models) (2024.naacl-long)
Copied to clipboard
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen
| Challenge: | Efficient Benchmarking is a method for reducing computation costs of LM evaluation without compromising reliability. |
| Approach: | They propose to reduce the computation costs of LM evaluation without compromising reliability by using a new measure - Decision Impact on Reliability. |
| Outcome: | The proposed benchmarks reduce computation costs by x100 or more, while maintaining reliability. |
Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) often rely on spurious correlations rather than causally relevant features to improve accuracy and generalizability. |
| Approach: | They propose a benchmark that categorizes shortcuts into occurrence, style, and concept . they aim to explore the nuanced ways shortcuts influence the performance of LMs . |
| Outcome: | The proposed benchmark categorizes shortcuts into occurrence, style, and concept . it systematically investigates models’ resilience and susceptibilities to sophisticated shortcuts . |
Can Language Models Serve as Temporal Knowledge Bases? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have only considered language models as knowledge bases in a static setting . memorizing conflicting information is still challenging for LMs and hinders memorization of other unrelated one-to-one relationships. |
| Approach: | They propose two requirements for treating language models as temporal knowledge bases . they propose a dataset which is aimed at probing temporally-scoped knowledge . |
| Outcome: | The proposed model can store conflicting information and use stored knowledge for temporal knowledge queries. |
Demystifying optimized prompts in language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) are not robust to out-of-distribution inputs. |
| Approach: | They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them. |
| Outcome: | The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data. |
Using Similarity Measures to Select Pretraining Data for NER (N19-1)
Copied to clipboard
| Challenge: | Existing studies on how to select appropriate data to pretrain word vectors or LMs are lacking. |
| Approach: | They propose to quantify aspects of similarity between pretraining and target data. |
| Outcome: | The proposed measures are good predictors of the usefulness of pretrained models for Named Entity Recognition over 30 data pairs. |
Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies have focused on auto-regressive models for generalization in named entity (NE) typing (NET) and recognition (NER) . however, little has been done in this direction for auto-Regressive LMs despite their popularity and potential to express a wide variety of NLP tasks in the same unified format. |
| Approach: | They propose to probe auto-regressive LMs for NET and NER generalization by resorting to meta-learning to assess the model's memorization of NEs. |
| Outcome: | The proposed model performs well on NET and NER generalization tasks, while relying more on NE than contextual cues in few-shot NER. |
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained multilingual language models (LMs) can be 'rewired' into effective multilingual sentence encoders (SEs) however, it remains unclear how to best leverage them to represent sub-sentence lexical items in cross-lingual lexicals. |
| Approach: | They propose a method for exposing cross-lingual lexical knowledge by additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. |
| Outcome: | The proposed method exposes cross-lingual lexical knowledge by additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. |
Scientific and Creative Analogies in Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing analogy datasets focus on a limited set of analogical relations with a high similarity of the two domains between which the analogy holds. |
| Approach: | They propose a dataset that encodes analogy in pretrained language models . they use a system that maps attributes and relational structures across dissimilar domains . |
| Outcome: | The proposed dataset shows that state-of-the-art models achieve low performance on analogy tasks . |
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing efforts to train pre-trained language models have brought significant improvements to various NLP applications. |
| Approach: | They propose to compress bulky LMs while preserving useful information for a specific task. |
| Outcome: | The proposed method can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. |
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have enabled synthetic text generation that is often comparable to the organic text. |
| Approach: | They propose and test several ML-based methods to attribute authorship of synthetic text to language models (LMs) they propose to use a fine-tuned version of XLNet to achieve excellent accuracy . |
| Outcome: | The proposed method achieves excellent accuracy (91% to near perfect 98%) across a range of experiments where the synthetic text may be generated using pre-trained LMs, fine-tuned LM, or by varying text generation parameters. |
BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate LMs rely on objective function and are therefore limited to masked or causal LM types. |
| Approach: | They propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. |
| Outcome: | The proposed framework can probe for knowledge across different LM types. |
Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing retrieval-augmented language models assume query relevance and irrelevance as dichotomy . existing models are highly brittle to the presence of conflicting information in both the fine-tuning and in-context few-shot learning scenarios. |
| Approach: | They propose methods for handling knowledge conflicts by fine-tuning a discriminator or prompting it to elicit its discriminative capability. |
| Outcome: | The proposed approaches significantly enhance model robustness on open-domain QA. |
How Much Syntactic Supervision is “Good Enough”? (2023.findings-eacl)
Copied to clipboard
| Challenge: | RNNGs with syntactic supervision underperformed RNNs with some syntaktic supervision, whereas RNNS with mild supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. |
| Approach: | They propose a method where syntactic LMs are gradually ablated from full syntatic supervision to zero syntastic supervision by preserving NP, VP, PP, SBAR nonterminal symbols. |
| Outcome: | The proposed method underperforms the RNNGs with zero syntactic supervision, and the LMs with mild syntaktic supervision perform better than the state-of-the-art GPT-2-XL. |
When2Call: When (not) to Call Tools (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on the accuracy of tool calling and less on when LMs should (not) call tools. |
| Approach: | They develop a benchmark which evaluates tool-calling decision-making . they leverage multiple-choice nature of the benchmark to develop 'preference optimization' training regime . |
| Outcome: | The proposed benchmark shows that state-of-the-art LMs show room for improvement on When2Call. |
HistAlign: Improving Context Dependency in Language Generation by Aligning with History (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) can generate hallucinations and incoherent outputs due to their weak context dependency. |
| Approach: | They propose a training approach to ensure good cache alignment so that the model receives useful signals from the history. |
| Outcome: | The proposed approach improves text coherence and faithfulness on diverse language generation tasks. |
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters. |
| Approach: | They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set. |
| Outcome: | The proposed method improves performance in all cases, while using a domain expert 23x smaller. |
NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark (2025.findings-acl)
Copied to clipboard
Vladislav Mikhailov, Tita Enstad, David Samuel, Hans Christian Farsethås, Andrey Kutuzov, Erik Velldal, Lilja Øvrelid
| Challenge: | NorEval is a new evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). |
| Approach: | They propose a new evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs) NorEval consists of 24 high-quality human-created datasets, of which five are created from scratch. |
| Outcome: | The evaluation framework and materials are publicly available. |
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing work on word alignment has focused on unsupervised learning on parallel text. |
| Approach: | They propose to combine pre-trained contextualized word embeddings with multilingually trained language models to achieve competitive results on word alignment tasks. |
| Outcome: | The proposed model outperforms state-of-the-art models on five language pairs and can train multilingual word aligners that can obtain robust performance on different language pairs. |
Beyond Memorization: The Challenge of Random Memory Access in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Language Models (LMs) have shown their effectiveness in knowledge-intensive tasks. |
| Approach: | They investigate whether a generative language model is able to access its memory sequentially or randomly. |
| Outcome: | The proposed LMs are able to access memory sequentially or randomly. |
Evaluating Step-by-Step Reasoning through Symbolic Verification (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. |
| Approach: | They propose to use symbolic examples to iteratively reason over symbolic examples and to recover Prolog’s backward chaining algorithm to iterate over KBs. |
| Outcome: | The proposed model performs better on length generalization benchmarks than CoT on explanations and chain-of-thoughts (CoT) tasks. |
We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit (2023.acl-long)
Copied to clipboard
| Challenge: | ellipsis is a linguistic phenomenon characterized by the omission of one or more sentence elements. |
| Approach: | They investigated how prototypicality affects the ability of Language Models to handle elliptical sentences . they found that models were better suited to evaluating argument thematic fit . |
| Outcome: | The proposed dataset shows that the models perform better for typical events than for atypical ones in different elliptical contexts. |
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach (2025.coling-main)
Copied to clipboard
| Challenge: | Using language models (LMs) has increased in use, and the use of biases and stereotypes is creating social problems. |
| Approach: | They propose a method to mitigate LM biases by continual training on biased data . they use masked language modeling to construct a Bias Vector as the difference between biased LMs and pre-trained LM weights . |
| Outcome: | The proposed method improves on the GLUE and SEAT benchmarks. |
Detoxifying Language Models Risks Marginalizing Minority Voices (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing detoxification techniques have been proposed to mitigate toxic LM generations . e.g., detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups . |
| Approach: | They propose to use detoxification techniques to reduce toxic LM generations without affecting perplexity or generation quality on nontoxic inputs. |
| Outcome: | The proposed methods hurt equity on language used by marginalized groups, the authors show . they show that detoxification makes LMs more brittle to distribution shift, they say . |
Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | DP-training methods that require per-example gradients are difficult to implement . however, a method that prioritizes query privacy is not feasible. |
| Approach: | They propose a method that prioritizes ensuring query privacy prior to training a deep retrieval system. |
| Outcome: | The proposed method shows that it improves retrieval quality compared to direct DP-training while maintaining query-level privacy guarantees. |
Red Teaming Language Models with Language Models (2022.emnlp-main)
Copied to clipboard
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, Geoffrey Irving
| Challenge: | Prior work has found that language models (LMs) can harm users in hard-to-predict ways, and human annotation is expensive, limiting the number and diversity of test cases. |
| Approach: | They propose to generate test inputs using an LM itself, and use a classifier to detect harmful behavior on test input. |
| Outcome: | The proposed approach detects tens of thousands of offensive responses in a 280B parameter LM chatbot. |
SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of Indian Culture (2025.findings-acl)
Copied to clipboard
| Challenge: | Language models excel in syntactic and semantic analysis, while small language models struggle in region-specific contexts. |
| Approach: | They evaluate SANSKRITI on leading Large Language Models, Indic Language Model, and Small Language Model (SLM) it covers 16 key attributes of Indian culture including rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities. |
| Outcome: | The SANSKRITI dataset covers 16 attributes of Indian culture . it reveals that many models struggle in region-specific contexts . |
Adapting Language Models to Compress Contexts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based language models have a finite context window and expensive computational cost of processing long text documents. |
| Approach: | They propose to adapt pre-trained LMs into AutoCompressors to compress text into summary vectors . authors propose to use summary vector to speed up inference over long contexts based on a finite context window . |
| Outcome: | The proposed model can compress long contexts into summary vectors, which are accessible as soft prompts. |
Evaluating Text GANs as Language Models (N19-1)
Copied to clipboard
| Challenge: | Generative Adversarial Networks (GANs) do not suffer from the problem of exposure bias. |
| Approach: | They propose to approximate the distribution of text generated by a GAN and compare it to traditional probability-based LM metrics. |
| Outcome: | The proposed method performs significantly worse than state-of-the-art LMs on several GAN-based models and can accelerate progress in GAN text generation. |
Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature is agnostic about a parsing strategy of hierarchical models . a recent study showed that hierarchically model hierarchic structures capture grammatical dependencies much better than RNNs in targeted syntactic evaluations. |
| Approach: | They evaluated three LMs with head-final left-branching structures and Recurrent Neural Network Grammars with top-down and left-corner parsing strategies as hierarchical models. |
| Outcome: | The proposed model outperforms top-down and left-corner models against human reading times in Japanese. |
CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following (2024.acl-long)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) are increasingly exposed to private data and are becoming more and more prevalent. |
| Approach: | They propose a collaborative generation framework that integrates large and small language models to address privacy concerns logically. |
| Outcome: | The proposed framework combines large and small models to address privacy concerns logically. |
CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels (2023.acl-long)
Copied to clipboard
| Challenge: | Utilizing language models without internal access is becoming an attractive paradigm in the field of NLP . prompting has shown progressive performance enhancements in situations where data labels are scarce or unavailable. |
| Approach: | They propose a method that uses a weak-supervision signal to train a lightweight model without internal access to data labels. |
| Outcome: | The proposed method improves text classification accuracy with weak-supervision signal without accessing weights or gradients of the LM model or data labels. |
CLiMP: A Benchmark for Chinese Language Model Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Linguistically informed analyses of language models (LMs) contribute to understanding and improvement of such models. |
| Approach: | They introduce a corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire. |
| Outcome: | The proposed corpus of Chinese linguistic minimal pairs (CLiMP) covers 9 major Chinese linguist phenomena. |
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning. |
| Approach: | They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format. |
| Outcome: | The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks. |
Graph Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Language Models (LMs) are the workhorses of NLP, but their interplay with structured knowledge graphs (KGs) is still actively researched. |
| Approach: | They propose a Graph Language Model (GLM) that integrates the strengths of both approaches and mitigates their weaknesses. |
| Outcome: | Empirical evaluations show that the proposed model surpasses both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility. |
Exploring the Capacity of Pretrained Language Models for Reasoning about Actions and Change (2023.acl-long)
Copied to clipboard
| Challenge: | Recent transformer-based language models (LMs) provide reasoning over textual benchmarks . RAC is essential to understand and interact with the ever-changing environment . |
| Approach: | They propose to use a transformer-based language model to learn to reason over textual benchmarks. |
| Outcome: | The proposed model minimizes the influence of other linguistic requirements to focus on RAC. |
CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals (2022.findings-acl)
Copied to clipboard
| Challenge: | Using contextual universal embeddings, we train neural language models on one type of contextual data and adapts to novel context types. |
| Approach: | They propose a framework to modularize the training of neural language models that use diverse forms of context by eliminating the need to jointly train context and within-sentence encoders. |
| Outcome: | The proposed framework trains LMs on one type of contextual data and adapts to novel context types. |
How Inclusively do LMs Perceive Social and Moral Norms? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) are used in decision-making systems and as interactive assistants. |
| Approach: | They propose to prompt 11 LMs on rules-of-thumb and compare their outputs with 100 human annotators. |
| Outcome: | The proposed model is compared with 100 human annotators to find out if they are inclusive of diverse human values. |
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021.emnlp-main)
Copied to clipboard
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung
| Challenge: | GPT-3 has been used to train large-scale language models on hundreds of billion scale data. |
| Approach: | They propose a Korean variant of GPT-3 that uses Korean tokens to train in-context models. |
| Outcome: | The proposed method shows state-of-the-art zero-shot and few-shot learning on downstream tasks in Korean. |
Automatic Generation of Socratic Subquestions for Teaching Math Word Problems (2022.emnlp-main)
Copied to clipboard
| Challenge: | We hypothesize that questioning can enhance human performance and assist solvers . |
| Approach: | They propose to use large language models to generate sequential questions for math word problem-solving . they propose to apply these models to a variety of math word problems . |
| Outcome: | The proposed model improves the performance of a math word problem solver by generating more questions than other models. |
Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to the creation of large Transformer-based language models (LMs). |
| Approach: | They propose a protocol which distinguishes methods that yield desirable results . they apply this protocol to a popular debiasing method, Self-Debiase, and compare it to one called Instructive Debiaser. |
| Outcome: | The proposed protocol provides essential insights into the generalizability and interpretability of debiasing methods that may otherwise go overlooked. |
Learning from Impairment: Leveraging Insights from Clinical Linguistics in Language Modelling Research (2025.coling-main)
Copied to clipboard
| Challenge: | Using neurolinguistics and aphasiology, we examine the theoretical underpinnings of some influential linguistically motivated training approaches targeting the syntactic domain. |
| Approach: | They examine the theoretical underpinnings of linguistically motivated training approaches derived from neurolinguistics and aphasiology to develop human-like learning strategies for language models. |
| Outcome: | The proposed frameworks can be used to improve the recovery and generalization of linguistic skills in aphasia treatment and to develop human-like learning strategies. |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)
Copied to clipboard
| Challenge: | a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers . |
| Approach: | They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span . |
| Outcome: | The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings. |
Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Large Language Models (RALMs) do not consistently outperform the original retrieval-free Language Model (LM). |
| Approach: | They propose a trainable framework that can adaptively retrieve from different knowledge sources and effectively decrease unpredictable reader errors. |
| Outcome: | The proposed framework significantly improves performance over the RALM with a single retriever by significantly reducing inconsistent behaviors. |
Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer (2024.findings-naacl)
Copied to clipboard
| Challenge: | Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedically knowledge base graph, ignoring the inter-concept interactions and a concept’s local neighborhood. |
| Approach: | They propose a Graph-Augmented Multi-Objective Transformer which captures both inter-concept and intra-conception interactions from the multilingual UMLS graph. |
| Outcome: | The proposed model captures inter- and intra-concept interactions from the multilingual UMLS graph using pre-trained language models and graph neural networks. |
Wugnectives: Novel Entity Inferences of Language Models from Discourse Connectives (2026.eacl-long)
Copied to clipboard
| Challenge: | Using context + knowledge of discourse connectives to make predictions about discourse connective . |
| Approach: | They present a dataset of 8,880 stimuli that evaluates LMs’ inferences about novel entities in contexts where connectives link the entities to particular attributes. |
| Outcome: | The proposed dataset evaluates LMs’ inferences about new entities in contexts where connectives link the entities to particular attributes. |
Less is More: Summary of Long Instructions is Better for Program Synthesis (2022.emnlp-main)
Copied to clipboard
| Challenge: | Despite the success of large pre-trained language models, they show below-par performance on the larger and more complicated programming related questions. |
| Approach: | They propose to use a human-generated summary of programming questions to improve LMs' performance. |
| Outcome: | The proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on an average in terms of strict accuracy. |
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained neural language models (LMs) are prone to generating toxic language . we investigate the extent to which pretrained LMs can be prompted to generate toxic language. |
| Approach: | They propose to use a dataset to test the extent to which pretrained LMs can generate toxic language . they use toxicity-based text generation algorithms to generate toxic text from natural prompts . |
| Outcome: | The proposed model can degenerate into toxic text even from seemingly innocuous prompts. |
Is a Question Decomposition Unit All We Need? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LMs) have achieved state-of-the-art performance on many NLP benchmarks. |
| Approach: | They propose to decompose a hard question into simpler questions that are easier for models to answer. |
| Outcome: | The proposed approach significantly improves model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) by decomposing a hard question into simpler questions that are easier for models to answer. |
Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Ambiguity is embedded throughout natural language, and even simple utterances can have multiple interpretations when read in isolation. |
| Approach: | They propose a task-agnostic framework for evaluating a system’s ability to determine when to ask for clarification. |
| Outcome: | The proposed framework outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification. |
Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) excel in retrieving popular facts, but encounter difficulty with infrequent entity-relation pairs compared to retrievers. |
| Approach: | They propose to use a WiTQA dataset to explore the effects of combinations of entities and relations on LMs. |
| Outcome: | The proposed model can retain popular relations of less common entities while retaining the same popular relations. |
Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot dialogue state tracking (DST) involve embedding prompts into language models, but these methods have inherent limitations. |
| Approach: | They propose a plug-and-play architecture designed for zero-shot dialogue state tracking (DST) dual low-rank adaptation targets dialogue context processing and prompt optimization without incurring additional inference latency. |
| Outcome: | The proposed architecture outperforms baseline methods on multi-domain datasets and the MultiWOZ dataset. |
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian (2024.emnlp-main)
Copied to clipboard
Peng Liu, Lemei Zhang, Terje Farup, Even Lauvrak, Jon Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang
| Challenge: | Norwegian is under-represented within the most impressive breakthroughs in NLP tasks. |
| Approach: | they investigate the impact of existing Norwegian language models on Norwegian generation tasks . they pre-trained 4 Norwegian Open Language Models from parameter scales and architectures . |
| Outcome: | The proposed benchmark evaluates the performance of language models on Norwegian generation tasks. |
ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Language Models excel in understanding textual descriptions of proteins, but struggle to process texts. |
| Approach: | They propose a framework for Protein-to-Text Generation for Text-based Protein Understanding that integrates a PLM as its protein understanding module. |
| Outcome: | The proposed framework surpasses existing baselines and is highly efficient in protein-to-text generation. |
KoBEST: Korean Balanced Evaluation of Significant Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | a well-formulated benchmark allows objective and precise evaluation of diverse models. |
| Approach: | They propose a benchmark for Korean balanced evaluation of significant tasks that requires advanced Korean linguistic knowledge. |
| Outcome: | The proposed benchmarks are based on five Korean-language downstream tasks . the data is annotated by humans and thoroughly reviewed to guarantee high data quality. |
Safeguarding Language Models via Self-Destruct Trapdoor (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing mechanisms to restrict behavior of language models (LMs) are vulnerable to misuse and misalignment. |
| Approach: | They propose a mechanism to restrict specific behaviors in language models by exploiting hardware properties. |
| Outcome: | The proposed mechanism can be applied to trigger overflows for specific behaviors or target hardware malfunctions. |
Harnessing Black-Box Control to Boost Commonsense in LM’s Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen remarkable progress in massively Pre-Trained Language Models such as GPT-3 . however, their generated outputs lack commonsense at times . |
| Approach: | They propose a framework that steers a frozen Pre-Trained Language Model towards more commonsense generation by training an auxiliary model. |
| Outcome: | The proposed framework produces plausible outputs that incorporate concepts in a meaningful way. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
Attention weights accurately predict language representations in the brain (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In Transformer-based language models, the attention mechanism converts token embeddings into contextual embeddables that incorporate information from neighboring words. |
| Approach: | They analyze fMRI recordings of English language learners and extract attention weights from them to determine how well they can predict brain responses. |
| Outcome: | The resulting hidden state embeddings are more accurate than lexical embeddngs or RNN-based models. |
SSS: Editing Factual Knowledge in Language Models towards Semantic Sparse Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to modify LMs suffer from sub-optimal locality, where irrelevant neighborhood examples can be adversely influenced. |
| Approach: | They propose to use a model editing method to modify specific examples in LMs to improve locality and reasoning capability by directing the hidden state of edit example towards spaces where semantics are sparse. |
| Outcome: | The proposed method improves locality and reasoning capability on two datasets. |
Pretraining Language Models Using Translationese (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that large language models perform well in low-resource languages . a vast majority of languages don't have comparable data as compared to English . |
| Approach: | They propose to use Translationese as synthetic data for pre-training language models for low-resource languages. |
| Outcome: | The proposed method reduces performance of LMs trained on clean data in Indian languages . the proposed model performs better in English than in other languages, but is not comparable to English. |
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines how epistemic markers of certainty, uncertainty, or evidentiality affect LMs' attitudes toward knowledge . accuracies of expressions of high certainty and low certainty are associated with questions . prior work focused on learning the mapping between internal probabilities of a model and an ordinal output . |
| Approach: | They develop a typology of epistemic markers and inject 50 markers into questions . they find that LMs are highly sensitive to epistemical markers in prompts . |
| Outcome: | The proposed model epistemology study shows that LMs are sensitive to epistemic markers in prompts . expressions of high certainty result in 7% decrease in accuracy, while factive verbs hurt performance . |
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (D18-1)
Copied to clipboard
| Challenge: | Using recurrent neural networks to build language models for code-switched text is an important problem with implications to downstream applications such as speech recognition and machine translation. |
| Approach: | They propose a novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately and a generative model estimated using the training data. |
| Outcome: | The proposed techniques yield significant reductions in perplexity on Mandarin-English task and improve on baseline models. |
Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Analogical reasoning plays a critical role in human cognition, enabling us to understand new concepts by associating them with familiar ones. |
| Approach: | They propose to use free-form analogies to aid students in understanding scientific concepts . they also show that analogies generated by student LMs can improve their own performance . |
| Outcome: | The proposed model can help students understand scientific concepts, the authors show . |
Compressing Language Models for Specialized Domains (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) excel at tasks across diverse domains, yet require substantial computational resources during inference. |
| Approach: | They propose a calibration method to improve the in-domain performance of compressed LMs in a post-training setting. |
| Outcome: | The proposed method outperforms existing methods on domain-specific tasks while preserving general performance. |
ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for predicting chemical reactions are limited by insufficient training data and inability to utilize textual information. |
| Approach: | They propose a framework that leverages chemical knowledge encoded in language models to assist GNNs, thereby enhancing the accuracy of real-world chemical reaction predictions. |
| Outcome: | The proposed framework improves state-of-the-art GNN-based methods across chemical reaction datasets especially in out-of distribution settings. |
HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes (2022.acl-long)
Copied to clipboard
| Challenge: | a challenge in building AI systems physically present in the world is partial observability, a problem that exists when the entire state of the environment is not known or available to the system. |
| Approach: | They propose a method to infer object hallucinations for the unobserved part of the environment using large pre-trained language models. |
| Outcome: | The proposed method performs better than state-of-the-art approaches on two datasets for dRER. |
A Regex Minimization Benchmark: A PSPACE-Complete Challenge for Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) have demonstrated impressive reasoning capabilities across domains . but their ability to handle PSPACE-complete problems remains underexplored . a new benchmark for regex minimization is proposed to evaluate LMs' reasoning capabilities . |
| Approach: | They propose a benchmark for regex minimization to evaluate LMs' reasoning power . they use a million regexes paired with their minimal equivalents to evaluate their performance . |
| Outcome: | The proposed model can solve NP-complete problems, but their ability to handle PSPACE-complete ones remains underexplored. |
Discovering Knowledge-Critical Subnetworks in Pretrained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models encode implicit representations of knowledge in their parameters, but localizing these representations and disentangling them from each other remains an open problem. |
| Approach: | They propose a masking scheme that can be applied to weights and neurons to discover such subnetworks. |
| Outcome: | The proposed method can remove specific knowledge from models while minimizing adverse effects on the original model. |
Constrained Language Models for Interactive Poem Generation (2022.lrec-1)
Copied to clipboard
Andrei Popescu-Belis, Àlex Atrio, Valentin Minder, Aris Xanthos, Gabriel Luthier, Simon Mattei, Antonio Rodriguez
| Challenge: | Neural language models cannot learn constraints from data, which is scarce for a well-resourced language such as French. |
| Approach: | They propose a system that combines neural language models with constraints that can be set by users on form, topic, emotion, and rhyming scheme. |
| Outcome: | The proposed system generates poems and stanzas using LMs and rule-based algorithms . it has been demonstrated at public events and log analysis shows that users found it engaging . |
Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets (2026.eacl-long)
Copied to clipboard
| Challenge: | Novel metaphor comprehension involves complex semantic processes and linguistic creativity. |
| Approach: | They propose a cloze-style surprisal method that conditions on full-sentence context. |
| Outcome: | The proposed method shows that LM surprisal yields moderate correlations with scores/labels of metaphor novelty. |
Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) perform complex mathematical reasoning in Physics where physical context requires that any symbolic manipulation satisfies complex semantics. |
| Approach: | They systematically remove crucial context from prompts to force instances where model inference may be algebraically coherent, yet unphysical. |
| Outcome: | The proposed models perform poorly in this domain, and their reasoning is not physics-informed. |
Transductive Learning of Neural Language Models for Syntactic and Semantic Analysis (D19-1)
Copied to clipboard
| Challenge: | despite its practical advantages, transductive learning is underexplored in natural language processing . despite the simplicity of the technique, it is understudied in natural languages . |
| Approach: | They conduct an empirical study of transductive learning for neural models . they fine-tune language models on an unlabeled test set to obtain test-set-specific word representations. |
| Outcome: | The proposed method improves state-of-the-art neural models in syntactic and semantic tasks. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments (2025.naacl-long)
Copied to clipboard
Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates. |
| Approach: | They propose to derive acceptability judgments from large language models using prompts and templates to comprehensively evaluate their grammatical knowledge. |
| Outcome: | The proposed methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge. |
A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to handle missing data in tabular datasets rely on numerical estimations instead of pre-trained language models (LMs). |
| Approach: | They propose a method that leverages pre-trained language models to create contextually relevant descriptors for missing values. |
| Outcome: | The proposed approach outperforms the best-performing baselines in MCAR, MAR, and MNAR scenarios and offers a cost-effective solution for resource-constrained environments. |
Learning to Extract Structured Entities Using Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) play a pivotal role in extracting structured information from unstructured text. |
| Approach: | They propose to reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. |
| Outcome: | The proposed model outperforms baselines and human evaluations on the extracted entities. |
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-text Rationales (2023.acl-long)
Copied to clipboard
Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, Xiang Ren
| Challenge: | Existing metrics like task performance of the LM generating the rationales or similarity between generated and gold rationale are not good indicators of their human utility. |
| Approach: | They propose to use a large language model to generate rationales with better human utility by estimating its conciseness and novelty. |
| Outcome: | The proposed model can measure human utility to a better extent by estimating its usefulness in answering similar unseen instances. |
Evaluating Theory of (an uncertain) Mind: Predicting the Uncertain Beliefs of Others from Conversational Cues (2025.acl-long)
Copied to clipboard
| Challenge: | Typically, beliefs are held or not held, but there are situations where an individual's beliefs are better represented more flexibly. |
| Approach: | They propose a set of tasks that challenge language models to model the uncertainty of participants in a dialogue. |
| Outcome: | The proposed tasks show that language models can model the uncertainty of participants in a conversation. |
“Women do not have heart attacks!” Gender Biases in Automatically Generated Clinical Cases in French (2025.findings-naacl)
Copied to clipboard
| Challenge: | Healthcare professionals are increasingly including Language Models (LMs) in clinical practice. |
| Approach: | They propose to use LMs to generate clinical cases in french and an automatic linguistic gender detection tool to measure gender biases. |
| Outcome: | The proposed model over-generates cases describing male patients, creating synthetic corpora that are not consistent with documented prevalence for these disorders. |
XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML (2025.emnlp-main)
Copied to clipboard
Ernesto Luis Estevanell Valladares, Suilan Estevez-Velarde, Yoan Gutierrez, Andrés Montoyo, Ruslan Mitkov
| Challenge: | XAutoLM is a meta-learning-augmented framework that can be used to optimize discriminative and generative LM fine-tuning pipelines. |
| Approach: | They propose a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. |
| Outcome: | XAutoLM surpasses zero-shot optimizer’s peak F1 on five of six tasks, reduces mean evaluation time of pipelines by up to 4.5x, and uncovers 50% more pipelines above zero- shot Pareto front. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Fingerprinting Fine-tuned Language Models in the Wild (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing fingerprinting methods to fingerprint language models are limited to attributing organic text . however, fine-tuned LMs can generate long, coherent, and grammatically valid synthetic text. |
| Approach: | They conduct extensive experiments to demonstrate the limitations of existing fingerprinting approaches. |
| Outcome: | The proposed fingerprinting methods are limited to attributing synthetic text generated by 10 pre-trained LMs. |
Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) have significant potential for clinical prediction tasks . however, unreliable decisions can result in significant costs due to compromised patient safety and ethical concerns . |
| Approach: | They propose to combine ensembling and multi-tasking approaches to reduce uncertainty in EHRs by using multi-tapping methods. |
| Outcome: | The proposed framework reduces model uncertainty in white-box and black-box settings, and improves model transparency in both settings. |
A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on how to finetune but neglects the issue of where to fine-tune language models is expensive. |
| Approach: | They propose to use transition traces of latent representation to compute deviations (or loss) and then estimate the gain of each layer in reducing deviation (or gain). |
| Outcome: | The proposed approach outperforms baseline methods and is cost-benefit balanced. |
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. |
| Approach: | They propose a commonsense reasoning dataset with dense annotations that allows multi-tiered evaluation of machines’ reasoning process. |
| Outcome: | The proposed model can achieve high end performance but struggle to support predictions with valid supporting evidence. |
Sinhala Encoder-only Language Models and Evaluation (2025.acl-long)
Copied to clipboard
Tharindu Ranasinghe, Hansi Hettiarachchi, Nadeesha Chathurangi Naradde Vidana Pathirana, Damith Premasiri, Lasitha Uyangodage, Isuri Nanomi Arachchige, Alistair Plum, Paul Rayson, Ruslan Mitkov
| Challenge: | Recent advances in language models (LMs) have produced excellent results in many NLP tasks, but their effectiveness is highly dependent on available pre-training resources. |
| Approach: | They propose to collect the largest monolingual corpus for Sinhala and compile a benchmark and evaluate LMs on it. |
| Outcome: | The proposed language models outperform the popular multilingual LMs in downstream NLP tasks. |
IHEval: Evaluating Language Models on Following the Instruction Hierarchy (2025.naacl-long)
Copied to clipboard
Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, Meng Jiang
| Challenge: | Instruction-tuned language models (LMs) are increasingly deployed as interactive services across various applications. |
| Approach: | They propose a benchmark to evaluate models' ability to follow the instruction hierarchy by comparing their models to a set of benchmarks. |
| Outcome: | The proposed benchmark covers 3,538 examples across nine tasks covering cases where instructions in different priorities either align or conflict. |
Evaluating the Knowledge Base Completion Potential of GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC) however, their ability to do this at scale and with high accuracy remains an open question. |
| Approach: | They propose to use language models to complete a large public KB, Wikidata, with 90% precision. |
| Outcome: | The proposed models can extend Wikidata by 27M facts at 90% precision. |
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
WordTies: Measuring Word Associations in Language Models via Constrained Sampling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Word associations are widely used in psychology to provide insights on how humans perceive and understand concepts. |
| Approach: | They propose an algorithm that allows an asymmetric measurement of associated words, given a cue word as input. |
| Outcome: | The proposed algorithm shares more overlap with human associations and observes the asymmetric property of human associations. |
SemRoDe: Macro Adversarial Training to Learn Representations that are Robust to Word-Level Attacks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to defend against word-level attacks have been limited. |
| Approach: | They propose a new approach called Semantic Robust Defence to enhance the robustness of language models by aligning the domains with a distance-based objective. |
| Outcome: | The proposed approach can be generalized across word embeddings, even when they share minimal overlap at both vocabulary and word-substitution levels. |
Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice (2022.acl-long)
Copied to clipboard
| Challenge: | Probabilistic multiclass classifiers with large number of output classes are commonplace in natural language processing. |
| Approach: | They propose to use argmax to predict words from a large vocabulary in NLP models . they find that 13 out of 150 models do indeed have such unargmaxable tokens . |
| Outcome: | The proposed algorithms detect unargmaxable tokens in large language models and translation models. |
Adapting Language Models When Training on Privacy-Transformed Data (2022.lrec-1)
Copied to clipboard
| Challenge: | Using data sanitization methods to remove personal information from spoken messages is not effective because privacy-transformed data is unlikely to match the test distribution. |
| Approach: | They propose to use a data sanitization approach to remove personal information from spoken messages by replacing named entities with other words from the same class. |
| Outcome: | The proposed approach removes personal information from the spoken messages using an automatic named entity recognition method. |
Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have found that higher quality language models provide more powerful predictors of human reading behavior, but empirical support for the QP hypothesis is mixed. |
| Approach: | They propose to test the quality–power hypothesis by using surprisal language models to test their ability to predict eye tracking data. |
| Outcome: | The proposed model is based on a set of language models with a 'quality-power' hypothesis. |
Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) sometimes generate that misalign with world knowledge. |
| Approach: | They propose a method to mitigate hallucinations by restoring the LM's internal fact recall pipeline by a targeted restoration of its internal fact-recall pipeline. |
| Outcome: | The proposed method shows superior performance compared to baselines. |
Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration (2025.acl-long)
Copied to clipboard
Tianyi Bai, Ling Yang, Zhen Hao Wong, Fupeng Sun, Xinlin Zhuang, Jiahui Peng, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan, Conghui He
| Challenge: | Efficient data selection is crucial to accelerate the pretraining of language models . limited research has addressed the inherent conflicts between data selection methods . |
| Approach: | They propose a multi-actor collaborative data selection mechanism that prioritizes data based on its specific criterion and updates prioritization rules using the current state of the model. |
| Outcome: | The proposed model accelerates convergence in LM pretraining and achieves an average relative performance gain of 10.5% across multiple language model benchmarks. |
A Survey of Multilingual Reasoning in Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | This survey provides the first in-depth review of multilingual reasoning in Language Models. |
| Approach: | This survey provides the first in-depth review of multilingual reasoning in LMs. |
| Outcome: | The present study provides the first in-depth review of multilingual reasoning in LMs. |
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) capture factual knowledge by filling in the blanks of cloze-style prompts. |
| Approach: | They propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
| Outcome: | The proposed method improves the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
Multilingual Sequence-to-Sequence Models for Hebrew NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on pretrained language models for Hebrew is under-parameterized and under-trained . previous work on pretraining Hebrew LMs focused on encoder-only architectures . |
| Approach: | They propose to use sequence-to-sequence generative architectures to train large LMs in morphologically rich languages such as Hebrew. |
| Outcome: | The proposed model improves on all existing Hebrew NLP benchmarks. |
Towards Pareto-Efficient RLHF: Paying Attention to a Few High-Reward Samples with Reward Dropout (2024.findings-emnlp)
Copied to clipboard
| Challenge: | RLHF is a bi-objective problem that has the nature of a Pareto optimization . reward dropout is generalizable and most effective with non-pretrained target models . |
| Approach: | They propose a method that guarantees a Pareto improvement by leveraging reinforcement learning to fine-tune language models. |
| Outcome: | The proposed method guarantees a Pareto improvement on two benchmark datasets . it is generalizable and most effective with non-pretrained target models, saving the effort of pretraining. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) produce a chain of thought (CoT) when prompted to think step-by-step, but it is unclear whether the reasoning encoded in the CoT is faithful to the models’ parametric beliefs. |
| Approach: | They propose a framework for measuring parametric faithfulness of generated reasoning by unlearning reasoning steps (FUR) they propose to erase information contained in reasoning steps from model parameters and measure faithfulness as the resulting effect on the model’s prediction. |
| Outcome: | The proposed framework erases information contained in reasoning steps from model parameters and measures faithfulness as the resulting effect on the model’s prediction. |
Neural reality of argument structure constructions (2022.acl-long)
Copied to clipboard
| Challenge: | lexicalist linguistic theories assume argument structure is predictable from meaning of verbs . construction grammarians propose argument structure constructions distinct from verbs. |
| Approach: | They adapt psycholinguistic studies to probe for the existence of argument structure constructions in Transformer-based language models. |
| Outcome: | The proposed method could be used to probe argument structure constructions in LMs . the study shows that LM learners prefer grouping by construction over verb grouping . |
What do Large Language Models Learn beyond Language? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretraining on text confers models with useful ‘inductive biases’ for non-linguistic reasoning. |
| Approach: | They investigate whether pre-training on text confers these models with helpful ‘inductive biases’ for non-linguistic reasoning. |
| Outcome: | The proposed models outperform non-pretrained models on 19 non-linguistic tasks and show that they retain inductive biases even when training on multi-lingual text and computer code. |
Limitations of Language Models in Arithmetic and Symbolic Induction (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that large pretrained Language Models (LMs) can perform remarkably well on a range of NLP tasks but they have limitations on basic symbolic manipulation tasks such as copy, reverse, and addition. |
| Approach: | They propose to use explicit positional markers, fine-grained computation steps, and LMs with callable programs to teach large pretrained Language Models. |
| Outcome: | The proposed model can perform 100% accuracy in OOD and repeating symbols. |
Linear Transformers with Learnable Kernel Functions are Better In-Context Models (2024.acl-long)
Copied to clipboard
Yaroslav Aksenov, Nikita Balagansky, Sofia Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
| Challenge: | Current Language Models (LMs) lack essential In-Context Learning capabilities, a domain where the Transformer excels. |
| Approach: | They propose a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. |
| Outcome: | The proposed model amplifies its In-Context Learning abilities on the Pile dataset. |
Multi-lingual and Multi-cultural Figurative Language Understanding (2023.findings-acl)
Copied to clipboard
Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, Graham Neubig
| Challenge: | Figures permeate human communication, but are understudied in NLP. |
| Approach: | They create a figurative language inference dataset for seven languages associated with a variety of cultures, using cultural and regional concepts for figurativ expressions. |
| Outcome: | The results show that the most common figurative expressions are found in Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. |
Fast Forwarding Low-Rank Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Modern optimizers provide a spectacular array of tweaks to stabilize training trajectories and accelerate Stochastic Gradient Descent (SGD). |
| Approach: | They propose a fast-forward approach to accelerate large segments of SGD training . they alternate between Adam SGD for burn-in and accelerating by line search . |
| Outcome: | The proposed approach speeds up training without compromising model performance. |
Verb Knowledge Injection for Multilingual Event Processing (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that pretrainers implicitly extract a non-negligible amount of linguistic knowledge from text corpora in an unsupervised fashion. |
| Approach: | They propose to inject explicit verb knowledge into dedicated adapter modules to complement the linguistic knowledge obtained during LM-pretraining. |
| Outcome: | The proposed model improves in English event extraction tasks, while injecting verb knowledge improves other languages. |
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. |
| Approach: | They propose a retrieval-augmentation method that improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
| Outcome: | The proposed method improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |
A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods do not directly target the balance between memory and sentence processing, which is central to human working memory. |
| Approach: | They propose a dual-task paradigm that combines arithmetic computation with sentence comprehension . they show a greater accuracy gap between plausible sentences and implausible sentences . |
| Outcome: | The proposed paradigm shows that plausibility-based comprehension mirrors humans’ rational inference. |
Harnessing Toulmin’s theory for zero-shot argument explication (2024.acl-long)
Copied to clipboard
| Challenge: | To better analyze informal arguments on public forums, we propose the task of argument explication, which makes explicit a text’s argumentative structure and implicit reasoning by outputting triples of propositions claim, reason warrant. |
| Approach: | They propose to prompt generative large language models to output explicit argument components proposed by Toulmin by prompting with the theory name. |
| Outcome: | The proposed method evaluates the outputs’ coverage and validity through a human study and automatic evaluation based on prior argumentation datasets and performs robustness checks over alternative LMs, prompts, and argumentation theories. |
Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
A Distributional Perspective on Word Learning in Neural Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being studied as models of human language learners. |
| Approach: | They propose a distributional approach to word learning that captures distributional knowledge and gradient preferences for the word’s appropriateness. |
| Outcome: | The proposed signatures capture knowledge of where the target word can and cannot occur as well as gradient preferences about the word’s appropriateness. |
Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to convert pretrained dense models to MoEs are limited to ReLU-based models with natural sparsity. |
| Approach: | They propose a G-MoEfication approach for arbitrary dense models where activation sparsity assumptions no longer hold. |
| Outcome: | The proposed method reduces the inference cost associated with dense models by sparsely activating experts. |
Can Large Language Models perform Relation-based Argument Mining? (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for RbAM fail to perform satisfactorily across different datasets. |
| Approach: | They propose to use relation-based argument mining to determine agreement (support) and disagreement (attack) relations amongst textual arguments in binary and ternary settings. |
| Outcome: | The proposed method outperforms the best performing (RoBERTa-based) baseline on two open-source LLMs and with GPT-3.5-turbo on several datasets for (binary and ternary) RbAM. |
On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information. |
| Approach: | They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM. |
| Outcome: | The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient. |
Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to calibrate language models are limited in inference-time efficiency or fail to provide informative signals. |
| Approach: | They propose an activation-based calibration method, ActCab, which trains a linear layer on top of the LM’s last-layer activations. |
| Outcome: | The proposed method improves on five popular QA benchmarks and reduces the average expected calibration error (ECE) score by up to 39%. |
French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English (2022.acl-long)
Copied to clipboard
| Challenge: | We introduce 1,679 sentence pairs in French that cover stereotypes in ten types of bias like gender and age. |
| Approach: | They build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages and cultures. |
| Outcome: | The proposed dataset allows for comparability across languages while characterizing biases that are specific to each country and language. |
Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
The Lou Dataset - Exploring the Impact of Gender-Fair Language in German Text Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Gender-fair language fosters inclusion by addressing all genders or using neutral forms. |
| Approach: | They present a dataset that provides high-quality reformulations for German text classification . they find substantial label flips, reduced prediction certainty, and altered attention patterns . |
| Outcome: | The proposed dataset provides high-quality reformulations for German text classification . it finds label flips, reduced prediction certainty, and significantly altered attention patterns . |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |
CausalEval: Towards Better Causal Reasoning in Language Models (2025.naacl-long)
Copied to clipboard
Longxuan Yu, Delin Chen, Siheng Xiong, Qingyang Wu, Dawei Li, Zhikai Chen, Xiaoze Liu, Liangming Pan
| Challenge: | Large language models (LLMs) have been used for a variety of tasks, including problem-solving, decision-making, and understanding of the world. |
| Approach: | They propose a review of existing methods aimed at enhancing LMs for causal reasoning . they categorize existing methods as reasoning engines or as helpers providing knowledge or data to traditional methods . |
| Outcome: | The proposed methods perform better than existing methods on a range of tasks. |
Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025.naacl-long)
Copied to clipboard
| Challenge: | a study shows that language models can explain vowel pronunciation based on tongue positions . a visual LM can explain the relationship between vowels and tongue positions, but it is unclear whether they align textual information with visual information. |
| Approach: | They created video and image datasets from MRI data to examine if LMs associate real tongue positions with vowel articulation. |
| Outcome: | The proposed model can explain vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge. |
Learning Task Decomposition to Assist Humans in Competitive Programming (2024.acl-long)
Copied to clipboard
| Challenge: | Using language models (LMs) to solve complex problems, humans might struggle to understand and repair flawed ones. |
| Approach: | They propose to automatically decompose complex problems into simpler pieces that correspond to specific subtasks and measure their assistive value. |
| Outcome: | The proposed method enables non-experts to solve 33.3% more problems and speeds them up by 3.3x . |
Zero-shot Topical Text Classification with LLMs - an Experimental Study (2023.findings-emnlp)
Copied to clipboard
Shai Gretz, Alon Halfon, Ilya Shnayderman, Orith Toledo-Ronen, Artem Spector, Lena Dankin, Yannis Katsis, Ofir Arviv, Yoav Katz, Noam Slonim, Liat Ein-Dor
| Challenge: | Topical text classification is an ancient, yet timely research area in natural language processing. |
| Approach: | They compare the zero-shot performance of a variety of LMs over a large dataset of 23 publicly available TTC datasets. |
| Outcome: | The proposed models outperform their counterparts over a large dataset and show that they perform better in a zero-shot scenario. |
Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing Language Models lack the power to store all required knowledge, resulting in a lack of ability to infer out-of-context knowledge. |
| Approach: | They propose a Knowledge Interaction Layer that can be flexibly plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively. |
| Outcome: | The proposed model can be plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively. |
GAProtoNet: A Multi-head Graph Attention-based Prototypical Network for Interpretable Text Classification (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models with black-box nature obscure decision-making process and lack interpretability. |
| Approach: | They propose a multi-head graph attention-based prototypical network that uses a vector and prototypes to learn an interpretable prototypical representation. |
| Outcome: | The proposed model achieves superior results without sacrificing the accuracy of the original black-box LMs. |
Meta-Learning Fast Weight Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens. |
| Approach: | They propose a neural component that uses gradient updates as linear attention to improve model performance. |
| Outcome: | The proposed model can be applied at training time and learn to make good use of gradient updates. |
ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a dataset of 931 videos with 4021 code-mixed Hindi-English utterances, we find that video content with multiple modalities is more accurate and more accurate than textual content. |
| Approach: | They propose to use a dataset to analyze toxic content in video content in non-English languages by leveraging language models. |
| Outcome: | The proposed framework achieves an Accuracy and Weighted F1 score of 94.29% and 94.35% for the first time in its class. |
FACTCHECKMATE: Preemptively Detecting and Mitigating Hallucinations in LMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) hallucinate. |
| Approach: | They introduce a classifier that predicts whether LMs hallucinate based on model’s hidden states before decoding begins. |
| Outcome: | The proposed model preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate . if a hallucinomy is detected, FactCheckmate intervenes by adjusting the model’s hidden states to produce more factual outputs. |
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for long-form text outputs are prone to biases and scaling up is expensive. |
| Approach: | They propose to evaluate VLMs with VLM feedback dataset . they use 15K customized score rubrics to train Prometheus-Vision . |
| Outcome: | The proposed model shows highest correlation with human evaluators and GPT-4V among open-source models. |
Balaur: Language Model Pretraining with Lexical Semantic Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. |
| Approach: | They propose a model that models LSRs directly in the LM’s hidden states throughout pretraining by allowing them to be controlled by the model. |
| Outcome: | The proposed model improves performance on hypernymy-informed tasks and on the original LM objective. |
Exploring Contextualized Neural Language Models for Temporal Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that deep contextualized language models (LMs) can extract temporal relations between events and time expressions. |
| Approach: | They propose a temporal relation extraction technique which extracts temporal relations between events and time expressions. |
| Outcome: | The proposed method significantly improves temporal dependency parsing, the authors show . their work compares the proposed method to other methods and shows where they may fail . |
GPT-3.5 for Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work shows that GPT-3.5 struggles with several error types, including punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level. |
| Approach: | They evaluate GPT-3.5 for grammatical error correction in multiple languages . they use it to re-rank correction hypotheses generated by other GEC models . |
| Outcome: | The proposed model performs well in English and Russian, but struggles with errors in other languages. |
Continual Training of Language Models for Few-Shot Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. |
| Approach: | They propose to continuously post-train an LM with unlabeled domains to expand its knowledge without forgetting previous skills. |
| Outcome: | The proposed system improves few-shot end-task learning in these domains. |
Why Does New Knowledge Create Messy Ripple Effects in LLMs? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has focused on post-training knowledge editing (KE) for language models to ensure that knowledge remains accurate and up-to-date. |
| Approach: | They propose to use a GradSim indicator to detect when and why updated knowledge ripples in language models. |
| Outcome: | The proposed indicator GradSim shows that LMs that fail to handle ripple effects have low GradSIM. |
Self-Recognition in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for language modeling are very capable, but depend on commercial providers to build them. |
| Approach: | They propose a model-generated security question to assess self-recognition in LMs . they find no evidence of general or consistent self-reason in any examined LM . |
| Outcome: | The proposed approach can be externally administered to keep track of frontier models as it does not require access to internal model parameters or output probabilities. |
Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models (2024.findings-acl)
Copied to clipboard
Daniela Occhipinti, Michele Marchi, Irene Mondella, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim, Marco Guerini
| Challenge: | a recent study has focused on the quality of data generated by automatic methods for fine-tuning Language Models in languages less resourced than English. |
| Approach: | They investigate whether human intervention improves the quality of machine-generated dialogues . they use a large-scale dataset to fine-tune three different sizes of an LM . |
| Outcome: | The results show that human intervention can improve the quality of training data . larger models are less sensitive to data quality, while smaller models are more sensitive . |
Context Limitations Make Neural Language Models More Human-Like (2022.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) have been used in cognitive modeling and engineering studies to simulate human cognitive load during reading. |
| Approach: | They propose to constrain LMs' context access to improve their simulation of human reading behavior by incorporating syntactic biases into their context access. |
| Outcome: | The proposed model improves the simulation of human reading behavior by incorporating syntactic biases into their context access. |
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies (2024.emnlp-main)
Copied to clipboard
Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Murari Tiyyala, Nicholas Andrews, Daniel Khashabi
| Challenge: | Analogical reasoning is an important part of human communication, says a new study . a benchmark to determine analogical reasoning ability in language models is needed . |
| Approach: | They propose to benchmark analogical reasoning ability in language models by collecting 340 analogies from human writings. |
| Outcome: | The proposed benchmark aims to determine analogical reasoning ability in language models. |
To Err Is Human, but Llamas Can Learn It Too (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, we fine-tune Llama 2 LMs for error generation and find that this approach yields synthetic errors akin to human errors. |
| Approach: | They propose to fine-tune Llama 2 LMs for error generation and train GEC Llma models using these artificial errors. |
| Outcome: | The proposed approach outperforms state-of-the-art models with gains ranging between 0.8 and 6 F0.5 points across all languages tested. |
Concept-aware Data Construction Improves In-context Learning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work curating in-context learners assumes that ICL emerges from vast over-parametrization or the scale of multitask training. |
| Approach: | They propose a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. |
| Outcome: | The proposed framework makes it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations and fares comparably to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data. |
Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning is susceptible to intrinsic bias present in pre-trained language models (LMs), leading to sub-optimal performance in prompt-based zero/few-shot settings. |
| Approach: | They propose a null-input prompting method to calibrate intrinsic bias encoded in pre-trained language models (LMs) they leverage a diverse set of auto-selected null meaning inputs generated from GPT-4 to probe intrinsic bias. |
| Outcome: | The proposed method significantly improves zero/few-shot learning performance of LMs for both in-context learning and prompt-based fine-tuning (on average 9% and 2%, respectively). |
Robust Text Classification: Analyzing Prototype-Based Networks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models exhibit a drop in performance on noisy data, which can cause classifiers to incorrectly change their predictions. |
| Approach: | They propose to use Prototype-Based Networks to classify examples based on their similarity to prototypical examples of a class (prototypes) they show that PBNs offer more robustness under both targeted and static adversarial attacks. |
| Outcome: | The proposed model is robust to noise and targets both targeted and static attacks. |
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language understanding tasks. |
| Approach: | They propose a pretrained language model that jointly learns representations for NL sentences and (semi-)structured tables. |
| Outcome: | The proposed model performs best on the weakly-supervised semantic parsing benchmark WikiTableQuestions while performing competitively on the text-to-SQL dataset Spider. |
CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate grounded responses are prone to errors due to the irrelevancy of input documents. |
| Approach: | They propose a framework that leverages the insight that a robust grounded response should be consistent with information derived solely from its cited sources. |
| Outcome: | Experiments on three open-domain question-answering datasets show that the proposed framework improves performance by 1.5% to 7% without any model fine-tuning. |
REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space (2025.findings-acl)
Copied to clipboard
| Challenge: | Current approaches to address this issue involve costly dataset scrubbing or model filtering through unlearning and model editing. |
| Approach: | They propose a method for unlearning sensitive information from language models . they curate email and URL datasets and a social security number dataset . |
| Outcome: | The proposed method shows superior performance and robustness to extraction attacks on real-world datasets. |
The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing (2023.findings-acl)
Copied to clipboard
| Challenge: | Using a knowledge graph question answering task, we replace the entire SPARQL vocabulary with alternate vocabularies. |
| Approach: | They replace the entire SPARQL vocabulary with alternate vocabularies . they find absolute gains in the range of 17% on the GrailQA dataset . |
| Outcome: | The proposed substitutions show that the model performance improves on the GrailQA dataset. |
LM vs LM: Detecting Factual Errors via Cross Examination (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) generate inconsistent, non-attributable or factually incorrect text, which hinders their usability. |
| Approach: | They propose a factuality evaluation framework for LMs that is based on cross-examination to detect inconsistencies between LM and examiner. |
| Outcome: | The proposed framework outperforms existing methods and baselines on factual claims on four benchmarks. |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models with less than 100B parameters perform poorly on chain-of-thought reasoning . we aim to equip smaller LMs with the step-by-step reasoning capability . |
| Approach: | They propose to equip smaller LMs with the step-by-step reasoning capability by tuning with CoT rationales. |
| Outcome: | The proposed dataset outperforms large LMs on 4 domain-specific tasks even with demonstrations . |
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023.findings-emnlp)
Copied to clipboard
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
| Challenge: | Existing methods for image-to-text generation store all knowledge within parameters, thus requiring computational-expensive fine-tuning. |
| Approach: | They propose a Retrieval-augmented Visual Language Model that stores all the knowledge within parameters and can be used to retrieve it from the external database. |
| Outcome: | The proposed model significantly boosts performance for image-to-text generation tasks with 4x less parameters compared with baseline methods. |
Language Modeling with Latent Situations (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches for improving coherence of language models are lacking . effective generation of coherent text requires reasoning about the world that text describes. |
| Approach: | They propose a family of approaches for improving coherence in language models . they use entity states as auxiliary supervision to improve LMs' reasoning ability . |
| Outcome: | The proposed method reduces errors by up to 16% by fine-tuning and prompting . it can be applied to LMs that are pre-trained to explicitly The suitcase is open . |
Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how novel language models generate training text . large LMs and constrained decoding strategies both decrease novelty . |
| Approach: | They develop a novel search tool inspired by genomic data to find n-grams in training data. |
| Outcome: | The proposed tool can search for n-grams over a corpus in constant time w.r.t. large LMs and more constrained decoding strategies both decrease novelty. |
Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions for word probability distributions are limited and the output softmax layer is inherently limited. |
| Approach: | They propose to use the output softmax layer to compute the word probability distribution instead of using pointer networks to break the bottleneck. |
| Outcome: | The proposed method improves factCC score by 2 points in CNN/DM and XSUM dataset, and MAUVE scores by 30% in bookSum paragraph-level dataset. |
A Systematic Investigation of Commonsense Knowledge in Large Language Models (2022.emnlp-main)
Copied to clipboard
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d’Autume, Phil Blunsom, Aida Nematzadeh
| Challenge: | Recent large language models (LMs) have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. |
| Approach: | They conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre-trained language models to better understand their ability to capture commonsensical knowledge. |
| Outcome: | The proposed model can exploit surface cues and annotation artefacts without task-specific supervision and is insufficient to achieve human-level commonsense performance. |
Transforming Sequence Tagging Into A Seq2Seq Task (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained, large, generative language models have had great success in a wide range of sequence tagging and structured prediction tasks. |
| Approach: | They propose to use a new format for casting input text sentences and their output labels into the input and target of a Seq2Seq model and introduce it to test their hypothesis. |
| Outcome: | The proposed format shows to be both simpler and more effective and devoid of hallucination. |
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have shown effectiveness in literature understanding tasks, especially when tuned via contrastive learning. |
| Approach: | They propose a multi-task contrastive learning framework that enables common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. |
| Outcome: | The proposed framework outperforms state-of-the-art pre-trained language models on a comprehensive dataset. |
DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | LMs are useful in a variety of downstream applications from summarization to fact-checking, often relying on factual knowledge memorized during pre-training. |
| Approach: | They use two knowledge conflict measures and a novel dataset DYNAMICQA to examine the effect of intra-memory conflict on LMs' ability to accept contextual knowledge. |
| Outcome: | The proposed model can accept contextual knowledge with a higher degree of accuracy than models with fewer truth values. |
Can Machines Resonate with Humans? Evaluating the Emotional and Empathic Comprehension of LMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Empathy plays a pivotal role in fostering prosocial behavior, often triggered by the sharing of personal experiences through narratives. |
| Approach: | They propose to use contrastive learning with masked LMs and supervised fine-tuning with large language models to improve empathy understanding in NLP models. |
| Outcome: | The proposed methods show that there is low agreement among annotators and that cultural differences are a factor in their interpretation of empathy. |
DisCo_Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec (2026.acl-long)
Copied to clipboard
Tao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui, Yong Ma, Yingying Gao, Chao Deng, Shilei Zhang, Junlan Feng
| Challenge: | DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. |
| Approach: | They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens . |
| Outcome: | DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. |
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-hop Question Answering (MHQA) adds layers of complexity to question answering tasks. |
| Approach: | They explore how LMs respond to multi-hop questions by permuting search results under various configurations. |
| Outcome: | The proposed model outperforms decoder-only models in MHQA tasks despite being significantly smaller in size . |
Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) improves large language models by incorporating non-parametric knowledge through evidence retrieved from external sources. |
| Approach: | They propose a training-free evidence compression technique that makes retrieved evidence more familiar to the target model while seamlessly integrating parametric knowledge from the model. |
| Outcome: | The proposed technique outperforms the most recent evidence compression baselines across open-domain QA datasets while achieving high compression rates. |
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023.findings-acl)
Copied to clipboard
| Challenge: | Sentence-level representations are beneficial for various natural language processing tasks. |
| Approach: | They propose a generative embedding inversion attack that reconstructs input sequences based only on their sentence embeddeds. |
| Outcome: | The proposed model outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as original inputs. |
RARR: Researching and Revising What Language Models Say, Using Language Models (2023.acl-long)
Copied to clipboard
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu
| Challenge: | Language models (LMs) excel at many tasks but often produce unsupported or misleading content. |
| Approach: | They propose a system that finds attribution for any text generation model and post-edits it to fix unsupported content. |
| Outcome: | The proposed system improves attribution while preserving the original output. |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. |
| Approach: | They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance. |
| Outcome: | The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models. |
SEGMENT+: Long Text Processing with Short-Context Language Models (2024.emnlp-main)
Copied to clipboard
Wei Shi, Shuang Li, Kerun Yu, Jinglei Chen, Zujie Liang, Xinhui Wu, Yuxi Qian, Feng Wei, Bo Zheng, Jiaqing Liang, Jiangjie Chen, Yanghua Xiao
| Challenge: | Existing frameworks that increase context window do not guarantee robust performance across long input tasks. |
| Approach: | They propose a framework that enables language models to handle extended inputs within limited context windows efficiently. |
| Outcome: | The framework improves performance on long-document question-answering and Needle-in-a-Haystack tasks. |
Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) demonstrate exceptional capabilities on tasks, but are vulnerable to extraction attacks. |
| Approach: | They propose Privacy Protection via Optimal Parameters (POP) which induces the model to forget about some of its training data. |
| Outcome: | The proposed method outperforms the state-of-the-art in retaining LM performance on 9 classification and 4 dialogue benchmarks. |
Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to model conversational traits are costly and time consuming. |
| Approach: | They propose a method that generates diverse user profiles at decoding-time by sampling from trait-specific Language Models. |
| Outcome: | The proposed method generates diverse user profiles at decoding-time without fine-tuning. |
Machine Theory of Mind Needs Machine Validation (2025.findings-acl)
Copied to clipboard
| Challenge: | In recent years there has been an explosion of interest in studying the extent to which language models (LMs) display a theory of mind (ToM) despite the growth of evaluation tools, the extent of evidence for ToM remains unclear. |
| Approach: | They conduct a survey of 16 recent studies aimed at measuring ToM in language models and found that only half do so for patterns only a machine might exploit. |
| Outcome: | The results show that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data. |
The Overlooked Repetitive Lengthening Form in Sentiment Analysis (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on emphatic and expressive language models with informal styles, such as memes and emojis. |
| Approach: | They propose a two-stage Explainable Instruction Tuning framework that can improve LLMs' performance and explainability for RLF with limited samples. |
| Outcome: | The proposed framework can match zero-shot GPT-4 in performance but not explainability for RLF with limited samples. |
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have demonstrated impressive molecule understanding ability on 1D text-related tasks, but lack 2D graph perception, a critical ability of human professionals in comprehending molecules’ topological structures. |
| Approach: | They propose to combine a cross-modal projector and a uni-modal adapter to enable an LM to understand both text- and graph-based molecular contents via a Q-Former. |
| Outcome: | The proposed model outperforms the baselines on tasks such as molecule captioning, IUPAC name prediction, and molecule-text retrieval. |
RePair: Automated Program Repair with Process-based Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Commercial-scale language models (LMs) have taken APR to unprecedented levels, but they are limited by parameters and humans interact with them through explicit prompts. |
| Approach: | They propose a method that utilizes process supervision to improve program repair by allowing users to input feedback from compilers and test cases. |
| Outcome: | The proposed method outperforms large outcome-based generation methods and is inspired by strategies used in programming competitions. |
Steering off Course: Reliability Challenges in Steering Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior studies have evaluated a few steering methods for language models, leaving gaps in understanding their robustness. |
| Approach: | They examine three steering methods for language models to examine their reliability . they use function vectors, task vectors and DoLa to steer models toward desirable outputs . |
| Outcome: | The proposed methods show that they are not robust enough to handle large models with large parameters. |
MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. |
| Approach: | They propose a leaner approach where a pre-trained LM is augmented with a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with . vectors. |
| Outcome: | The proposed method outperforms larger LMs with full input history on a long-distance dialogue dataset and does not suffer catastrophic forgetting when adapted to new tasks. |
Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | prevailing view in language acquisition research has long held that child-directed language is more effective than adultdirected language (ADL) |
| Approach: | They propose a frequency-controlled testing methodology to enable balanced comparisons across training corpora. |
| Outcome: | The proposed method outperforms models trained on English Child-Directed Language (CDL) but it does not yield stronger generalizations for acquiring syntax. |
From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current research on bias in language models focuses on data quality, not temporal influences of data. |
| Approach: | They propose a methodology to interpret the interaction between training data and model architecture in bias propagation during language modeling. |
| Outcome: | The proposed method analyzes the interaction between training data and model architecture in bias propagation during language modeling. |
HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to fine-tuning language models use zeroth-order optimizers to conserve GPU memory. |
| Approach: | They propose a full-parameter fine-tuning strategy which updates a subset of parameters at each training step. |
| Outcome: | The proposed approach reduces the amount of gradients and optimizer state parameters residing in GPU memory at the same time, thereby reducing GPU memory usage. |
How to Compute the Probability of a Word (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models estimate a probability distribution over strings in a natural language . many recent linguistic studies have been incorrectly computing word probabilities . |
| Approach: | They propose to use the correct method to compute word probabilities . they highlight issues when relying on models that use end-of-word tokenisers . |
| Outcome: | Empirically, correcting the widespread bug affects measured outcomes in sentences and lexical optimisation analyses. |
GRACE: Discriminator-Guided Chain-of-Thought Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) can assign a high likelihood to incorrect steps . Existing models (LLMs), however, struggle with complex multi-step reasoning. |
| Approach: | They propose a stepwise decoding approach that steers the decoding process towards producing correct reasoning steps. |
| Outcome: | The proposed approach outperforms existing methods on math and symbolic reasoning tasks. |
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on self-consistency check alone . prominent LMs exhibit a tendency to produce exceedingly confident, but erroneous, assertions . |
| Approach: | They propose a sampling-based method that expands on the principle of self-consistency checking to detect hallucinations at question-level and model-level. |
| Outcome: | The proposed method outperforms the state of the art in detecting non-factual and factual statements across multiple question-answering and open-domain generation benchmarks. |
Enhanced Simultaneous Machine Translation with Word-level Policies (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies assume that operations are carried out at the subword level . a novel policy dictates whether to READ or WRITE at each step of the translation process . |
| Approach: | They propose a method to boost SiMT models using language models to address subword disparity . they propose implementing a word-level policy that dictates whether to READ or WRITE . |
| Outcome: | The proposed policy improves the performance of SiMT models by boosting them with language models . the proposed policy plays a vital role in addressing the subword disparity between LMs and SiMT systems. |
Towards Faithful Knowledge Graph Explanation Through Deep Alignment in Commonsense Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for generating faithful explanations overlook path decoding faithfulness, leading to divergence between graph encoder outputs and model predictions. |
| Approach: | They propose an algorithm to assess KG representation reliability and an LM-KG distribution-aware Alignment algorithm to improve explanation faithfulness without ground truth. |
| Outcome: | The proposed algorithm improves explanation faithfulness without ground truth and significantly improves fidelity and model performance. |
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2012) show value-aligned language models exhibit stereotypes in word association tasks . ignoring racial nuances can perpetuate subtle biases in LMs . |
| Approach: | They propose a bias mitigation strategy that incentivizes representation of racial concepts in early model layers. |
| Outcome: | The proposed approach incentivizes representation of racial concepts in early model layers . it reduces implicit bias by reducing the number of ambiguous inputs, the authors show . |
Prompting-based Synthetic Data Generation for Few-Shot Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Language models have boosted the performance of Question Answering, but data annotation is costly. |
| Approach: | They propose to use large language models to improve Question Answering performance . they argue that domain-agnostic knowledge from LMs is sufficient to create a well-curated dataset. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on few-shot Question Answering. |
Tell, Don’t Show: Leveraging Language Models’ Abstractive Retellings to Model Literary Themes (2025.findings-acl)
Copied to clipboard
| Challenge: | Literature challenges traditional bag-of-words approaches for topic modeling because narrative language focuses on immersive sensory details instead of abstractive description or exposition. |
| Approach: | They propose a topic modeling approach that prompts generative language models to *tell* what passages *show*, thereby translating narratives’ surface forms into higher-level concepts and themes. |
| Outcome: | The proposed model can translate narratives’ surface forms into higher-level concepts and themes than by running LDA alone or directly asking LMs to list topics. |
Financial Language Model Evaluation (FLaME) (2025.findings-acl)
Copied to clipboard
| Challenge: | Language Models (LMs) have demonstrated impressive capabilities with core NLP tasks in finance, but their effectiveness is difficult to assess due to gaps in evaluation methodologies. |
| Approach: | They propose to use a framework to evaluate language models against ‘reasoning-reinforced’ LMs to measure their performance on finance NLP tasks. |
| Outcome: | The proposed frameworks are open-source and provide data and data for the study. |
Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can lead to undesired consequences when misaligned with human values . previous studies have shown misalignment of LLMs with human value using expert-designed or agent-based emulated bias scenarios . |
| Approach: | They investigate whether large language models (LLMs) are misaligned with human values . they find no significant differences in understanding of HVSB between LLMs . |
| Outcome: | The results show that large language models do not have lower misalignment rates and attack success rates . the study also shows that smaller language models have the ability to explain HVSB . |
Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing plans-following language models (LLMs) are not capable of multimodal input and output, resulting in inconsistent performance on multimodal tasks. |
| Approach: | They propose a multimodal plan-following language model that integrates both textual plans and visual information to bring cross-modality to instructional tasks. |
| Outcome: | The proposed model performs well on multimodal and textual dialogue in a plan-grounded setting. |
Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has identified retrieval heads as a subset of attention heads responsible for retrieving salient information in long-context language models. |
| Approach: | They introduce a retrieval head that uses attention scores to enhance retrieval from long context . they use QRRetriever to select the most relevant parts with the highest retrieval scores . |
| Outcome: | The proposed retrieval heads outperform other retrieval-based retrieval retrievers on BEIR benchmarks. |
Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior work has shown that privacy leakage of parametric knowledge often occurs from memorized pre-training data. |
| Approach: | They propose a metric that builds on differential privacy to estimate the privacy leakage of contextual knowledge during decoding by comparing parametric and contextual knowledge. |
| Outcome: | The proposed method overestimates the privacy leakage of parametric knowledge while separating parametric and contextual knowledge. |
Evaluating the Evaluators: Are readability metrics good measures of readability? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Plain language summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. |
| Approach: | They conduct a thorough survey of literature on plain language summarization (PLS) and find that traditional readability metrics are not compared to human judgments. |
| Outcome: | The proposed language models better capture deeper measures of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. |
CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)
Copied to clipboard
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
| Challenge: | CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Approach: | They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Outcome: | The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East. |
Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. |
| Approach: | They propose to use causal interpretability methods to characterize abstract mechanisms that LMs learn to use by transferring a wh-filler-gap structure into a gap-less c++ class. |
| Outcome: | The proposed methods can characterize the abstract mechanisms that LMs learn to use, and challenge claims that they can be learned only with strong innate priors. |
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models fail due to performance degradation . existing methods fail for models fine- tuned with low-rank adaptation . |
| Approach: | They propose to constrain the LoRA subspace prior to fine-tuning to ensure that updates relevant to one task do not adversely shift outputs for others. |
| Outcome: | The proposed method can integrate with most existing merging algorithms, reducing unintended interference among tasks. |
StructAM: Enhancing Address Matching through Semantic Understanding of Structure-aware Information (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to address matching rely on string-based similarity matching or manually-designed rules. |
| Approach: | They propose a method to match unstructured addresses to standard ones in a database using pre-trained language models and graph neural networks. |
| Outcome: | The proposed method outperforms state-of-the-art methods on real-world addresses . it incorporates spatial coordinates and contextual information from the surrounding area as auxiliary guidance. |
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
Information Locality as an Inductive Bias for Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Inductive biases are inherent in every machine learning system, argues a new study . m-local entropy measures how well symbols disambiguate the next symbol . |
| Approach: | They propose a framework that captures local uncertainty of a language by quantifying how effectively preceding symbols disambiguate the next symbol. |
| Outcome: | The proposed framework captures the local uncertainty of a language by quantifying how effectively symbols disambiguate the next symbol. |
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only LMs fail to solve the path-star task above 1/D chance due to a learned shortcut that absorbs training supervision. |
| Approach: | They propose a path-star task which is a minimal example of searching over a graph with D arms rooted at a single start node and a query to generate the arm with t from s to t. |
| Outcome: | The proposed task is solvable via decoder-only LMs and its minimal nature prevents its decomposition. |
Piece of Table: A Divide-and-Conquer Approach for Selecting Subtables in Table Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to QA tables rely on independent row or column selection, fail to capture cross-row and cross-column dependencies, or attempt global reasoning. |
| Approach: | They propose a divide-and-conquer subtable selection framework that aggregates local evidence without requiring explicit global reasoning. |
| Outcome: | The proposed framework outperforms previous approaches to table QA in the noisy context. |
How Private are Language Models in Abstractive Summarization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective protection of private information is essential for knowledge dissemination in sensitive domains such as medical and legal. |
| Approach: | They perform a comprehensive study of privacy risks in LM-based summarization using closed- and four-weight models of different sizes and families. |
| Outcome: | The proposed models show that they leak personally identifiable information in their summaries, compared to human-generated summary summators, which show significantly higher privacy protection levels. |
WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | a global dataset for multi-cultural value prediction task is lacking in the computer science community . a multi-culture awareness of LMs is critical to generating safe and personalized responses . |
| Approach: | They present a global multi-cultural value prediction task using a world value survey dataset . they construct more than 20 million examples of the type "(demographic attributes, value question) answer" they show that the task is challenging for strong open and closed-source models . |
| Outcome: | The proposed model can generate a rating response to a value question based on demographic contexts on 11.1%, 25.0%, 72.2%, and 75.0% of the questions. |
Multilingual Language Models Encode Script Over Linguistic Structure (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study suggests that multilingual language models organize representations around surface form, but the nature of this internal organization remains elusive. |
| Approach: | They analyze language-associated units across different model families and scales . romanization induces near-disjoint representations that align with neither native-script inputs nor English . |
| Outcome: | The results show that multilingual language models organize representations around surface form . romanization induces near-disjoint representations that align with neither native-script inputs nor English . |
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models (LMs) generate false or unverifiable content, often known as hallucination, despite ongoing efforts to enhance their factuality. |
| Approach: | They propose a tool that measures LMs’ factuality in real-world user interactions by evaluating their factual accuracy and categorizing content units as Supported, Unsupported, or Undecidable based on Web-retrieved evidence. |
| Outcome: | The proposed evaluation pipeline measures language models’ factuality in real-world user interactions. |
Prior Prompt Engineering for Reinforcement Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on algorithms, reward shaping, and data curation, but prior prompt engineering is understudied. |
| Approach: | They investigate prior prompt engineering (pPE) in reinforcement fine-tuning . they translate five representative iPE strategies into corresponding pPE approaches . |
| Outcome: | The proposed approaches outperform iPE-prompted models on in-domain and out-of-domain benchmarks. |
Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context language models (LMs) can handle tens of thousands of tokens in a single context window. |
| Approach: | They compare two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines. |
| Outcome: | The proposed pipelines outperform more complex methods on multiple long-context QA benchmarks. |
Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work has examined the training dynamics of language models. |
| Approach: | They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge. |
| Outcome: | The proposed model size and training checkpoints influence convergence of language models under different seeds. |
An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal (2026.acl-long)
Copied to clipboard
| Challenge: | Surprisal theory claims that difficulty of sentences increases linearly with surprise . a neural LM that can explain garden-path effects cannot be built, says a new study . |
| Approach: | They propose to fine-tune neural LMs to better align surprisal-based reading-time estimates with actual reading times. |
| Outcome: | a new study shows that fine-tuned neural LMs do not overfit on held-out items . the results show that they improve predictive power for human reading times . |
On Emergent Social World Models — Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) possess astonishing abilities and prove useful for a plethora of downstream tasks, but controversy persists regarding how to conceptualize their capacities. |
| Approach: | They analyze LMs’ performance across seven subcategories of ToM abilities using a large localizer dataset than used in prior work. |
| Outcome: | The proposed models recruit shared computational mechanisms for general Theory of Mind (ToM) and language-specific pragmatic reasoning on a substantially larger localizer dataset than used in prior work. |
Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Whether language models have inductive biases favoring typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs). |
| Approach: | They extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) . they also examine the generalization ability of LMs to process unseen longer test sentences . |
| Outcome: | The proposed models better capture features of natural languages and can process unseen longer test sentences. |
Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study on mental state reasoning in language models relies on a relatively small sample of closed-source LMs. |
| Approach: | They replicate and extend published work on false belief task by assessing LM mental state reasoning behavior across 41 open-weight models. |
| Outcome: | The results show that large LMs show higher sensitivity and predictive power . they also show that humans and LM models show a bias towards attributing false beliefs . |