Papers with LM
Benchmarking Distributional Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group remains uncertain. |
| Approach: | They construct a dataset expanding beyond political values and create human baselines for this task and evaluate the extent to which an LM can align with a particular group’s opinion distribution. |
| Outcome: | The proposed model can better describe opinion distributions than simulate demographic groups. |
Complex Reasoning in Natural Language (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Recent research shows that pretrained language models are often brittle for complex reasoning tasks. |
| Approach: | They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks . |
| Outcome: | This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness . |
Regularized Training of Nearest Neighbor Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | kNN-LM uses pre-trained language models and an exhaustive knn search to improve performance. |
| Approach: | They build upon kNN-LM, which uses a pre-trained language model and a knn search through the training data to achieve state-of-the-art results. |
| Outcome: | The proposed method improves on language modeling tasks on WIKI-2 and WIKI-103. |
Generating Benchmarks for Factuality Evaluation of Language Models (2024.eacl-long)
Copied to clipboard
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham
| Challenge: | Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself and might under-represent domain specific or rare facts. |
| Approach: | They propose a method that transforms a factual corpus into a benchmark evaluating an LM's propensity to generate true facts from the corpus . |
| Outcome: | The proposed framework transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. |
Speculative Contrastive Decoding (2024.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. |
| Approach: | They propose a decoding approach that leverages predictions from smaller language models to achieve both decoding acceleration and quality improvement. |
| Outcome: | The proposed method achieves both decoding acceleration and quality improvement on four diverse language tasks. |
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing tools for text-to-image synthesis can visualize machine imaginations for a given context. |
| Approach: | They propose a framework that uses machine-generated images to guide language models in open-ended text generation. |
| Outcome: | The proposed framework is effective on open-ended text generation tasks while showing minor degeneration. |
RDRec: Rationale Distillation for LLM-based Recommendation (2024.acl-short)
Copied to clipboard
| Challenge: | Existing models that bridge users and items through textual prompts for effective semantic reasoning do not consider the underlying rationales behind interactions, such as user preferences and item attributes. |
| Approach: | They propose a rationale distillation recommender model that learns rationales generated by a larger language model (LM) by leveraging reviews related to users and items. |
| Outcome: | The proposed model achieves state-of-the-art (SOTA) performance in top-N and sequential recommendations. |
Natural Language to Code Generation in Interactive Data Science Notebooks (2023.acl-long)
Copied to clipboard
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, Charles Sutton
| Challenge: | Data scientists use computational notebooks to perform data wrangling and analytic tasks. |
| Approach: | They build a benchmark program that synthesizes programs given NL intents from users by using a Python code language model. |
| Outcome: | The proposed model outperforms public code LMs in a dataset of 1078 code generation problems using the pandas data analysis framework in data science notebooks. |
To Clarify or not to Clarify: A Comparative Analysis of Clarification Classification with Fine-Tuning, Prompt Tuning, and Prompt Engineering (2024.naacl-srw)
Copied to clipboard
| Challenge: | Xu et al., 2019) show that pre-trained language model fine-tuning and prompt tuning are better than manual prompt engineering for clarification identification. |
| Approach: | They propose to use pre-trained language model fine-tuning, prompt tuning and manual prompt engineering to model clarification identification. |
| Outcome: | The proposed model outperforms pre-trained language model fine-tuning, prompt tuning and manual prompt engineering on the task of clarification identification. |
How Important is a Language Model for Low-resource ASR? (2024.findings-acl)
Copied to clipboard
| Challenge: | Using an n-gram language model in ASR may seem obvious, but its absence in most implementations suggests otherwise. |
| Approach: | They examine whether using an n-gram language model in ASR can improve accuracy in low-resource languages. |
| Outcome: | The proposed model is absent in most implementations, but it does improve accuracy in English and Mandarin. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Dual Contrastive Learning Framework for Incremental Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In incremental learning, large models learn and refresh knowledge continuously . many approaches have been proposed to preserve knowledge from previous tasks while learning new concepts in online NLP applications. |
| Approach: | They propose a dual contrastive learning framework that fosters transferability across different tasks . they use global contrastive and task-specific learning to promote a generalized embedding space . |
| Outcome: | The proposed framework outperforms the current state-of-the-art methods on text datasets. |
Aligning Generative Language Models with Human Values (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for learning human values do not consider contextual and abstract nature of human values. |
| Approach: | They propose a reinforcement learning based method that embeds human values judgements into each step of language generation. |
| Outcome: | The proposed method improves on human values judgements and shows higher alignment performance. |
On Retrieval Augmentation and the Limitations of Language Model Training (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent efforts to improve the performance of language models (LMs) have focused on scaling up model and training data size, though with steep accompanying energy and compute resource costs. |
| Approach: | They propose to augment a language model with k-nearest neighbors retrieval on its training data to reduce its perplexity. |
| Outcome: | The proposed model reduces storage costs by over 25x compared to traditional retrieval methods for GPT-2 and Mistral 7B . |
Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024.eacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are statistical models trained to assign probability to human-generated text. |
| Approach: | They evaluate language models' ability to reproduce variability that humans exhibit in the ‘next word prediction’ task. |
| Outcome: | The language models are trained to assign probability to human-generated text . they exhibit low calibration to human uncertainty, and advise against it . |
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (P18-1)
Copied to clipboard
| Challenge: | Recent studies have shed light on the information encoded by long-term memory networks. |
| Approach: | They propose to use a neural caching model to model the role of context in an LSTM LM . they analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped . |
| Outcome: | The proposed model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context, suggesting the distant past is modeled only as a rough semantic field or topic. |
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have been shown to generate more factual responses by employing modularity in combination with retrieval. |
| Approach: | They extend the recent approach of Adolphs et al. (2021) to include internet search as a module. |
| Outcome: | The proposed method outperforms the state-of-the-art model BlenderBot 2 on open-domain knowledge-grounded conversations for the same number of parameters. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
Selective Perception: Learning Concise State Descriptions for Language Model Actors (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent large language models support longer contexts, but requiring them to process redundant or irrelevant data increases inference time and cost. |
| Approach: | They propose a method that leverages a small finetuned LM to sample the minimal set of input features that maximizes the performance of a downstream LM. |
| Outcome: | The proposed method reduces the length of LM actor input by 87% and 99% while improving task success rates by 158% and 54% on NetHack and robot planning. |
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on controlled text generation has required attribute-based fine-tuning of the base language model or restricted the parameterization of the attribute discriminator. |
| Approach: | They propose a global score-based alternative for controllable text generation that combines arbitrary pre-trained black-box models for achieving desired attributes in the generated text. |
| Outcome: | The proposed method outperforms methods that require extra training or fine-tuning . the proposed method is based on a model with energy values of a linear combination of scores from black-box models . |
Humans and transformer LMs: Abstraction drives language learning (2026.eacl-long)
Copied to clipboard
| Challenge: | lexical semantic and syntactic categories emerge using novel divergence-based metrics . |
| Approach: | They compare transformer-based language model's linguistic categories learning to exemplar-based accounts of human language acquisition. |
| Outcome: | The proposed model can be used as an existence proof for human language acquisition. |
Personal Information Parroting in Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models memorize millions of PI instances, increasing privacy risks. |
| Approach: | They develop a model that parrots 13.6% of PI verbatim on a manually curated set of 483 instances . they recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting. |
| Outcome: | The proposed model outperforms the best regex-based PI detectors on a manually curated set of 483 instances of PI. |
oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent success of pre-trained language models has spurred widespread interest in their capabilities. |
| Approach: | They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls . |
| Outcome: | The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls. |
Self-imitation Learning for Action Generation in Text-based Games (2023.eacl-main)
Copied to clipboard
| Challenge: | Text-based games are situated systems where the game agents observe textual descriptions, and generate textual commands to interact with the environment. |
| Approach: | They propose a confidence-based self-imitation model to generate action candidates for the RL agent by exploiting past valuable trajectories to adapt a pre-trained language model towards a target game. |
| Outcome: | The proposed model performs well in multiple challenging games. |
AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant success across various domains, but their application in complex decision-making tasks often necessitates intricate prompt engineering or fine-tuning. |
| Approach: | They propose a lightweight Adapter Language Model (LM) which automatically refines task comprehension based on feedback from RL agents. |
| Outcome: | The proposed framework enhances synergy between LLMs and RL feedback while maintaining generalization abilities and enhancing decision-making capabilities in downstream tasks. |
Meta-learning via Language Model In-context Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have reduced "task learning and prediction" to a simple sequence prediction problem. |
| Approach: | They propose a meta-learning method that recasts task adaptation and prediction as a sequence prediction problem. |
| Outcome: | The proposed method outperforms MAML on two classification tasks and improves on binaryClfs. |
Reflect-RL: Two-Player Online RL Fine-Tuning for LMs (2024.acl-long)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. |
| Approach: | They propose a two-player system to fine-tune an LM using SFT and online RL . they use negative example generation to enhance error-correction ability of the reflection model . |
| Outcome: | The proposed system outperforms SFT and online RL without reflection on a GPT-2 XL 1.56B model. |
Fire Burns, Sword Cuts: Commonsense Inductive Bias for Exploration in Text-based Games (2022.acl-short)
Copied to clipboard
| Challenge: | Existing RL agents are far away from solving text-based games due to their combinatorially large action spaces that hinders efficient exploration. |
| Approach: | They propose an exploration technique that injects external commonsense knowledge, via a pretrained language model, into the agent during training when the agent is the most uncertain about its next action. |
| Outcome: | The proposed method exhibits improvement on the collected game scores during the training in four out of nine games from Jericho. |
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)
Copied to clipboard
| Challenge: | Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge. |
| Approach: | They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time. |
| Outcome: | The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity. |
Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model (2025.naacl-long)
Copied to clipboard
| Challenge: | Language model (LM) stores diverse factual knowledge in their parameters, which is learned during self-supervised training on unlabeled documents. |
| Approach: | They investigate the issue of "perplexity curse" in the continued training of language model (LM) they find that all studied LMs suffer from positional bias in the training document . |
| Outcome: | The proposed model is able to extract information from multiple questions with diverse queries. |
SLENDER: Structured Outputs for SLM-based NER in Low-Resource Englishes (2025.acl-industry)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) for low-resource variants of English remains challenging, as most models are trained on datasets predominantly focused on American or British English. |
| Approach: | They propose a new output format for Named Entity Recognition (NER) that achieves a three-fold reduction in inference time compared to JSON format. |
| Outcome: | The proposed output format achieves a three-fold reduction in inference time on average compared to JSON format, which is widely used for structured outputs. |
Context Generation Improves Open Domain Question Answering (2023.findings-eacl)
Copied to clipboard
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
| Challenge: | Existing closed-book question answering methods do not fully exploit the parameterized knowledge. |
| Approach: | They propose a closed-book QA framework which uses a coarse-to-fine approach to extract the relevant knowledge and answer a question. |
| Outcome: | The proposed method outperforms open-book QA methods on three QA benchmarks. |
Do Language Models Know When They’re Hallucinating References? (2024.findings-eacl)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. |
| Approach: | They propose to use hallucinated book and article references as "model organism" of hallucinism research . authors propose queries to the language model to identify hallucinous references . |
| Outcome: | The authors show that language models can identify hallucinated references without external resources . they show that LMs often produce inconsistent author lists for hallucinos, but also accurately recall the authors of real references . |
Pretrained Bidirectional Distillation for Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on language knowledge transfer from pretrained models to neural machine translation models. |
| Approach: | They propose to use masked language pretraining to efficiently transfer bidirectional language knowledge to NMT models. |
| Outcome: | The proposed method can significantly improve machine translation performance and achieve competitive or even better results than previous methods. |
Benchmarking Long-tail Generalization with Likelihood Splits (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to test out of distribution generalization have their own strengths and weaknesses. |
| Approach: | They propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. |
| Outcome: | The proposed approach can be customized to construct meaningful splits for a wide range of tasks. |
Enhancing Financial Table and Text Question Answering with Tabular Graph and Numerical Reasoning (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing models that learn tabular structures in financial documents do not understand tables and numbers. |
| Approach: | They propose to infuse explicit tabular structures through a graph neural network to improve model's performance in question answering. |
| Outcome: | The proposed model outperforms the baseline model in low-resource settings while outperforming the graph module. |
DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining. |
| Approach: | They propose a resource-efficient and modular domain specialization by means of domain adapters in which domain knowledge is encoded. |
| Outcome: | The proposed framework extracts domain-specific terms and then uses them to build DomainCC and DomainReddit resources based on masked language modeling and response selection objectives. |
AMenDeD: Modelling Concepts by Aligning Mentions, Definitions and Decontextualised Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Contextualised Language Models (LMs) improve on word embeddings by encoding meaning of words in context. |
| Approach: | They propose to learn a unified embedding space in which all three types of representations can be integrated. |
| Outcome: | The proposed model outperforms existing approaches in ontology completion tasks. |
Why Generate When You Can Discriminate? A Novel Technique for Text Classification using Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for text classification using autoregressive language models are limited . authors propose a novel technique for text classification using autoreregressives . |
| Approach: | They propose a two-step technique for text classification using autoregressive language models . they use a set of perplexity and log-likelihood based numeric features to elicit a text instance . |
| Outcome: | The proposed technique eliminates parameter updates in LMs and does not limit training examples . it is evaluated across 5 datasets and compares with multiple competent baselines . |
Condenser: a Pre-training Architecture for Dense Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work fine-tunes deep LMs to encode text sequences into single dense vector representations, but dense encoders require a lot of data and sophisticated techniques to train and suffer in low data situations. |
| Approach: | They propose to pre-train Transformer language models (LMs) with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. |
| Outcome: | The proposed model improves on various text retrieval and similarity tasks by large margins over standard LMs. |
In-Context Retrieval-Augmented Language Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing RALM methods focus on modifying the LM architecture to facilitate incorporation of external information, complicating deployment. |
| Approach: | They propose to condition a language model on relevant documents from a grounding corpus during generation by conditioning on external knowledge sources. |
| Outcome: | The proposed method significantly improves language modeling performance and provides natural source attribution mechanism. |
RL with KL penalties is better viewed as Bayesian inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) is used in fine-tuning large language models to penalize them for undesirable features of generated sequences. |
| Approach: | They analyze challenges associated with treating a language model as an RL policy . they find that RL is equivalent to variational inference: approximating a Bayesian posterior . |
| Outcome: | The proposed approach is flawed because it turns the LM into a degenerate distribution, the authors show . they show that the proposed approach avoids the distribution collapse problem and offers a first-principles derivation for its objective. |
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022.emnlp-main)
Copied to clipboard
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi
| Challenge: | Pre-trained language models struggle with consistent reasoning, and prompting methods are often noisy and inconsistent. |
| Approach: | They propose a few-shot inference method inspired by the Socratic way of conversation that generates a tree of explanations that bear logical relations between each other and frames it as a satisfiability problem. |
| Outcome: | The proposed method achieves 20% better accuracy than state-of-the-art prompting methods and performs competitively with supervised models. |
Execution-Based Evaluation for Open-Domain Code Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ODEX is the first open-domain EXecution-based natural language (NL) to Python code generation dataset. |
| Approach: | They propose to use a dataset to extend the scope of coding queries to more realistic settings by using open-domain EXecution-based natural language (NL) to Python. |
| Outcome: | The proposed dataset has 945 NL-Code pairs and 1,707 human-written test cases. |
Neural Search Space in Gboard Decoder (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Gboard decoder uses context, a lexicon and language models to provide a user-friendly keyboard. |
| Approach: | They propose a Neural Search Space which replaces an N-gram LM with a neural network LM and dynamically constructs the search space during decoding. |
| Outcome: | The proposed system improves the quality of the decoded keyboards on various locales with acceptable latency increases. |
Sort by Structure: Language Model Ranking as Dependency Probing (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing algorithms for pre-trained language models lack performance indicators for linguistic tasks such as structured prediction. |
| Approach: | They propose to measure the degree to which labeled trees are recoverable from an LM’s contextualized embeddings by probing to rank LMs for parsing dependencies in a given language. |
| Outcome: | The proposed approach predicts the best LM choice 79% of the time using less compute than training a full parser. |
Leveraging Only the Category Name for Aspect Detection through Prompt-based Constrained Clustering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect category detection (ACD) aims to automatically identify user-concerned aspects from online reviews. |
| Approach: | They propose a method that relies on the category name of each aspect and a pretrained language model to generate constraints for clustering. |
| Outcome: | The proposed framework performs better than existing weakly supervised methods on nine benchmark datasets. |
Multi-Stage Prompting for Knowledgeable Dialogue Generation (2022.findings-acl)
Copied to clipboard
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model and large-scale knowledge bases. |
| Approach: | They propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM. |
| Outcome: | The proposed model outperforms the state-of-the-art retrieval-based model in terms of knowledge relevance and correctness by 5.8% and 5%, respectively. |
Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study (2021.acl-long)
Copied to clipboard
Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, Sunita Sarawagi
| Challenge: | Recent research in multilingual language models (LMs) has demonstrated their ability to effectively handle multiple languages in a single model. |
| Approach: | They propose to exploit relatedness among languages in a language family to overcome corpora limitations of LRLs. |
| Outcome: | The proposed model exploits relatedness among languages in a language family to overcome corpora limitations for low web-resource languages. |
Residual Learning of Neural Text Generation with n-gram Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that n-gram models can achieve satisfactory performance on a large proportion of testing cases. |
| Approach: | They propose to learn a neural LM that fits the residual between an n-gram LM and the real-data distribution. |
| Outcome: | The proposed model achieves additional performance gains over popular standalone models on three typical language tasks. |
An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for textual data augmentation (DA) are highly data-hungry and struggle to perform satisfactorily under noisy conditions. |
| Approach: | They propose to leverage a diffusion language model to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens. |
| Outcome: | The proposed method captures in-domain knowledge and generates pseudo samples by reconstructing strong label-related tokens. |
SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (2022.acl-long)
Copied to clipboard
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, Rui Zhang
| Challenge: | Existing methods to handle long text are limited due to time and memory complexity and limited input lengths. |
| Approach: | They propose a multi-stage split-then-summarize framework for long input summarization . their framework can process input text of arbitrary length by adjusting the number of stages . |
| Outcome: | The proposed framework outperforms existing methods on three long meeting summarization datasets and on a long document summarizing dataset. |
Acquiring Bidirectionality via Large and Small Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing unidirectional language models are still used for token-level classification tasks, but they lack bidirectionality. |
| Approach: | They propose to use bidirectional language models to train a small backward LM and concatenate its representations to those of an existing LM for downstream tasks. |
| Outcome: | The proposed model improves performance by more than 10 points in token-classification tasks and in rare domains. |
Efficiently and Thoroughly Anonymizing a Transformer Language Model for Dutch Electronic Health Records: a Two-Step Method (2022.lrec-1)
Copied to clipboard
| Challenge: | Neural Networks (NNs) are used to model large amounts of data, such as text data, and have shown to be very useful for language modelling. |
| Approach: | They propose to use a Dutch language model for hospital notes to anonymize a model trained on large amounts of data and publish it online. |
| Outcome: | The proposed method predicts a name-like token 0.2% of the time, compared to the original training data. |
Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection (2021.acl-long)
Copied to clipboard
| Challenge: | Emotion detection in dialogues requires the identification of thematic topics underlying a conversation, commonsense knowledge, and the intricate transition patterns between affective states. |
| Approach: | They propose a Topic-Driven Knowledge-Aware Transformer model that integrates topic representation and commonsense knowledge from ATOMIC for dialogue emotion detection. |
| Outcome: | The proposed model outperforms state-of-the-art models on four dialogue datasets . it can detect topics which help distinguish emotion categories, the authors show . |
Crafting In-context Examples according to LMs’ Parametric Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | In-context examples can improve the performance of knowledge-rich tasks such as question answering by triggering a language model to surface information stored in its parametric knowledge. |
| Approach: | They propose to construct in-context example sets based on model's parametric knowledge by prompting models with 'unknown' examples. |
| Outcome: | The proposed model can perform better on in-context examples in three multi-answer question answering datasets, and prompting with ‘unknown’ examples decreases the performance. |
SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization are limited due to limited training data. |
| Approach: | They propose to re-use existing multilingual summarization and translation pipelines to perform cross-lingual summaries in a sequence. |
| Outcome: | The proposed approach outperforms existing methods in many languages with only 10% of the fine-tuning samples. |
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In (2023.acl-long)
Copied to clipboard
| Challenge: | Prior work on retrieval augmentation fine-tuned the retriever and the LM, making them closely coupled. |
| Approach: | They propose a generic retrieval plug-in that can be used to fine-tune retrieval augmentation and a LM to learn a user's preferences. |
| Outcome: | The proposed retriever improves the generalization of large language models on the MMLU and PopQA datasets by learning LM’s preferences from a known source LM . |
ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to data-to-text generation require limited training examples . a data-based approach is based on a set of pre-trained language models with optional finetuning. |
| Approach: | They propose a data-to-text generation task that makes use of any given (or no) examples. |
| Outcome: | The proposed approach improves on baselines on a dataset with zero/few/full-shot settings. |
Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical knowledge is encoded in large pre-trained language models (LMs) this is done through supervised classification tasks to predict the grammamatical properties of a span using only the token representations coming from the LM encoder. |
| Approach: | They propose to use a supervised 'edge probing' task to detect grammatical knowledge in large pre-trained language models (LMs) this is done by encoding grammamatical properties using only token representations coming from the LM encoder. |
| Outcome: | The proposed model performs well when fine-tuned or in adversarial situations where the model is forced to learn wrong correlations. |
“It doesn’t look good for a date”: Transforming Critiques into Preferences for Conversational Recommendation Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Conversational recommendation systems (CRSs) aim to refine options over multiple turns of a conversation, but they are not as flexible as real conversations. |
| Approach: | They propose a method for transforming a user critique into a positive preference . they use a large neural language model to perform critique-to-preference transformation . |
| Outcome: | The proposed method improves recommendations in restaurant domain using a new dataset of restaurant critiques. |
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on question answering models relies on retrieved documents for provenance, but recent studies show that models can retain vast amounts of factual knowledge . retrieval-based generation approaches combine parametric knowledge sources with a large number of retrieved evidence documents, achieving state-of-the-art performance on open retrieval datasets. |
| Approach: | They propose to use parametric and parametric knowledge to generate free-form questions from retrieved evidence documents. |
| Outcome: | The proposed model can use parametric and parametric knowledge to generate free-form answers from retrieved evidence documents. |
Generating EDU Extracts for Plan-Guided Summary Re-Ranking (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to generate summary candidates for re-ranking produce redundant, and often low quality, content. |
| Approach: | They propose a method to generate candidates for re-ranking that addresses these issues by grounding each abstract on its own unique content plan and creating distinct plan-guided abstracts using a model's top beam. |
| Outcome: | The proposed method outperforms baseline decoding methods on CNN, NYT, and Xsum and shows that prompting GPT-3 to follow EDU plans outperformed sampling-based methods by 1.05 points. |
BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate LMs rely on objective function and are therefore limited to masked or causal LM types. |
| Approach: | They propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. |
| Outcome: | The proposed framework can probe for knowledge across different LM types. |
MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing UMR methods can be broadly divided into two categories: early-fusion approaches, such as Marvel, which projects visual features into the language model space for integrating with text modality, and late-fusion methods, such UniVL-DR, which encode visual and textual inputs using separate encoders and obtain fused embeddings through addition. |
| Approach: | They propose to map different modalities into a shared embedding space for multi-modal retrieval. |
| Outcome: | Experiments on the WebQA+ and EVQA+ datasets show that MiMIC outperforms both early- and late-fusion approaches. |
Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have led to significant improvements on various NLP tasks in past years, but a theoretical framework for studying their relationships is still missing. |
| Approach: | They propose to use language model decomposition to represent a set of pre-trained LMs and derive a closed-form solution. |
| Outcome: | The proposed model is based on a language model decomposition (LMD) and its variants. |
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination of text lacking grounding in input data is a problem in neural data-to-text generation. |
| Approach: | They propose to combine probabilistic output of a generator language model with the output of an “text critic” classifier which guides the generation by assessing the match between the input data and the generated text. |
| Outcome: | The proposed method improves on the WebNLG and OpenDialKG benchmarks. |
Attention-based Contextual Language Model Adaptation for Speech Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing language models do not incorporate utterance level contextual information . however, for some domains like voice assistants, additional context provides a rich input signal . |
| Approach: | They propose a method for training neural speech recognition models on text and contextual data. |
| Outcome: | The proposed model reduces perplexity by 7.0% relative over a standard LM . it also improves perxicity by 2.8% relative to a state-of-the-art model for contextual LM. |
Gradient-Based Language Model Red Teaming (2024.eacl-long)
Copied to clipboard
| Challenge: | generative language models generate unsafe responses by producing adversarial prompts . red teaming is labor-intensive and difficult to scale when done by humans. |
| Approach: | They propose a red teaming method that generates diverse prompts that are likely to cause an LM to generate unsafe responses. |
| Outcome: | The proposed method is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach. |
Towards Tracing Knowledge in Language Models Back to the Training Data (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on training data attribution (TDA) may offer effective tools for identifying such examples, known as "proponents". |
| Approach: | They propose a benchmark to identify which training examples taught an LM to generate a particular factual assertion. |
| Outcome: | The proposed methods have lower proponent-retrieval precision than baselines that do not have access to the LM. |
CIE: Controlling Language Model Text Generations Using Continuous Signals (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to control language models with intent are brittle and hard to scale. |
| Approach: | They propose to use a set of LMs to fine-tune to expect a control vector that is interpolated between a "low" and a 'high' token embedding. |
| Outcome: | The proposed method can be finetuned to expect a control vector that is interpolated between a “low” and a ‘high” token embedding. |
Learning To Retrieve Prompts for In-Context Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | In-context learning is a new paradigm in natural language understanding . large pre-trained language models can be expensive to update . |
| Approach: | They propose an efficient method for retrieving training examples as prompts from annotated data and an LM. |
| Outcome: | The proposed method outperforms prior work and multiple baselines on three sequence-to-sequence tasks. |
Binarized LSTM Language Model (N18-1)
Copied to clipboard
| Challenge: | Long short-term memory (LSTM) language models are widely used for automatic speech recognition and natural language processing (NLP) however, they are limited by the word embedding layer. |
| Approach: | They propose to encode words into binary vectors and use binarized LSTM parameters to achieve high memory compression. |
| Outcome: | The proposed model achieves 11.3 compression ratio without loss of performance and 31.6 compression ratio with acceptable performance degradation. |
In-Context Learning for Few-Shot Dialogue State Tracking (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot and few-shot learning dialogue state tracking are hard and expensive. |
| Approach: | They propose an in-context learning framework for zero-shot and few-shot learning dialogue state tracking (DST) a large pretrained language model takes a test instance and a few exemplars as input and directly decodes the dialogue state . |
| Outcome: | The proposed framework outperforms state-of-the-art models in few-shot settings . it is flexible and scalable, and requires less data to adapt to new domains and scenarios . |
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-cuning techniques. |
| Approach: | They propose a pre-training architecture that learns to condense information into the dense vector through LM pre-training and a coCondenser architecture which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. |
| Outcome: | The proposed architecture reduces the need for heavy data engineering and large batch training. |
Challenges in Detoxifying Language Models (2021.findings-emnlp)
Copied to clipboard
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, Po-Sen Huang
| Challenge: | Prior work often relies on automatic evaluation of LM toxicity. |
| Approach: | They evaluate toxicity mitigation strategies for automated and human evaluations . they find human raters disagree with high automatic toxicity scores after strong toxicity reduction interventions . |
| Outcome: | The proposed methods reduce LM toxicity but lower coverage for marginalized texts . human raters disagree with high toxicity scores after strong toxicity reduction interventions . |
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) models with limited data are ineffective when the two languages are not available for one language. |
| Approach: | They propose an approach that reuses a language model that is pretrained on two languages with large monolingual data to initialize an unsupervised neural machine translation system. |
| Outcome: | The proposed method outperforms a competitive cross-lingual pretraining model in English-Macedonian (En-Mk) and English-Albanian (En Sq) it yields more than +8.3 BLEU points for all four translation directions. |
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LM) based on transformers generate plausible long texts . a discriminator-guided approach allows to apply constraints more finely and dynamically. |
| Approach: | They propose to use a discriminator-guided approach to generate constrained texts without fine-tuning the LM. |
| Outcome: | The proposed method is easier and cheaper to train than fine-tuning the LM. |
Low-resource Taxonomy Enrichment with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Taxonomies represent hierarchical relationships between terms or entities. |
| Approach: | They propose a framework for taxonomy enrichment in low-resource settings with pretrained language models as knowledge bases to compensate for the shortage of information. |
| Outcome: | The proposed framework predicts whether inputted term pairs have hierarchical relationships and leverages implicit knowledge from the LM to generate queries efficiently. |
Red Teaming Language Models with Language Models (2022.emnlp-main)
Copied to clipboard
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, Geoffrey Irving
| Challenge: | Prior work has found that language models (LMs) can harm users in hard-to-predict ways, and human annotation is expensive, limiting the number and diversity of test cases. |
| Approach: | They propose to generate test inputs using an LM itself, and use a classifier to detect harmful behavior on test input. |
| Outcome: | The proposed approach detects tens of thousands of offensive responses in a 280B parameter LM chatbot. |
SumCSE: Summary as a transformation for Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Sentence embedding models are typically trained using contrastive learning (CL) using human annotations directly or by repurposing other annotated datasets. |
| Approach: | They propose to use generative language models to generate CL data using annotated data. |
| Outcome: | The proposed method outperforms the previous best unsupervised method by 1.8 points and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark. |
Controllable Natural Language Generation with Contrastive Prefixes (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work on controllable natural language generation has focused on fine-tuning existing models or using attribute discriminators. |
| Approach: | They propose a lightweight framework for controllable GPT2 generation that utilizes attribute-specific vectors to steer natural language generation. |
| Outcome: | The proposed framework can guide generation towards desired attributes while keeping high linguistic quality. |
EnsLM: Ensemble Language Model for Data Diversity by Semantic Clustering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that data diversity affects the performance of LMs if we train a single LM over the entire dataset. |
| Approach: | They propose an autoencoding topic model with a mixture prior to perform clustering for the data. |
| Outcome: | The proposed model can learn knowledge from different samples while extracting cluster-specific features. |
Evaluating Text GANs as Language Models (N19-1)
Copied to clipboard
| Challenge: | Generative Adversarial Networks (GANs) do not suffer from the problem of exposure bias. |
| Approach: | They propose to approximate the distribution of text generated by a GAN and compare it to traditional probability-based LM metrics. |
| Outcome: | The proposed method performs significantly worse than state-of-the-art LMs on several GAN-based models and can accelerate progress in GAN text generation. |
Dialogue-oriented Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PrLMs) have shown impressive improvements for various downstream tasks including various dialogue related ones. |
| Approach: | They propose to use pre-trained language models to simulate dialogue features on general plain text with common language model training objectives to improve performance. |
| Outcome: | The proposed method is fine-tuned on three public multi-turn dialogue datasets and achieves significant and consistent improvement over the plain PrLMs. |
Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (2023.findings-acl)
Copied to clipboard
| Challenge: | In recent years, transformer-based language models (LMs) have become the default approach for many NLP tasks. |
| Approach: | They compare the performance of transformer-based language models with machine-translated corpora. |
| Outcome: | The proposed model can be improved with real data, but further research is needed. |
CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels (2023.acl-long)
Copied to clipboard
| Challenge: | Utilizing language models without internal access is becoming an attractive paradigm in the field of NLP . prompting has shown progressive performance enhancements in situations where data labels are scarce or unavailable. |
| Approach: | They propose a method that uses a weak-supervision signal to train a lightweight model without internal access to data labels. |
| Outcome: | The proposed method improves text classification accuracy with weak-supervision signal without accessing weights or gradients of the LM model or data labels. |
Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Study (2022.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive models combined with stochastic decodings are the most promising for generating CNs with regard to an unseen target of hate. |
| Approach: | They propose to use pre-trained language models to generate counter-narratives in English by adding an automatic post-editing step to refine generated CNs. |
| Outcome: | The proposed pipeline could be used to generate counter-narratives in English using pre-trained language models and stochastic decoding mechanisms. |
Can Language Models Learn Embeddings of Propositional Logic Assertions? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for automating reasoning can no longer be used for natural language tasks. |
| Approach: | They propose to use transformer-based language models to reason about knowledge expressed in natural language rather than using LMs to perform reasoning directly. |
| Outcome: | The proposed approach is feasible to some extent, but lacks robustness. |
Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing frameworks for guiding a language model in reasoning tasks are limited by their tendency to generate low-quality rationales that are repetitive and vacuous. |
| Approach: | They propose a framework that leverages a lightweight language model for guiding a black-box large LM in reasoning tasks. |
| Outcome: | The proposed framework outperforms baselines in answer prediction accuracy. |
Cross-Lingual Dependency Parsing by POS-Guided Word Reordering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual dependency parsing rely on large corpus size and cost. |
| Approach: | They propose a cross-lingual dependency parsing approach based on word reordering . they propose to train a model that transfers knowledge learned in one or multiple languages to target languages . |
| Outcome: | The proposed approach outperforms the baseline approach in Hindi and Latin by 15.3% and 6.7%. |
Geometric Signatures of Compositionality Across a Language Model’s Lifetime (2025.acl-long)
Copied to clipboard
| Challenge: | linguistic compositionality allows atoms to locally combine to create global meaning . a rich array of meanings at the level of a phrase may be explained by simple rules of composition. |
| Approach: | They propose to relate the degree of compositionality in a dataset to the intrinsic dimension of its representations under an LM, a measure of feature complexity. |
| Outcome: | The proposed model is based on a geometric view of the compositionality of a dataset and the intrinsic dimension of its representations under an LM. |
Prevent the Language Model from being Overconfident in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models are based on partial translation and a language model that predicts the next token based only on partial. |
| Approach: | They propose a Margin-based Token-level Objective and a Sentence-level Goal to maximize the Margin . they propose to model the next token based on partial translation . |
| Outcome: | The proposed approach improves translation adequacy and fluency on English-to-German, Chinese-to English and French translation tasks. |
Contextual morphologically-guided tokenization for Latin encoder models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing tokenization methods focus on information-theoretical goals like high compression and low fertility rather than linguistic goals like morphological alignment. |
| Approach: | They propose to incorporate morphological knowledge into tokenization to improve both morphology and downstream performance. |
| Outcome: | The proposed tokenization improves overall performance on four downstream tasks. |
Improving Controllable Text Generation with Position-Aware Weighted Decoding (2022.findings-acl)
Copied to clipboard
| Challenge: | Controllable text generation is a challenging task in natural language generation, which aims to generate diverse text related to specified attributes. |
| Approach: | They propose a framework that uses a lightweight controller to adjust bias signals from the controller at different decoding positions. |
| Outcome: | Experiments on positive sentiment control, topic control, and language detoxification show the proposed framework works on 4 SOTA models. |
Disentangling Questions from Query Generation for Task-Adaptive Retrieval (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work generates synthetic queries from domain-specific documents to jointly train the retriever. |
| Approach: | They propose a query generator that better adapts to wide search intents expressed in the BeIR benchmark. |
| Outcome: | The proposed query generator outperforms baselines and existing models on tasks with underexplored intents while using a query generator 47 times smaller than the previous state-of-the-art. |
Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevailing practice for adapting language models (LMs) to new domains. |
| Approach: | They propose a mask specific language model that weights the importance of domain-specific terms during fine-tuning to avoid insensitivity. |
| Outcome: | The proposed approach outperforms advanced masking strategies such as span- and PMI-based masking. |
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are a powerful transfer learning approach for knowledge graph (KG) completion. |
| Approach: | They propose a parameter-lite transfer learning approach for pretrained language models for knowledge graph (KG) completion. |
| Outcome: | The proposed model outperforms the state-of-the-art models on a knowledge graph completion benchmark by tuning 1% of the parameters. |
Critic-Guided Decoding for Controlled Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. |
| Approach: | They propose a method that combines reinforcement learning and weighted decoding to train a critic from reward models. |
| Outcome: | The proposed method generates more coherent and well-controlled texts than previous methods on three controlled generation tasks, topic control, sentiment control, and detoxification. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) identifies sentiment information related to specific aspects . previous studies have proposed using fixed examples for instruction tuning . |
| Approach: | They propose an instruction learning method with retrieval-based example ranking for ABSA tasks. |
| Outcome: | The proposed method is superior to existing models on three ABSA subtasks. |
Adaptable and Interpretable Neural MemoryOver Symbolic Knowledge (2021.naacl-main)
Copied to clipboard
| Challenge: | Past research has shown that large neural language models encode surprising amounts of factual information, but augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. |
| Approach: | They propose a neural LM that includes an interpretable neuro-symbolic KB in the form of a "fact memory" their LM improves performance on knowledge-intensive question-answering tasks, sometimes dramatically . |
| Outcome: | The proposed model improves on knowledge-intensive question-answering tasks, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art open-book model. |
Aligning to Constraints for Data-Efficient Language Model Customization (2025.findings-naacl)
Copied to clipboard
Fei Wang, Chao Shang, Shuai Wang, Sarthak Jain, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, Dan Roth
| Challenge: | General-purpose language models (LMs) are aligned to diverse user intents, but fall short when it comes to specific applications. |
| Approach: | They propose a framework that uses constraints to automatically produce supervision signals for user alignment with constraints. |
| Outcome: | The proposed framework can produce supervision signals for user alignment with constraints. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |
Class-based LSTM Russian Language Model with Linguistic Information (2020.lrec-1)
Copied to clipboard
| Challenge: | LSTM models can be used in speech recognition systems at N-best or lattice rescoring stage. |
| Approach: | They propose to use word frequency and linguistic information to generate class-based LSTM Russian language models with various numbers of classes. |
| Outcome: | The proposed models outperform word-based models and word2vec models in terms of perplexity, training time, and word error rate. |
SLING: Sino Linguistic Evaluation of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find that the accuracy of LMs is far below human performance. |
| Approach: | They propose a benchmark of Sino LINGuistics which consists of 38K sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. |
| Outcome: | The proposed model performs better on local phenomena than hierarchical models and has a strong gender and number bias. |
CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context (2024.lrec-main)
Copied to clipboard
Yangruibo Ding, Zijian Wang, Wasi U. Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, Bing Xiang
| Challenge: | Pre-trained language models (LMs) for code have shown promising performance in code completion tasks but ignore the rich semantics in other files within the same project. |
| Approach: | They propose a framework that jointly learns the in-file and cross-file context on top of code LMs and a static-analysis-based tool that locates and retrieves the most relevant project-level cross- file context for code completion. |
| Outcome: | The proposed framework improves existing code LMs with a 33.94% relative increase in exact match and 28.69% in identifier matching when the cross-file context is provided. |
Code-Mixed Text Augmentation for Latvian ASR (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study attempts to tackle code-mixed speech recognition by improving the language model of a hybrid system. |
| Approach: | They propose an inflected transliteration and phonetic transcription model for code-mixed Latvian sentences . they leverage a large human-translated English-Latvian parallel text corpus to generate synthetic Latvian phrases . |
| Outcome: | The proposed system improves on a human-translated English-Latvian parallel text corpus . the results show that the proposed system can generate code-mixed Latvian sentences . |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Language Models excel in understanding textual descriptions of proteins, but struggle to process texts. |
| Approach: | They propose a framework for Protein-to-Text Generation for Text-based Protein Understanding that integrates a PLM as its protein understanding module. |
| Outcome: | The proposed framework surpasses existing baselines and is highly efficient in protein-to-text generation. |
BECEL: Benchmark for Consistency Evaluation of Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing definitions of behavioural consistency are inconsistent across many studies. |
| Approach: | They propose a behavioural consistency model and propose behavioural taxonomy that classifies consistencies into several sub-categories. |
| Outcome: | The proposed model performs poorly on 19 test cases while exhibiting high inconsistency in many cases. |
Ask Optimal Questions: Aligning Large Language Models with Retriever’s Preference in Conversation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to perform conversational search are sub-optimal due to the limited ability to incorporate signals from the retrieval results. |
| Approach: | They propose to optimize a language model for reformulating search queries in line with retrievers’ preferences by combining a large-scale dataset with Retrievers’ Feedback. |
| Outcome: | The proposed framework outperforms existing methods on two benchmarks and surpasses the state-of-the-art methods. |
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are useful interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. |
| Approach: | They evaluate the effectiveness of up-scaling and augmenting the LM with a passage retrieval database to reduce inconsistency. |
| Outcome: | The proposed models reduce inconsistency but retrieval augmentation is more efficient. |
Modeling Preconditions in Text with a Crowd-sourced Dataset (2020.findings-emnlp)
Copied to clipboard
Heeyoung Kwon, Mahnaz Koupaee, Pratyush Singh, Gargi Sawhney, Anmol Shukla, Keerthi Kumar Kallur, Nathanael Chambers, Niranjan Balasubramanian
| Challenge: | Existing methods for modeling preconditions in text are limited due to the lack of large scale labeled data grounded in text. |
| Approach: | They propose a crowd-sourced annotation of preconditions between event pairs in newswire that is larger than prior annotations. |
| Outcome: | The proposed model outperforms existing models on two task sets, showing that precondition knowledge is not easily accessible in LM-derived representations alone. |
Improving the Efficiency of Visually Augmented Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Autoregressive Language Models lack visual knowledge due to reporting bias in textual corpora. |
| Approach: | They propose to use visual representations obtained from CLIP multimodal system to augment autoregressive language models with visual knowledge. |
| Outcome: | The proposed model outperforms VALM for visual language understanding, natural language understanding and language modeling tasks despite being significantly more efficient and simpler. |
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered . |
| Approach: | They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets . |
| Outcome: | The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark. |
Long-Tailed Question Answering in an Open World (2023.acl-long)
Copied to clipboard
| Challenge: | Existing QA approaches require access to seen tasks or do not explicitly model samples from unseen tasks. |
| Approach: | They propose an open-tailed QA model that encourages knowledge sharing between head, tail and unseen tasks and explicitly mines knowledge from a large pre-trained language model. |
| Outcome: | The proposed model outperforms the state-of-the-art on a large-scale dataset. |
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the spatial reasoning capabilities of modern language models (LMs) however, there has been limited research into the spatial thinking capabilities of LMs. |
| Approach: | They propose a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work. |
| Outcome: | The proposed method significantly improves LMs' ability on spatial understanding, which in turn helps solve two external datasets, bAbI, and boolQ. |
How to Train Long-Context Language Models (Effectively) (2025.acl-long)
Copied to clipboard
| Challenge: | a new study shows that language models can process extremely long contexts with minimal training. |
| Approach: | They use supervised fine-tuning and continued training to evaluate a language model's long-context capabilities. |
| Outcome: | The proposed model outperforms Llama-3.1-8B-Instruct on most long-context tasks . the model can process 512K tokens, one of the longest context windows of LMs . |
JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing KG-augmented models for commonsense question answering ignore the effectively fusing and reasoning over question context representations and the KG representations. |
| Approach: | They propose a novel model which combines a logical reasoning and a dynamic pruning mechanism to solve these limitations. |
| Outcome: | The proposed model improves existing models and performs interpretable reasoning on the CommonsenseQA and OpenBookQA datasets. |
On the Copying Behaviors of Pre-Training for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that initializing NMT models with pre-trained language models (LM) can speed up the model training and boost the model performance. |
| Approach: | They propose a method to control copying behaviors in NMT models by initializing them with pre-trained language models (LM) they propose to use a metric called copy ratio to control the copying behavior in decoding. |
| Outcome: | The proposed method improves translation performance by controlling copying behaviors for pre-training based models. |
Tree Prompting: Efficient Task Adaptation without Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are the main interface for applying them to new tasks, but their large size makes them difficult to fine-tune with gradients for specific downstream tasks. |
| Approach: | They propose to use training data to form a decision tree based on prompt-LM calls, with each prompt determined by the outcomes of previous calls. |
| Outcome: | The proposed method improves accuracy over competing methods and is competitive with fine-tuning. |
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) generate factually incorrect answers based on their parametric memory. |
| Approach: | They propose a retrieval-augmented large language model that can dynamically select the most suitable strategy based on query complexity. |
| Outcome: | The proposed approach improves the performance of QA systems on open-domain QA datasets. |
Evaluating Theory of (an uncertain) Mind: Predicting the Uncertain Beliefs of Others from Conversational Cues (2025.acl-long)
Copied to clipboard
| Challenge: | Typically, beliefs are held or not held, but there are situations where an individual's beliefs are better represented more flexibly. |
| Approach: | They propose a set of tasks that challenge language models to model the uncertainty of participants in a dialogue. |
| Outcome: | The proposed tasks show that language models can model the uncertainty of participants in a conversation. |
StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing LM-based VC models require offline conversion from source semantics to acoustic features, limiting their deployment to real-time applications. |
| Approach: | They propose a streaming LM-based model for zero-shot voice conversion that uses a fully causal context-aware LM with a temporal-independent acoustic predictor to facilitate real-time conversion given arbitrary speaker prompts and source speech. |
| Outcome: | The proposed model achieves comparable performance to non-streaming VC systems while maintaining a fully causal context-aware LM with a temporal-independent acoustic predictor. |
Language Model Detoxification in Dialogue with Contextualized Stance Control (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on Language Model detoxification has focused on reducing the toxicity of the generation itself without consideration of the context. |
| Approach: | They propose a method to do context-dependent detoxification without taking into account the stance of the generated response. |
| Outcome: | The proposed method can learn the context-dependent stance control strategies while keeping a low self-toxicity of the underlying LM. |
DEMix Layers: Disentangling Domains for Modular Language Modeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Extensive experiments with autoregressive transformer LMs show that DEMix layers reduce test-time perplexity and increase training efficiency. |
| Approach: | They introduce a new domain expert mixture layer that enables conditioning a language model on the domain of the input text. |
| Outcome: | Experiments with 1.3B LMs show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation. |
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Exposure bias is a central problem for auto-regressive language models (LM) it is believed that teacher forcing would cause test-time generation to be incrementally distorted due to the training-generation discrepancy. |
| Approach: | They propose to quantify the impact of exposure bias in quality, diversity, consistency and consistency by using ground-truth data prefixes instead of prefix generated by the model. |
| Outcome: | The proposed model performs better when the training-generation discrepancy is removed . the model is more robust and self-recovery ability is shown to counter exposure bias. |
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022.emnlp-main)
Copied to clipboard
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo
| Challenge: | Language Models (LMs) become outdated as the world changes, a phenomenon called temporal misalignment. |
| Approach: | They propose a lifelong benchmark that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
| Outcome: | The proposed benchmark can be trained on the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
Language Models as Agent Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on collections of documents written by individual human agents to achieve specific goals in the outside world. |
| Approach: | a new study shows that language models are models of communicative intentions in a specific, narrow sense . despite recent progress, today's language models still make odd predictions and conspicuous errors . |
| Outcome: | a survey of LMs shows that they can model communicative intentions in a specific, narrow sense . despite recent progress, current models still make odd predictions and conspicuous errors . |
Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many human-centered NLP tasks focus on assessing human-attributes of a user based on their language. |
| Approach: | They evaluate different ways of representing documents and users using different LM and HuLM architectures to predict task outcomes as dynamically changing states and averaged trait-like user-level attributes. |
| Outcome: | The proposed representations predict valence, arousal, empathy, and distress as well as trait-like user-level attributes. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
WordTies: Measuring Word Associations in Language Models via Constrained Sampling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Word associations are widely used in psychology to provide insights on how humans perceive and understand concepts. |
| Approach: | They propose an algorithm that allows an asymmetric measurement of associated words, given a cue word as input. |
| Outcome: | The proposed algorithm shares more overlap with human associations and observes the asymmetric property of human associations. |
Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020.emnlp-main)
Copied to clipboard
Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, Todd Ward
| Challenge: | Existing approaches to transfer learning target data to in-domain text . prior work has adapted pre-trained LMs to specific domains . |
| Approach: | They extend the vocabulary of a pretrained language model with domain-specific terms to create synthetic tasks that help it transfer to downstream tasks. |
| Outcome: | The proposed approaches show significant performance gains on extractive reading comprehension, document ranking and duplicate question detection tasks. |
Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. |
| Approach: | They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures. |
| Outcome: | The proposed method improves the identification of high-level structures using phrase-guided masking. |
Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that DiscoGP extracts sheaves that preserve 93-100% of a model’s performance while comprising only 1-7% of the original weights and connections. |
| Approach: | They propose a framework for extracting self-contained modular units within neural language models (LMs) they use a gradient-based pruning algorithm to prune the original LM to a sparse skeleton . |
| Outcome: | The proposed framework preserves 93-100% of the original model's performance while preserving only 1-7% of the model''s original weights and connections. |
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (D19-1)
Copied to clipboard
| Challenge: | a recent study has shown that deep neural networks are effective with various tasks . a new study examines how representations of tokens evolve between layers under different learning objectives . |
| Approach: | They use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers. |
| Outcome: | The proposed model outperforms untrained models on word identity prediction tasks . the model outpersforms models trained on other linguistic tasks based on the model's objective . |
NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in knowledge base construction techniques focus on the acquisition of positive (true) KB statements, but negative (false) statements are important for discriminative reasoning. |
| Approach: | They propose a framework that ranks potential negatives in commonsense KBs using a contextual language model. |
| Outcome: | The proposed framework ranks negatives in commonsense KBs using a language model . it yields positives that are more grammatical, coherent, and informative . |
Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)
Copied to clipboard
Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas
| Challenge: | In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Approach: | They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Outcome: | The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting. |
TT-SI: Self-Improving LLM Agents with Test-Time Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for language model fine-tuning are expensive and inefficient . existing methods rarely assess whether a training sample provides novel information . |
| Approach: | They propose a test-time self-improvement algorithm that generates a sample that model struggles with . they also explore Test-Time Distillation, which leverages 'stronger supervisors' |
| Outcome: | The proposed algorithm improves performance with +5.48% absolute accuracy gain on average across benchmarks. |
REPLUG: Retrieval-Augmented Black-Box Language Models (2024.naacl-long)
Copied to clipboard
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
| Challenge: | Existing retrieval-augmented language models require access to internal representations to enhance performance. |
| Approach: | They introduce a retrieval-augmented language modeling framework that treats the language model as a black box and augments it with a tuneable retrieval model. |
| Outcome: | The proposed framework improves performance on language modeling tasks by 6.3% and 5.1%. |
Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM (2024.emnlp-main)
Copied to clipboard
| Challenge: | Contrastive decoding (CD) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. |
| Approach: | They propose a new unsupervised decoding method called Asymptotic Probability Decoding (APD) that extrapolates the probability curves from the LMs of different sizes to infer the asymptototic probabilities from an infinitely large LM. |
| Outcome: | The proposed method improves the next-token distribution of a large expert language model using a small amateur LM. |
Subword Segmental Language Modelling for Nguni Languages (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword segmentation is a standard practice in NLP, but is viewed as a preprocessing step for low-resource languages with complex morphologies. |
| Approach: | They propose a subword segmental language model that learns how to segment words while being trained for autoregressive language modelling. |
| Outcome: | The proposed model outperforms existing models on unsupervised morphological segmentation and outperfies standard subword segmenters on all 4 languages. |
Efficient AMR Parsing with CLAP: Compact Linearization with an Adaptable Parser (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers face efficiency challenges because of their large model size and computational time, which limit their accessibility within the research community. |
| Approach: | They propose a novel linearization system that simplifies encoding and reduces the number of tokens by between 40% and 50%. |
| Outcome: | The proposed system reduces the number of tokens by 40% and 50% while maintaining high performance while reducing training and inference times. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Probing for Constituency Structure in Neural Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using standard probing techniques, we examine whether contextual neural language models implicitly learn syntactic structure. |
| Approach: | They investigate to which extent contextual neural language models implicitly learn syntactic structure. |
| Outcome: | The proposed model is able to represent constituents of different categories within the neuron activations of a LM such as RoBERTa with high performance even on manipulated data. |
Word Sense Induction with Neural biLM and Symmetric Patterns (D18-1)
Copied to clipboard
| Challenge: | Existing methods for word sense induction use a language model to predict probable substitutes for target words. |
| Approach: | They propose to use a language model to predict probable substitutes for target words . they replace the ngram-based language model with a recurrent model to generate strong substitute vectors . |
| Outcome: | The proposed method surpasses the current state-of-the-art on the SemEval 2013 task by a large margin. |
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (2024.emnlp-main)
Copied to clipboard
Krista Opsahl-Ong, Michael Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, Omar Khattab
| Challenge: | Language Model Programs (LMs) require crafting prompts that are jointly effective for all modules. |
| Approach: | They propose a novel algorithm for optimizing language model (LM) prompts for all modules by using program- and data-aware techniques and stochastic mini-batch evaluation functions. |
| Outcome: | The proposed algorithm outperforms baseline optimizers on five of seven diverse LM programs by as high as 13% accuracy. |
Fixed Input Parameterization for Efficient Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that attaching prompts to the input is effective at conditioning Language Models (LMs) however, prompts are always included in the input text during inference, thus incurring substantial computational and memory overhead. |
| Approach: | They propose to inject a fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. |
| Outcome: | The proposed method can be up to 280 times more efficient in terms of FLOPs than previous approaches. |
Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for argument mining are limited by the scarcity of manually annotated data and the highly domain-dependent nature of argumentation. |
| Approach: | They propose a novel transfer learning strategy to fine tune pretrained Transformer-based Language Models on a selectively masked language modeling task and a new prompt-based strategy for inter-component relation prediction. |
| Outcome: | The proposed method outperforms existing models on both within- and out-of-domain datasets while leveraging on the discourse context. |
PKAG-DDI: Pairwise Knowledge-Augmented Language Model for Drug-Drug Interaction Event Text Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Drug-drug interactions arise when multiple drugs are administered concurrently. |
| Approach: | They propose a pairwise knowledge-augmented generative method for DDIE text generation that integrates biological functions from a knowledge set into a language model. |
| Outcome: | The proposed method outperforms existing methods in DDIE text generation on two professional datasets. |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
EsCoLA: Spanish Corpus of Linguistic Acceptability (2024.lrec-main)
Copied to clipboard
| Challenge: | Acceptability is one of the general language understanding evaluation benchmarks (GLUE) probing tasks . EsCoLA consists of 11,174 sentences and their acceptability judgements as found in well-known Spanish reference grammars. |
| Approach: | They propose to use a corpus of linguistic acceptability (ESCoLA) EsCoLA consists of 11,174 sentences and their acceptability judgements . |
| Outcome: | The proposed task is based on 11,174 sentences and their acceptability judgements as found in well-known Spanish reference grammars. |
Language Detoxification with Attribute-Discriminative Latent Space (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detoxify toxic text require excessive memory, computations and time. |
| Approach: | They propose a method to generate toxic text using an attribute-discriminative latent space. |
| Outcome: | The proposed method outperforms baselines on detoxified language and dialogue generation tasks while being time- and memory-efficient. |
On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information. |
| Approach: | They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM. |
| Outcome: | The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient. |
IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained LMs store linguistic and relational knowledge . pre-training LM models can answer "fill-in-the-blank" questions based on pre-defined relations . |
| Approach: | They propose an open information extraction benchmark for pre-trained language models . they turn pre-trained LMs into zero-shot OIE systems to examine open relational information . |
| Outcome: | The proposed benchmark outperforms state-of-the-art methods on factual OIE datasets without training sets. |
Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows the potential of building more powerful Natural Language Processing systems by composing multiple skills of LMs into pipelines. |
| Approach: | They propose to combine weight and prompt optimization strategies to optimize a modular LM pipeline. |
| Outcome: | The proposed strategies outperform optimizing weights and prompts alone by 60% and 6% on average across LMs and tasks. |
Language Model Transformers as Evaluators for Open-domain Dialogues (2020.coling-main)
Copied to clipboard
| Challenge: | Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. |
| Approach: | They propose to use transformer neural networks to predict one or more words based on an already given context to provide an efficient, automatic indication of dialogue quality. |
| Outcome: | The proposed language models show that human evaluators have a positive correlation between the output of the models and scores. |
LongTail-Swap: benchmarking language models’ abilities on rare words (2025.findings-emnlp)
Copied to clipboard
Robin Algayres, Charles-Éric Saint-James, Mahi Luthra, Jiayi Shen, Youssef Benchekroun, Dongyan Lin, Rashel Moritz, Juan Pino, Emmanuel Dupoux
| Challenge: | LongTail-Swap is a benchmark that focuses on the tail of the word distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. |
| Approach: | They introduce LongTail-Swap, a benchmark that measures the ability of language models to learn new words with very little exposure, like infants do. |
| Outcome: | The proposed benchmark measures the ability of language models to learn new words with very little exposure, like infants do. |
LM-Critic: Language Models for Unsupervised Grammatical Error Correction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work casts GEC as a translation problem using encoder-decoder models to map bad (ungrammatical) sentences into good (grammatically) sentences. |
| Approach: | They propose to use a pretrained language model to define an LM-Critic that judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. |
| Outcome: | The proposed method outperforms existing methods in both the unsupervised and supervised setting. |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |
Language Model Prior for Low-Resource Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation is based on large parallel corpora and requires expensive training and training. |
| Approach: | They propose to incorporate a LM as prior in a neural translation model (TM) they add a regularization term which pushes the output distributions to be probable under the LM prior . |
| Outcome: | The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. |
Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025.naacl-long)
Copied to clipboard
| Challenge: | a study shows that language models can explain vowel pronunciation based on tongue positions . a visual LM can explain the relationship between vowels and tongue positions, but it is unclear whether they align textual information with visual information. |
| Approach: | They created video and image datasets from MRI data to examine if LMs associate real tongue positions with vowel articulation. |
| Outcome: | The proposed model can explain vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge. |
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them (2026.acl-long)
Copied to clipboard
| Challenge: | Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. |
| Approach: | They propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs. |
| Outcome: | The proposed method improves phonological awareness across three phonology-related tasks while preserving math and general reasoning ability. |
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence (2022.emnlp-main)
Copied to clipboard
| Challenge: | researchers have posited Dungeons and Dragons as a challenge problem to test systems on various language-related capabilities. |
| Approach: | They frame Dungeons and Dragons specifically as a dialogue system challenge . they train a large language model to generate the next game turn, conditioning it on different information. |
| Outcome: | The proposed game generates the next conversational turn and predicts the state of the game given the dialogue history. |
Query-Efficient Black-Box Red Teaming via Bayesian Optimization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating test cases and querying fail to be query-efficient . generative models can be used for open-domain dialogue, prompt continuation, text-to-image generation . |
| Approach: | They propose a query-efficient method that iteratively finds diverse positive test cases leading to model failures by utilizing user input and past evaluations. |
| Outcome: | The proposed method finds a significantly larger number of diverse positive test cases under limited query budget than baseline methods. |
What Are the Odds? Language Models Are Capable of Probabilistic Reasoning (2024.emnlp-main)
Copied to clipboard
Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, Daniel McDuff
| Challenge: | Language models (LMs) are capable of remarkably complex linguistic tasks, but numerical reasoning is an area in which they struggle. |
| Approach: | They evaluate the probabilistic reasoning capabilities of language models using idealized and real-world statistical distributions. |
| Outcome: | The proposed model can make inferences about distributions, even if assumptions are incorrect or misspecified. |
Unsupervised Multi-View Post-OCR Error Correction With Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work used text generation techniques or redundancy in similar passages for OCR error correction, which is not appropriate in cases of low corpus redundancies or weak document contextual information. |
| Approach: | They propose to use a pretrained language model to reconcile different OCR views in unsupervised way so that their combination contains fewer errors than each individual view. |
| Outcome: | The proposed model can reconcile multiple OCR views so that their combined version contains fewer errors than the best OCR view. |
Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)
Copied to clipboard
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis
| Challenge: | Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text. |
| Approach: | They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. |
| Outcome: | The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains. |
Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants (2026.acl-long)
Copied to clipboard
Jaspreet Ranjit, Hyundong Justin Cho, Claire J. Smerdon, Yoonsoo Nam, Myles Phung, Jonathan May, John R. Blosnich, Swabha Swayamdipta
| Challenge: | Using language models, annotators can help develop novel suicide interventions . 85% of cases where LM predictions disagree with existing annotations are analyzed . |
| Approach: | They propose a human-in-the-loop algorithm that leverages language models as an assistant to annotators and experts to facilitate data-driven insights from NVDRS data. |
| Outcome: | The proposed algorithm can be used to support the development of novel suicide interventions . it finds that LM predictions match existing data annotations about 85% of the time . |
Pretraining Language Models with Text-Attributed Heterogeneous Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pretraining tasks for Language Models (LMs) focus on learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs. |
| Approach: | They propose a topology-aware pretraining task that explicitly considers the topological and heterogeneous information in TAHGs by optimizing an LM and an auxiliary heterogenous graph neural network. |
| Outcome: | The proposed framework explicitly considers the topological and heterogeneous information in TAHGs. |
Adapting a Language Model While Preserving its General Knowledge (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing DA-training methods do not explicitly identify what knowledge should be preserved and what should be changed by the domain corpus. |
| Approach: | They propose to use an unlabeled corpus of aparticular domain to train a pre-trained general-purpose language model to adapt the LM so that end-tasks in the domain can give improved performances. |
| Outcome: | The proposed method improves the performance of pre-trained general-purpose language models by contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and specific knowledge. |
Continual Training of Language Models for Few-Shot Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. |
| Approach: | They propose to continuously post-train an LM with unlabeled domains to expand its knowledge without forgetting previous skills. |
| Outcome: | The proposed system improves few-shot end-task learning in these domains. |
Hints on the data for language modeling of synthetic languages with transformers (2023.acl-long)
Copied to clipboard
| Challenge: | Language Models (LMs) are becoming more useful for providing representations for NLP applications. |
| Approach: | They evaluated whether the critical amount of data varies for different morphological typologies . they found that the size of the vocabulary due to morphology is directly correlated with LM perplexity . |
| Outcome: | The proposed method reduces perplexity by more than a half for a polysynthetic language like Quechua . |
Why Does New Knowledge Create Messy Ripple Effects in LLMs? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has focused on post-training knowledge editing (KE) for language models to ensure that knowledge remains accurate and up-to-date. |
| Approach: | They propose to use a GradSim indicator to detect when and why updated knowledge ripples in language models. |
| Outcome: | The proposed indicator GradSim shows that LMs that fail to handle ripple effects have low GradSIM. |
Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to control speaker-related gender inflections in ST involve dedicated model retraining on gender-labeled data. |
| Approach: | They propose to use a gender-based inference-time solution to control speaker-related gender inflections in ST by replacing the implicitly learned internal language model with gender-specific external LMs. |
| Outcome: | The proposed approach outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. |
Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models (2024.findings-acl)
Copied to clipboard
Daniela Occhipinti, Michele Marchi, Irene Mondella, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim, Marco Guerini
| Challenge: | a recent study has focused on the quality of data generated by automatic methods for fine-tuning Language Models in languages less resourced than English. |
| Approach: | They investigate whether human intervention improves the quality of machine-generated dialogues . they use a large-scale dataset to fine-tune three different sizes of an LM . |
| Outcome: | The results show that human intervention can improve the quality of training data . larger models are less sensitive to data quality, while smaller models are more sensitive . |
Logical Natural Language Generation from Open-Domain Tables (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on neural natural language generation focus on surface-level realizations with limited emphasis on logical inference. |
| Approach: | They propose a task where a model is tasked with generating natural language statements that can be logically entailed by facts in an open-domain semi-structured table. |
| Outcome: | The proposed task is based on the existing TabFact dataset with a wide range of logical/symbolic inferences. |
Concept-aware Data Construction Improves In-context Learning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work curating in-context learners assumes that ICL emerges from vast over-parametrization or the scale of multitask training. |
| Approach: | They propose a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. |
| Outcome: | The proposed framework makes it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations and fares comparably to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data. |
Cross-Modal Taxonomic Generalization in (Vision-) Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models learn from surface form to learn from more grounded evidence. |
| Approach: | They propose to use a vision-language model to learn hypernyms from images . they find that the model can recover this knowledge and generalize even when there is no hypernomia in the image. |
| Outcome: | The proposed model can recover this knowledge and generalize even when the model receives no evidence of hypernyms during training. |
A New Formulation of Zipf’s Meaning-Frequency Law through Contextual Diversity (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have examined Zipf's meaning-frequency law as a relationship between word frequency and the number of meanings based on contextualized word vectors . |
| Approach: | They propose to use word frequency as a relationship between word frequency and contextual diversity to examine Zipf's meaning-frequency law for a wider variety of words and corpora than previous studies have shown. |
| Outcome: | The proposed formulation gives a new interpretation of Zipf's meaning-frequency law and enables us to examine it for a wider variety of words and corpora than previous studies have shown. |
KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Task-oriented dialogs (TOD) require a model to generate a response that optimizes for task-related metrics. |
| Approach: | They propose a faster generation procedure that samples from independent next-word distributions and introduce a fine-grained reward function to help the model focus on learning key information in a dialog. |
| Outcome: | The proposed algorithm achieves state-of-the-art performance on an offline task with 15% training time reduction compared to a standard RL algorithm using auto-regressive generation. |
Bridging Information-Theoretic and Geometric Compression in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current language models (LMs) encode training data into finitely many variables that allow generalization to infinitely many grammatical utterances. |
| Approach: | They propose to analyze compression in language models from geometric and information-theoretic perspectives. |
| Outcome: | The proposed model can model human language in a relatively small dimension. |
Minority Positive Sampling for Switching Points - an Anecdote for the Code-Mixing Language Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Multilingual people code-mix using English phonetic typing and insertion of anglicisms in their native language. |
| Approach: | They propose to use minority positive sampling to selectively induce more sample to achieve better performance. |
| Outcome: | The proposed model performs better than other models, but switching points are the main challenge . |
Dynamic Heterogeneous-Graph Reasoning with Language Models and Knowledge Representation Learning for Commonsense Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for QA use knowledge graphs, but they ignore subgraph optimization and subgraph deepening. |
| Approach: | They propose a dynamic heterogeneous-graph reasoning method with LMs and knowledge representation learning that optimizes the structure and knowledge representing of the HKG using a two-stage pruning strategy and knowledge-representation learning. |
| Outcome: | The proposed method improves on existing methods at CommonsenseQA and OpenBookQA. |
Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how novel language models generate training text . large LMs and constrained decoding strategies both decrease novelty . |
| Approach: | They develop a novel search tool inspired by genomic data to find n-grams in training data. |
| Outcome: | The proposed tool can search for n-grams over a corpus in constant time w.r.t. large LMs and more constrained decoding strategies both decrease novelty. |
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that an adversary can extract training data from Pretrained Language Models including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses. |
| Approach: | They propose to use knowledge unlearning to reduce privacy risks for LMs by performing gradient ascent on target token sequences instead of trying to unlearn all the data at once. |
| Outcome: | The proposed method can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. |
Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries (2025.emnlp-main)
Copied to clipboard
| Challenge: | To answer one-to-many factual queries, a language model must simultaneously recall knowledge and avoid repeating previous answers. |
| Approach: | They propose a promote-then-suppress mechanism that enables LMs to recall all answers and suppress previously generated ones. |
| Outcome: | The proposed model first recalls all answers, and then suppresses previously generated ones. |
Triplet-Free Knowledge-Guided Response Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work focused on constructing ”latent” knowledge and learning how to ground it based on pseudo triplets. |
| Approach: | They propose to pretrain a response language model to measure relevance and consistency between any context and response and use search engines to collect top-ranked passages to serve as guiding knowledge without explicitly optimizing the ‘‘best’ latent knowledge. |
| Outcome: | The proposed model pretrains a response language model to measure relevance and consistency between any context and response, then uses search engines to collect the top-ranked passages to serve as the guiding knowledge without explicitly optimizing the ‘‘best’ latent knowledge. |
Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs (2023.findings-emnlp)
Copied to clipboard
Young-Suk Lee, Md Sultan, Yousef El-Kurdi, Tahira Naseem, Asim Munawar, Radu Florian, Salim Roukos, Ramón Astudillo
| Challenge: | Empirical studies with different instruction-tuned LMs show that our proposed method yields higher-quality instruction tuning data than Self-Instruct. |
| Approach: | They propose to use in-context learning techniques to train strong conversational agents . they propose to categorize and simplify ICL templates to make prompt learning easier . |
| Outcome: | Empirical results show that the proposed method yields higher-quality instruction tuning data than Self-Instruct and improves performance of both vanilla and instruction-tuned LMs. |
mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans (2024.findings-acl)
Copied to clipboard
| Challenge: | Currently, multilingual datasets are created through translation, which cannot evaluate such language-specific aspects. |
| Approach: | They propose to curate a dataset for language-specific knowledge and commonsense . they propose to use multilingual commonsensiaq to leverage language models for a more efficient construction . |
| Outcome: | The proposed method reduces the creation cost by using multilingual LMs to create QAs . the proposed approach is based on the construction process of CSQA but with language models . |
ThaiLMCut: Unsupervised Pretraining for Thai Word Segmentation (2020.lrec-1)
Copied to clipboard
Suteera Seeha, Ivan Bilan, Liliana Mamani Sanchez, Johannes Huber, Michael Matuschek, Hinrich Schütze
| Challenge: | ThaiLMCut is a semi-supervised word segmentation model for word segmenting in Thai . it uses a bi-directional character language model to leverage useful linguistic knowledge from unlabeled data. |
| Approach: | They propose a semi-supervised approach to Thai word segmentation using a character language model. |
| Outcome: | The proposed approach outperforms state-of-the-art models on the benchmark InterBEST2009. |
DisCo_Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec (2026.acl-long)
Copied to clipboard
Tao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui, Yong Ma, Yingying Gao, Chao Deng, Shilei Zhang, Junlan Feng
| Challenge: | DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. |
| Approach: | They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens . |
| Outcome: | DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. |
An Empirical Comparison of LM-based Question and Answer Generation Methods (2023.findings-acl)
Copied to clipboard
| Challenge: | Question and answer generation (QAG) is a task of generating question-answer pairs given a context. |
| Approach: | They propose to leverage sequence-to-sequence language model fine-tuning to generate question-answer pairs given a context. |
| Outcome: | The proposed model outperforms other more convoluted approaches in the end-to-end model and is computationally light at both training and inference times. |
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have demonstrated impressive molecule understanding ability on 1D text-related tasks, but lack 2D graph perception, a critical ability of human professionals in comprehending molecules’ topological structures. |
| Approach: | They propose to combine a cross-modal projector and a uni-modal adapter to enable an LM to understand both text- and graph-based molecular contents via a Q-Former. |
| Outcome: | The proposed model outperforms the baselines on tasks such as molecule captioning, IUPAC name prediction, and molecule-text retrieval. |
Representational Analysis of Binding in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has shown that LMs use a concept called Binding ID (BI) to mark entity-attribute pairs, but have not captured the information from entity activations. |
| Approach: | They propose to localize the Binding ID mechanism by localizing BI information in LMs by encoding it in a low-rank subspace. |
| Outcome: | The proposed model can infer attributes for a given entity from a container . |
A Reality Check on Context Utilisation for Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
Lovisa Hagström, Sara Vera Marjanovic, Haeun Yu, Arnav Arora, Christina Lioma, Maria Maistro, Pepa Atanasova, Isabelle Augenstein
| Challenge: | Existing studies on LM context utilisation of retrieved information have focused on synthetic text. |
| Approach: | They propose a dataset of unreliable, insufficient and difficult-to-understand contexts with real-world queries and contexts manually annotated for stance to compare them to synthetic datasets. |
| Outcome: | The proposed model outperforms synthetic datasets and exaggerates rare context characteristics, leading to inflated context utilisation results. |
RePair: Automated Program Repair with Process-based Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Commercial-scale language models (LMs) have taken APR to unprecedented levels, but they are limited by parameters and humans interact with them through explicit prompts. |
| Approach: | They propose a method that utilizes process supervision to improve program repair by allowing users to input feedback from compilers and test cases. |
| Outcome: | The proposed method outperforms large outcome-based generation methods and is inspired by strategies used in programming competitions. |
MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. |
| Approach: | They propose a leaner approach where a pre-trained LM is augmented with a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with . vectors. |
| Outcome: | The proposed method outperforms larger LMs with full input history on a long-distance dialogue dataset and does not suffer catastrophic forgetting when adapted to new tasks. |
AnyTOD: A Programmable Task-Oriented Dialog System (2023.emnlp-main)
Copied to clipboard
Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu
| Challenge: | a neuro-symbolic approach allows zero-shot adaptation to unseen tasks and domains . a neural LM keeps track of events that occur during a conversation and a symbolic program implements dialog policy is executed to recommend actions. |
| Approach: | They propose an end-to-end, zero-shot task-oriented dialog system . it is designed to adapt to unseen tasks or domains without prior training . |
| Outcome: | The proposed system can be programmed to adapt to unseen tasks without training . it reduces data collection and training requirements for enabling new TOD 1 16189 tasks . |
Substance over Style: Evaluating Proactive Conversational Coaching Agents (2025.acl-long)
Copied to clipboard
Vidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff
| Challenge: | Recent NLP research has focused on single-turn tasks with well-defined objectives or evaluation criteria. |
| Approach: | They describe five multi-turn coaching agents that exhibit distinct conversational styles and evaluate them through a user study. |
| Outcome: | The authors compare user feedback with third-person evaluations from health experts and an LM to find that stylistic components in absence of core functionality are viewed negatively. |
GRACE: Discriminator-Guided Chain-of-Thought Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) can assign a high likelihood to incorrect steps . Existing models (LLMs), however, struggle with complex multi-step reasoning. |
| Approach: | They propose a stepwise decoding approach that steers the decoding process towards producing correct reasoning steps. |
| Outcome: | The proposed approach outperforms existing methods on math and symbolic reasoning tasks. |
Composable Text Controls in Latent Space with ODEs (2023.emnlp-main)
Copied to clipboard
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, Zhiting Hu
| Challenge: | Existing approaches to composable text operations often require plug-and-play . a single LM can perform arbitrary text operation composition in the latent space . |
| Approach: | They propose an efficient approach for composable text operations in the latent space of text . they connect pretrained LMs to the laten space and adapt them to the space . |
| Outcome: | The proposed approach improves on existing methods in the latent space of text. |
Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. |
| Approach: | They propose to find a prompt that induces LMs to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. |
| Outcome: | The proposed model is able to generate a distribution as close as possible to a target given a prompt, and it can be used to approximate distributions with low or high entropy. |
From Tower to Spire: Adding the Speech Modality to a Translation-Specialist LLM (2025.findings-emnlp)
Copied to clipboard
Kshitij Ambilduke, Ben Peters, Sonal Sannigrahi, Anil Keshwani, Tsz Kin Lam, Bruno Martins, Andre Martins, Marcely Zanon Boito
| Challenge: | Spire is a speech-augmented language model capable of translating speech input into 10 languages and transcribing text input in both directions. |
| Approach: | They introduce a speech-augmented language model capable of translating speech input into 10 languages . they integrate the model into existing multilingual LMs via speech discretization . |
| Outcome: | Spire integrates speech-augmented language model into existing multilingual model using speech discretization and pre-training using only 42.5 K hours of speech. |
Non-Essential Is NEcessary: Order-agnostic Multi-hop Question Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing multi-hop question generation methods treat answer-irrelevant documents as non-essential and remove them as impurities, which can lead to a decrease in model performance. |
| Approach: | They propose a task which leverages non-essential data in the training phase to create a robust model and extract the consistent embeddings in real-world inference environments. |
| Outcome: | The proposed model can perform ranker and generator without external modules and achieves state-of-the-art on a hotpotQA dataset. |
Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior work has shown that privacy leakage of parametric knowledge often occurs from memorized pre-training data. |
| Approach: | They propose a metric that builds on differential privacy to estimate the privacy leakage of contextual knowledge during decoding by comparing parametric and contextual knowledge. |
| Outcome: | The proposed method overestimates the privacy leakage of parametric knowledge while separating parametric and contextual knowledge. |
Subjective Behaviors and Preferences in LLM: Language of Browsing (2025.emnlp-main)
Copied to clipboard
Sai Sundaresan, Harshita Chopra, Atanu R. Sinha, Koustava Goswami, Nagasai Saketh Naidu, Raghav Karan, N Anushka
| Challenge: | Large Language Models (LLMs) fuel expectations that a single trained model can effectively align with preferences of myriad users for a given task within a domain. |
| Approach: | They introduce clusterwise LM training, HeTLM, appropriate for subjective behaviors . authors say small LM outperforms large pretrained LMs; heterogeneous cluster specific set of parameters outperformed single LM . |
| Outcome: | The proposed model outperforms large pretrained or finetuned models in the domain of subjective behavior and preferences. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only LMs fail to solve the path-star task above 1/D chance due to a learned shortcut that absorbs training supervision. |
| Approach: | They propose a path-star task which is a minimal example of searching over a graph with D arms rooted at a single start node and a query to generate the arm with t from s to t. |
| Outcome: | The proposed task is solvable via decoder-only LMs and its minimal nature prevents its decomposition. |
Language Model Probabilities are Not Calibrated in Numeric Contexts (2025.acl-long)
Copied to clipboard
Charles Lovering, Michael Krumdick, Viet Dac Lai, Varshini Reddy, Seth Ebner, Nilesh Kumar, Rik Koncel-Kedziorski, Chris Tanner
| Challenge: | Using language model outputs, we find that even in simple settings, the best LMs (1) are poorly calibrated and (2) have systematic biases. |
| Approach: | They argue that language model outputs should capture natural distributions over multiple options within their textual contexts. |
| Outcome: | The proposed model outputs are calibrated to the numeric content of their contexts. |
How Persuasive Is Your Context? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, through aseries of experiments, we show that TPS captures a more nuanced notion of persuasiveness than previously proposed metrics. |
| Approach: | They introduce a targeted persuasion score to quantify how persuasive a given context is to an LM. |
| Outcome: | Empirically, the proposed model captures a more nuanced notion of persuasiveness than previously proposed metrics. |