Papers with GPT-2
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
Word Acquisition in Neural Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Language models acquire individual words during training, based on unigram token frequencies, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. |
| Approach: | They examine how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory. |
| Outcome: | The models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. |
Pragmatic inference of scalar implicature by LLMs (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Approach: | They investigate how Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Outcome: | The proposed models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. |
Unsupervised Natural Question Answering with a Small Model (D19-66)
Copied to clipboard
| Challenge: | a recent demonstration of the power of huge language models such as GPT-2 to memorise the answers to factoid questions raises questions about the extent to which knowledge is embedded directly within these large models. |
| Approach: | They propose to use unsupervised learning techniques to add knowledge explicitly without extensive training. |
| Outcome: | The proposed architecture allows for explicit addition of knowledge without extensive training. |
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation. |
| Approach: | They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. |
| Outcome: | The results show that the representations of all words are not isotropic in any layer of the contextualizing model. |
Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity (2024.starsem-1)
Copied to clipboard
| Challenge: | Aspect is a linguistic category describing how actions and events unfold over time. |
| Approach: | They propose to use semantic projections to examine whether the vector dimensions of annotated verbs reflect human linguistic distinctions. |
| Outcome: | The proposed models encode the aspects of stativity, durativity and telicity in most of their layers, while durativité is the most challenging feature. |
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models for story generation suffer from repetition, logic conflicts, and lack of long-range coherence . |
| Approach: | They propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories by multi-task learning. |
| Outcome: | The proposed model can generate more reasonable stories than state-of-the-art models, compared with existing models, showing that it can capture useful semantic and syntactic features. |
Nationality Bias in Text Generation (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that nationality biases in language models can be a factor in improving the performance of social NLP models. |
| Approach: | They propose to use a text generation model, GPT-2, to analyze how the number of internet users and the country’s economic status affects the sentiment of stories. |
| Outcome: | The proposed model accentuates biases about country-based demonyms and reduces them with the use of adversarial triggering. |
Automatic Detection of Entity-Manipulated Text using Factual Knowledge (2022.acl-short)
Copied to clipboard
| Challenge: | Current fake news detectors that exploit stylometric signals from the text are insufficient for distinguishing manipulated text from human written text. |
| Approach: | They propose a neural network detector that detects manipulated news articles by reasoning about the facts mentioned in the article. |
| Outcome: | The proposed detector outperforms the state-of-the-art detector in accuracy. |
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing (2021.naacl-main)
Copied to clipboard
| Challenge: | a study of neural language models shows that syntactic probes do not properly isolate syntax. |
| Approach: | They show that syntactic probes do not properly isolate syntax . they train two probes trained on normal data and find they perform worse . |
| Outcome: | The proposed method outperforms the baseline models on the most popular models, but their lead is reduced by 53%. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
TaKG: A New Dataset for Paragraph-level Table-to-Text Generation Enhanced with Knowledge Graphs (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing table-to-text generation benchmarks have some limitations, such as E2E and ToTTo focusing on singlesentence generation tasks. |
| Approach: | They propose a new table-to-text generation dataset called TaKG that uses a set of knowledge graphs to enhance table input. |
| Outcome: | The proposed model outperforms existing models for short-text generation tasks and shows reliable performance on long-text generated across a variety of metrics. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
On Retrieval Augmentation and the Limitations of Language Model Training (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent efforts to improve the performance of language models (LMs) have focused on scaling up model and training data size, though with steep accompanying energy and compute resource costs. |
| Approach: | They propose to augment a language model with k-nearest neighbors retrieval on its training data to reduce its perplexity. |
| Outcome: | The proposed model reduces storage costs by over 25x compared to traditional retrieval methods for GPT-2 and Mistral 7B . |
Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024.starsem-1)
Copied to clipboard
| Challenge: | BERT, RoBERTa, XLNet, and GPT-2 models effectively discern emotional connotations of words, demonstrating superior performance and greater resilience against biases. |
| Approach: | They propose to use contextualized word representations to examine how words can be used to distinguish emotional connotations across contexts. |
| Outcome: | The proposed models show that they can distinguish emotional connotations of words in different contexts. |
KuiLeiXi: a Chinese Open-Ended Text Adventure Game (2021.acl-demo)
Copied to clipboard
Yadong Xi, Xiaoxi Mao, Le Li, Lei Lin, Yanjiang Chen, Shuhan Yang, Xuhan Chen, Kailun Tao, Zhi Li, Gongzheng Li, Lin Jiang, Siyan Liu, Zeng Zhao, Minlie Huang, Changjie Fan, Zhipeng Hu
| Challenge: | Recent advances in pre-trained language models have made it possible to generate human-like text. |
| Approach: | They propose to integrate an open-ended text adventure game in Chinese, named KuiLeiXi, where players interact with the AI until the plot goals are reached. |
| Outcome: | The proposed game lacks incentives and relies on players to explore on their own. |
Paraphrasing with Large Language Models (D19-56)
Copied to clipboard
| Challenge: | Recent work has shown large language models are adept at text generation and fine-tuning for downstream NLP tasks. |
| Approach: | They propose a system that generates paraphrased examples in autoregressive fashion using a neural network without the need for techniques such as top-k word selection or beam search. |
| Outcome: | The proposed system generates paraphrased examples in autoregressive fashion without the need for techniques such as top-k word selection or beam search. |
Naturalistic Causal Probing for Morpho-Syntax (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for probing are limited and lack understanding of their limitations and weaknesses. |
| Approach: | They propose a strategy for input-level intervention on naturalistic sentences . they use morpho-syntactic features of a sentence to intervene on the rest of the sentence . |
| Outcome: | The proposed approach allows for input-level intervention on naturalistic sentences while keeping the rest of the sentence unchanged. |
Investigating Wit, Creativity, and Detectability of Large Language Models in Domain-Specific Writing Style Adaptation of Reddit’s Showerthoughts (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have shown the ability to generate content that is difficult or impossible to distinguish from human writing. |
| Approach: | They compare GPT-2 and GPT-Neo fine-tuned on Reddit data and GTP-3.5 invoked in a zero-shot manner, against human-authored texts. |
| Outcome: | The proposed model can generate short, creative texts that are difficult to distinguish from human writing, but human evaluators rate them worse than the model. |
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2020.tacl-1)
Copied to clipboard
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman
| Challenge: | Recent studies have examined how linguistic knowledge of language models (LMs) varies across English phenomena. |
| Approach: | They propose a benchmark to evaluate linguistic knowledge of language models on major grammatical phenomena in English. |
| Outcome: | The proposed benchmark evaluates the linguistic knowledge of language models on major grammatical phenomena in English. |
Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work focuses on generating sentences satisfying pre-specified attributes such as topic and sentiment, yet suffers from increases in storage and inference time. |
| Approach: | They propose a method that uses a pre-trained continuous vector to generate a fixed pre-trainable language model to satisfy a specified attribute. |
| Outcome: | The proposed model can achieve improvements on eleven attribute-specific generation tasks with 0.08% extra training parameters. |
Comparing Probabilistic, Distributional and Transformer-Based Models on Logical Metonymy Interpretation (2020.aacl-main)
Copied to clipboard
| Challenge: | Logical metonymies are type clashes between an event-selecting verb and an entity-denoting noun . they are typically interpreted by inferring a hidden event on the basis of contextual cues . |
| Approach: | They propose to use probabilistic and distributional models to model logical metonymy interpretation . they compare models with the best Transformer-based models and some traditional distributional ones . |
| Outcome: | The proposed models perform well on a complex scenario, but low performance on some datasets suggests that logical metonymy is still a challenging phenomenon for computational modeling. |
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour (2022.aacl-short)
Copied to clipboard
| Challenge: | Language models trained on text-only corpora have no direct access to the physical world and thus suffer from reporting bias. |
| Approach: | They investigate reporting bias from the perspective of colour in larger language models such as PaLM and GPT-3. |
| Outcome: | The proposed models outperform smaller models on the basis of colour and more closely track human judgements than smaller models. |
“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility (2023.eacl-main)
Copied to clipboard
Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, Chitta Baral
| Challenge: | Recent work has found that large-scale language models lack commonsense reasoning ability . a dataset evaluating large-level language models is needed to evaluate their understanding of feasibility . |
| Approach: | They propose a question-answering dataset that tests understanding of feasibility . they propose to use commonsense reasoning to reason about when an action is feasible . |
| Outcome: | The proposed dataset shows that state-of-the-art models struggle to answer feasibility questions correctly. |
Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? A Petroglyph Revisited (2025.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive Transformer language models do not require explicit positional encodings (PEs) this is because a cascade of (permutation invariant) set processors collectively exhibit sequence-sensitive behavior in the autoregressively setting. |
| Approach: | They propose to explain why autoregressive Transformers require explicit positional encodings (PEs) this property has been known since early efforts adopting the Transformer for language modeling . |
| Outcome: | The proposed model can distinguish sequences with permuted tokens without the need for explicit PEs. |
Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Sign language gloss translation aims to translate the sign glosses into spoken language texts, which is challenging due to the scarcity of labeled gloss-text parallel data. |
| Approach: | They propose a back translation technique that generates pseudo-parallel data by translating in-domain spoken language texts into sign glosses. |
| Outcome: | The proposed method outperforms the BT methods on three benchmarks of sign language gloss translation in different languages. |
DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models (2024.findings-acl)
Copied to clipboard
Baohang Zhou, Zezhong Wang, Lingzhi Wang, Hongru Wang, Ying Zhang, Kehui Song, Xuhui Sui, Kam-Fai Wong
| Challenge: | Existing methods to detect pretraining data from large language models are unrealistic to them. |
| Approach: | They propose to detect pre-training data from LLM in a black-box way by using GPT-2 as reference model and feed it with sequence probabilities to detect whether it was used to train it. |
| Outcome: | The proposed framework outperforms existing methods on the benchmark datasets and shows that it is effective on different popular LLMs. |
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023.tacl-1)
Copied to clipboard
| Challenge: | Current language models generate high-quality text, but are they copying it or have they learned generalizable linguistic abstractions? |
| Approach: | They propose a suite of analyses for assessing the novelty of generated text . they focus on sequential structure (n-grams) and syntactic structure (syntactical structure). |
| Outcome: | The proposed model-generated text is as novel as the baseline human-generated model- generated text, but it is copied substantially, the authors show . |
Conversational Agent for Daily Living Assessment Coaching Demo (2021.eacl-demos)
Copied to clipboard
| Challenge: | Conversational Agent for Daily Living Assessment Coaching (CADLAC) is a multi-modal conversational agent system designed to impersonate “individuals” with various levels of ability in activities of daily living. |
| Approach: | They propose to use a multi-modal conversational agent system to impersonate individuals with various levels of ability in activities of daily living to train assessors how to conduct interviews . |
| Outcome: | The system is implemented on the MindMeld platform for conversational AI and features a bidirectional long short-term memory topic tracker that allows the agent to navigate conversations spanning 18 different ADL domains. |
Efficient Provably Secure Linguistic Steganography via Range Coding (2026.acl-long)
Copied to clipboard
| Challenge: | Linguistic steganography is a promising field in safeguarding information . previous methods have achieved perfect imperceptibility but at the expense of embedding capacity. |
| Approach: | They propose to use a classical entropy coding method to achieve secure steganography . they propose to employ a rotation mechanism to achieve embedding efficiency . |
| Outcome: | The proposed method outperforms existing methods in embedding capacity and embeddability. |
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive. |
| Approach: | They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks. |
| Outcome: | The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks. |
Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infrequent names are less similar to initial representations, and are more self-similar, suggesting that models rely on less context-informed representations of uncommon and minority names. |
| Approach: | They use a dataset of U.S. first names with labels based on predominant gender and racial group to examine effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias. |
| Outcome: | The results show that infrequent names are less similar to initial representations and have a Spearman’s rho between frequency and self-similarity as low as .763 . |
An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)
Copied to clipboard
| Challenge: | a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed . |
| Approach: | They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure . |
| Outcome: | The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks. |
The workweek is the best time to start a family – A Study of GPT-2 Based Claim Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Argument generation is a challenging task whose impact on social media is growing . we examine how argument generation can be enhanced to provide better arguments . |
| Approach: | They propose a pipeline for argument generation based on GPT-2 . they examine the types of claims it produces, and their veracity . |
| Outcome: | The proposed pipeline improves argument generation quality and provides a clear stance on a debate topic. |
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
Neural Data-to-Text Generation with LM-based Text Augmentation (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural data-to-text generation is a difficult task for many new applications because of a lack of training data. |
| Approach: | They propose a few-shot approach that augments the data available for training by generating new text samples based on replacing specific values by alternative ones from the same category and pairing the new text with data samples. |
| Outcome: | The proposed approach outperforms fully supervised sequence-to-sequence models with less than 10% of the training set on both datasets. |
How do decoding algorithms distribute information in dialogue responses? (2023.findings-eacl)
Copied to clipboard
| Challenge: | Using different decoding algorithms, we find that human dialogue generation is beneficial for adherence to the Uniform Information Density principle. |
| Approach: | They investigate whether decoding algorithms implicitly follow the Uniform Information Density principle by distributing information evenly in utterances. |
| Outcome: | The proposed method encourages non-uniform responses, but under low/high surprisal conditions, resulting in poor quality responses. |
When classifying grammatical role, BERT doesn’t care about word order... except when it matters (2022.acl-short)
Copied to clipboard
| Challenge: | Recent work has shown large language models are surprisingly word order invariant . however, word order knowledge is crucial in defining later-layer representations of words . |
| Approach: | They probe grammatical role representations in English BERT and GPT-2 to find word order crucial . they find word orders are crucial in defining later-layer representations of words in non-prototypical positions . |
| Outcome: | The proposed model is based on natural prototypical inputs where word order is crucial for correct classification. |
As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models are limited in their ability to train for English, which is a problem for many other languages. |
| Approach: | They propose to adapt existing generative language models to new languages by retraining lexical embeddings without tuning the Transformer layers. |
| Outcome: | The proposed method achieves lexical embeddings for Italian and Dutch that are aligned with the original English lexicals. |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
Generating Counter Narratives against Online Hate Speech: Data and Strategies (2020.acl-main)
Copied to clipboard
| Challenge: | Hate Speech (HS) is a pervasive issue that spreads quickly and widely . research has focused on avoiding undesired effects that come with content moderation . |
| Approach: | They propose to use large scale unsupervised language models to generate responses to hate effectively using large scale models. |
| Outcome: | The proposed methods lack quality data and produce generic/repetitive responses. |
Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks. |
| Approach: | They propose a framework that uses pre-trained language to model each speaker separately . it can be generalized to more challenging, non-collaborative tasks such as persuasion . |
| Outcome: | The proposed framework outperforms or is on par with state-of-the-art methods on two popular datasets: CamRest676 and MultiWOZ. |
Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modeling (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have highlighted a variety of biases in pre-trained language models . however, these studies focus on fine-grained analysis of educational corpora and text that is not English . |
| Approach: | They analyze bias across text and through multiple architectures on a corpus of 9,165 German peer-reviews collected from university students over five years. |
| Outcome: | The proposed dataset shows that pre-trained language models exhibit conceptual, racial, and gender biases. |
JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs (2023.findings-eacl)
Copied to clipboard
| Challenge: | In this paper, we compare syntactic knowledge of language models across different languages. |
| Approach: | They introduce a dataset for targeted syntactic evaluations of language models in Japanese. |
| Outcome: | The proposed dataset compares the syntactic knowledge of language models across languages. |
AutoMeTS: The Autocomplete for Medical Text Simplification (2020.coling-main)
Copied to clipboard
| Challenge: | Semi-automated text simplification approaches can be used to simplify text faster and at a higher quality. |
| Approach: | They propose to use autocomplete to simplify medical texts using aligned English Wikipedia sentences and pretrained neural language models to analyze the additional context. |
| Outcome: | The proposed model outperforms the best individual model by 2.1% and achieves a word prediction accuracy of 64.52%. |
Representation Learning for Conversational Data using Discourse Mutual Information Maximization (2022.naacl-main)
Copied to clipboard
Bishal Santra, Sumegh Roychowdhury, Aishik Mandal, Vasu Gurram, Atharva Naik, Manish Gupta, Pawan Goyal
| Challenge: | Existing language modeling pretraining objectives do not take structural information of conversational text into account. |
| Approach: | They propose a structure-aware Mutual Information based loss-function DMI for training dialog-representation models that captures the inherent uncertainty in response prediction. |
| Outcome: | The proposed model outperforms strong baseline models on nine diverse tasks. |
GPT-D: Inducing Dementia-related Linguistic Anomalies by Deliberate Degradation of Artificial Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large numbers of model parameters have shown impressive performance on the task of discriminating between language produced by cognitively healthy individuals and those with Alzheimer’s disease (AD). |
| Approach: | They propose to use a Transformer DL model pre-trained on general English text to combine an artificially degraded version of itself with a model that generalizes well to spontaneous conversations. |
| Outcome: | The proposed method generalizes well to spontaneous conversations and generates text with characteristics associated with AD, demonstrating the induction of dementia-related linguistic anomalies. |
Direct Prompt Optimization with Continuous Representations (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for prompt optimization for language models lack extensibility and search space. |
| Approach: | They propose a method that integrates greedy strategies into optimization with continuous representations to address instability caused by rounding. |
| Outcome: | The proposed approach can improve prompt optimization performance on text classification and attack tasks, as well as models, including GPT-2, OPT, Vicuna, and LLaMA-2. |
A Unifying View On Task-oriented Dialogue Annotation (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research attention in task-oriented dialogue systems focuses on end-to-end neural models. |
| Approach: | They present a dataset that combines annotated corpora from four domains to provide a unified ontology and annotation schema for task-oriented dialogues. |
| Outcome: | The proposed dataset improves language, information content and performance in dialogues with two recent models. |
LLMDet: A Third Party Large Language Models Generated Text Detection Tool (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing detection tools rely on access to LLMs and can only distinguish between machine-generated and human-authored text. |
| Approach: | They propose a model-specific, secure, efficient, and extendable detection tool that can source text from specific LLMs. |
| Outcome: | The proposed tool can source text from specific LLMs, such as GPT-2, OPT, LLaMA, and others. |
Advancing Precise Outline-Conditioned Text Generation with Task Duality and Explicit Outline Control (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on outline-conditioned text generation focus on generating text using provided outlines as rough sketches, but lack of clarity and rationality of the rough outlines hampers quality of the generated text. |
| Approach: | They propose a novel task that requires generating stories based on specific, sentence-level outlines. |
| Outcome: | The proposed framework improves the quality of precise outline-conditioned text generation. |
Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (2024.acl-long)
Copied to clipboard
| Challenge: | Existing syntactic language models require a gold tree and sequential training to generate sentences. |
| Approach: | They propose an unsupervised syntactic language model that incrementally generates a sentence with its syntaktic tree in a left-to-right manner. |
| Outcome: | The proposed model outperforms existing models on grammar induction and comprehension tasks while holding a substantial acceleration on training. |
Scientific and Creative Analogies in Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing analogy datasets focus on a limited set of analogical relations with a high similarity of the two domains between which the analogy holds. |
| Approach: | They propose a dataset that encodes analogy in pretrained language models . they use a system that maps attributes and relational structures across dissimilar domains . |
| Outcome: | The proposed dataset shows that state-of-the-art models achieve low performance on analogy tasks . |
Towards Understanding Gender-Seniority Compound Bias in Natural Language Generation (2022.lrec-1)
Copied to clipboard
Samhita Honnavalli, Aesha Parekh, Lily Ou, Sophie Groenwold, Sharon Levy, Vicente Ordonez, William Yang Wang
| Challenge: | Existing studies have not investigated how gender biases in natural language processing (NLP) are compounded with other societal biase. |
| Approach: | They propose a framework for probing compound bias by examining seniority in pre-trained neural generation models. |
| Outcome: | The proposed framework amplifies bias by considering women as junior and men as senior more often than ground truth in both domains. |
TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can produce informative and fluent text with the help of large-scale datasets, but they suffer insufficient learning problem with limited training data. |
| Approach: | They propose to use table transformation module with template to rewrite structured table in natural language as input for GPT-2 and exploit multi-task learning with two auxiliary tasks to preserve table’s structural information. |
| Outcome: | The proposed model outperforms existing systems on most few-shot settings. |
Beyond Memorization: The Challenge of Random Memory Access in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Language Models (LMs) have shown their effectiveness in knowledge-intensive tasks. |
| Approach: | They investigate whether a generative language model is able to access its memory sequentially or randomly. |
| Outcome: | The proposed LMs are able to access memory sequentially or randomly. |
Quadapter: Adapter for GPT-2 Quantization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) fine-tunes model parameters in response to the numerical error arising from quantization. |
| Approach: | They propose a quantization adapter that scales activations channel-wise to make activations quantization-friendly by scaling them channel- wise. |
| Outcome: | The proposed method prevents overfitting and improves quantization performance. |
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are not fully considered for societal biases . pre-training models can be useful for many NLP tasks, but they can be harmful when used at scale. |
| Approach: | They investigate gender and racial bias across pre-trained language models . they evaluate bias within pre-trainers using three metrics: WEAT, sequence likelihood, and pronoun ranking. |
| Outcome: | The proposed model fails to detect gender and racial biases in pre-trained models . the model is ineffective when word embedding, demonstrating the need for more robust bias testing in transformers. |
AutoDSPy: Automating Modular Prompt Design with Reinforcement Learning for Small and Large Language Models (2025.emnlp-industry)
Copied to clipboard
Nafew Azim, Abrar Ur Alam, Hasan Bin Omar, Abdullah Mohammad Muntasir Adnan Jami, Jawad Ibn Ahad, Muhammad Rafsan Kabir, Md. Ismail Hossain, Fuad Rahman, Mohammad Ruhul Amin, Shafin Rahman, Nabeel Mohammed
| Challenge: | Large Language Models excel at complex reasoning tasks, yet their performance hinges on the quality of their prompts and pipeline structures. |
| Approach: | They propose a framework that fully automates large language models' pipeline construction using reinforcement learning. |
| Outcome: | Experimental results show that autoDSPy outperforms DSPy benchmarks in accuracy gains and time. |
ZeroAE: Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text classification use only encoders or decoders that do not allow for the use of labels in unseen domains. |
| Approach: | They propose an autoencoder that encodes text into two disentangled spaces and decodes it to generate text with labels in the unseen domains. |
| Outcome: | The proposed model outperforms the existing methods in label-partially-unseen and label-fully-un-seeen scenarios and even outperfects the SOTA methods. |
Tree-Based Representation and Generation of Natural and Mathematical Language (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for generating and modeling mathematical language are limited . existing models for modeling and generating mathematical language simply treat mathematical expressions as text . |
| Approach: | They propose to combine mathematical expressions and text-based models to generate mathematically valid expressions. |
| Outcome: | The proposed model outperforms baselines on mathematical expression generation tasks. |
Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Large-scale "natural language supervision" using image captions has enabled the first "zero-shot" AI image classifiers, which allow users to create their own image classes using natural language, yet outperform supervised models on common language-and-image tasks. |
| Approach: | They compare the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal image classifier which adapts the GPT2 architecture to encode image captions. |
| Outcome: | The proposed classifier outperforms GPT-2 on word-level semantic intrinsic evaluation tasks and achieves a new corpus-based state of the art for the RG65 evaluation. |
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent generative language models have shown promise in abstractive summarization tasks. |
| Approach: | They propose to use Fr echet embedding distance and angular embeddable similarity to evaluate the performance of generative language models in abstractive summarization tasks. |
| Outcome: | The proposed metric shows close relation with human judgments and has overall better correlations with them. |
INSET: Sentence Infilling with INter-SEntential Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Missing sentence generation fosters a wide range of applications in natural language generation . Developing models for sentence infilling can potentially facilitate many text generation applications . |
| Approach: | They propose a framework to decouple the problem from natural language processing . they propose generating missing sentences that can syntactically and semantically bridge context . |
| Outcome: | The proposed model learns a sentence representation and generates 'missing sentences' the proposed model can be used for document auto-completion and meeting note expansion . |
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language models generate false statements that can lead to deception and distrust. |
| Approach: | They propose to measure whether a language model is truthful in generating answers to questions. |
| Outcome: | The proposed model was truthful on 58% of questions, while human performance was 94%. |
GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. |
| Approach: | They propose a prompt-based parameter-efficient fine-tuning approach that leverages insights into ICL’s information flow dynamics and hardwires the desired information flow into the GNN. |
| Outcome: | The proposed approach surpasses prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. |
Masked Language Model Scoring (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained masked language models require finetuning for most tasks. |
| Approach: | They evaluate pretrained masked language models out of the box via their pseudo-log-likelihood scores (PLLs) they attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 . |
| Outcome: | The proposed model outperforms autoregressive language models in a variety of tasks. |
Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible (2026.eacl-long)
Copied to clipboard
| Challenge: | linguists have discovered patterns which hold across virtually all known natural languages . lingulists are able to learn languages by comparing their learning curves to those of humans . |
| Approach: | They compare LLM learning curves on existing and "impossible" datasets . they find that GPT-2 learns each language and its impossible counterpart equally easily . |
| Outcome: | The proposed model learns each language and its impossible counterpart equally easily, the study shows . the study also shows that the proposed model does not provide any kind of separation between the possible and the impossible . |
Adapting a Language Model for Controlled Affective Text Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for affective text generation fail to capture emotional aspects of conversations without explicit affective information. |
| Approach: | They propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2 and incorporate emotion into the model to ensure grammatical correctness. |
| Outcome: | The proposed model outperforms existing models in all intensities and is robust to human evaluations. |
Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialogues (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Imlicit user feedback, user emotions and demographic information are promising sources for improving the accuracy and user engagement of dialogue responses, but the impact of such information on task completion and factual consistency is not known. |
| Approach: | They introduce the first English task-oriented and document-grounded dialogue dataset annotated with this information. |
| Outcome: | The proposed dataset shows that the model's responses are more informative and factual consistent. |
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent results show that the mix-of-experts architecture is parameter inefficient . large-scale pre-trained language models can achieve excellent performance in many NLP tasks. |
| Approach: | They propose to build a parameter-efficient mix-of-experts architecture by sharing information across experts. |
| Outcome: | The proposed architecture increases model capacity without increasing computation costs. |
Punctuations and Predicates in Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform tasks in ways that diverge significantly from human reasoning. |
| Approach: | They examine the computational importance of punctuation tokens in large language models . they use zeroing and layer-swapping techniques to examine their necessity and sufficiency . |
| Outcome: | The proposed model differs in GPT-2, DeepSeek, and Gemma in that punctuation is necessary and sufficient in multiple layers . the findings offer new insight into the internal mechanisms of punctuations in LLMs and have implications for interpretability and model analysis. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained Transformers are remarkably robust to pruning. |
| Approach: | They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance. |
| Outcome: | The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance. |
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained neural language models (LMs) are prone to generating toxic language . we investigate the extent to which pretrained LMs can be prompted to generate toxic language. |
| Approach: | They propose to use a dataset to test the extent to which pretrained LMs can generate toxic language . they use toxicity-based text generation algorithms to generate toxic text from natural prompts . |
| Outcome: | The proposed model can degenerate into toxic text even from seemingly innocuous prompts. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a test suite to evaluate commonsense reasoning capability of neural machine translation is presented . language models pretrained on large-scale corpora achieve a commonsensing accuracy of lower than 72% on target translations of this test suite. |
| Approach: | They propose a test suite to evaluate the commonsense reasoning capability of neural machine translation. |
| Outcome: | The proposed test suite performs poorly on commonsense reasoning of the three ambiguity types in terms of reasoning accuracy and reasoning consistency. |
Attention weights accurately predict language representations in the brain (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In Transformer-based language models, the attention mechanism converts token embeddings into contextual embeddables that incorporate information from neighboring words. |
| Approach: | They analyze fMRI recordings of English language learners and extract attention weights from them to determine how well they can predict brain responses. |
| Outcome: | The resulting hidden state embeddings are more accurate than lexical embeddngs or RNN-based models. |
Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory. |
| Approach: | They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules. |
| Outcome: | The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. |
Language model acceptability judgements are not always robust to context (2023.acl-long)
Copied to clipboard
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
| Challenge: | a recent study found that models prefer acceptable inputs over acceptable ones. |
| Approach: | They find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but unstable when contexts match the test stimuli in syntactic structure. |
| Outcome: | The proposed model performance improves when contexts match syntactic structure, and declines when they are unacceptable. |
Active Learning Principles for In-Context Learning with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has significantly enhanced predictive performance in few-shot learning settings. |
| Approach: | They propose to use pool-based Active Learning to identify the most informative demonstrations for few-shot learning over a single iteration to identify best demonstrations. |
| Outcome: | The proposed model outperforms all other methods, including random sampling, in the analysis of 24 classification and multi-choice tasks. |
A Plug-and-Play Method for Controlled Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for controlling language generation are not able to produce fluent text . current methods require additional models or fine-tuning to ensure specific words are included . |
| Approach: | They propose a plug-and-play decoding method that allows for controlled language generation . they add a shift in the probability distribution over our vocabulary towards semantically similar words . |
| Outcome: | The proposed method outperforms competing methods in human evaluations and does not impact fluency. |
Merely Judging Metaphor is Not Enough: Research on Reasonable Metaphor Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current metaphor detection tasks only provide labels without interpreting how to understand them. |
| Approach: | They propose to improve the current metaphor detection task by using mainstream Large Language Models. |
| Outcome: | The proposed model is based on the original sentence, target word, and usage . the model is then evaluated using manual evaluation . |
Progressive Generation of Long Text with Pretrained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for "long" text generation are limited to outputs of 50-200 tokens . however, our proposed ProGen generates coherent long passages of text in a progressive manner . |
| Approach: | They propose a method for generating coherent long passages of text in a progressive manner . they first produce domain-specific content keywords and then refine them into complete passages . human evaluation validates that their proposed generation is more coherent . |
| Outcome: | The proposed method produces domain-specific content keywords and refines them into complete passages in multiple stages. |
Sparse Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current text generators require sampling from a modified softmax to avoid degenerate text . entmax sampling creates a mismatch between training and testing conditions . |
| Approach: | They propose to use entmax transformation to train and sample from a sparse language model to avoid degenerate text. |
| Outcome: | The proposed model improves fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. |
Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns (2022.acl-long)
Copied to clipboard
| Challenge: | Existing work to predict gaze patterns during naturalistic reading has not been conducted on general text characteristics. |
| Approach: | They propose to use two eye-tracking corpora of naturalistic reading and two language models to test their performance. |
| Outcome: | The proposed models predict eye-tracking measures during naturalistic reading and language processing. |
Employing Argumentation Knowledge Graphs for Neural Argument Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating arguments use end-to-end knowledge graphs or are controlled with respect to the argument's topic, aspects, or stance. |
| Approach: | They construct and populate three knowledge graphs and encode them into debate portals and relevant paragraphs from Wikipedia. |
| Outcome: | The proposed model produces arguments with superior quality than those generated without knowledge. |
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021.emnlp-main)
Copied to clipboard
| Challenge: | Similarity measures are a vital tool for understanding how language models represent and process language. |
| Approach: | They propose to use cosine similarity and Euclidean distance to understand how words cluster in semantic space. |
| Outcome: | The proposed measures show that rogue dimensions dominate similarity measures and reveal representational quality. |
Investigating representations of verb bias in neural language models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Languages typically provide more than one grammatical construction to express certain types of messages. |
| Approach: | They propose a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. |
| Outcome: | The proposed model outperforms recurrent architectures even under comparable parameter and training settings. |
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for language understanding and understanding can be trained to provide contextualized representations of words based on text data. |
| Approach: | They propose a large-scale language VAE model Optimus that is pre-trained on large text corpus and fine-tuned for various language generation and understanding tasks. |
| Outcome: | The proposed model achieves new state-of-the-art on VAE language modeling benchmarks. |
Improving Text Auto-Completion with Next Phrase Prediction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Language models such as GPT-2 require considerable training effort to adapt to specific writing domains (e.g., medical). |
| Approach: | They propose an intermediate training strategy that encourages language models to complete partial queries with enriched phrases and eventually improve their text auto-completion performance. |
| Outcome: | The proposed approach outperforms baselines in auto-completion tasks for email and academic-writing domains with only around 1.2B tokens. |
Are Experts Needed? On Human Evaluation of Counselling Reflection Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Language models have been used to generate reflections automatically, but human evaluation is challenging due to the cost of hiring experts. |
| Approach: | They ask laypeople and experts to annotate synthetic reflections and human reflections from actual therapists. |
| Outcome: | The proposed method shows that laypeople and experts are reliable annotators and have moderate-to-strong inter-group correlation. |
Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models Using Minimal Pairs (2024.lrec-main)
Copied to clipboard
| Challenge: | a new method is being developed to probe internal linguistic characteristics in neural language models layer by layer . |
| Approach: | They propose a method that uses minimal pairs benchmark to probe internal linguistic characteristics in neural language models layer by layer. |
| Outcome: | The proposed method captures grammaticality labels in language models layer by layer . it is based on the cognitive neurosciences of the brain and its representations as "neural activations". |
RAST: Domain-Robust Dialogue Rewriting as Sequence Tagging (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for dialogue rewriting suffer from the robustness issue, i.e., performances drop dramatically when testing on a different dataset. |
| Approach: | They propose a sequence-tagging-based approach that reduces the search space while preserving the core of the task. |
| Outcome: | The proposed model significantly reduces the search space while still covering the core of the task. |
Embeddings models for Buddhist Sanskrit (2022.lrec-1)
Copied to clipboard
| Challenge: | Despite extensive scholarly endeavors, much uncertainty still surrounds this body of literature, especially regarding matters of chronology, authorship, compositional history. |
| Approach: | They propose a corpus of Buddhist texts, a general corpus and word similarity and word analogy datasets for embeddings models. |
| Outcome: | The proposed models perform better on semantic similarity and word analogy tasks than on contextual models. |
StereoSet: Measuring stereotypical bias in pretrained language models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing literature on stereotypical biases in language models is limited . current evaluations focus on measuring bias without considering language modeling ability . |
| Approach: | They propose to measure stereotypical biases in four domains: gender, profession, race, and religion . they compare stereotypical and language modeling ability of popular models like BERT, GPT-2, RoBERTa and XLnet . |
| Outcome: | The proposed model shows strong stereotypical biases in gender, profession, race, and religion domains. |
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation (2025.coling-main)
Copied to clipboard
| Challenge: | Structured pruning and knowledge distillation are often not efficient and require a fixed architecture, limiting flexibility. |
| Approach: | They propose a method which integrates knowledge distillation and structured pruning by replacing transformer blocks with smaller, efficient versions during training. |
| Outcome: | The proposed method outperforms L1 pruning and maintains four-fifths of performance on language modeling and commonsense reasoning tasks. |
INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models (2023.findings-emnlp)
Copied to clipboard
H S V N S Kowndinya Renduchintala, Krishnateja Killamsetty, Sumit Bhatia, Milan Aggarwal, Ganesh Ramakrishnan, Rishabh Iyer, Balaji Krishnamurthy
| Challenge: | Pre-trained language models have a remarkable improvement in generalization capability . however, this leads to prohibitively long training times and a detrimental environmental impact . |
| Approach: | They propose to use submodular optimization to select highly informative subsets of training data to train multiple PTLMs using only fractions of data. |
| Outcome: | The proposed framework achieves 99% of the performance of fully-trained models using only fraction of training data. |
Transformers in the loop: Polarity in neural models of language (2022.acl-long)
Copied to clipboard
| Challenge: | Recent Transformer-based language representation models (LRMs) show impressive results on practical text analysis tasks, but do they have access to complex linguistic notions? |
| Approach: | They propose to use polarity as a case study to compare metrics derived from language models to human judgments obtained in psycholinguistic experiments. |
| Outcome: | The proposed model is more accurate than linguistic theory predictions for polarity, and allows us to use language models to discover new insights into natural language grammar beyond existing linguistic theories. |
Differentially Private Learning Needs Better Model Initialization and Self-Distillation (2025.naacl-long)
Copied to clipboard
| Challenge: | Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. |
| Approach: | They propose a method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, applies DP finetuning on private data, and performs self-distillation to refine outputs. |
| Outcome: | The proposed method outperforms vanilla DPSGD with significant improvements in lexical diversity and grammar errors. |
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Conventional fine-tuning works through updating all of the parameters in the pre-trained model, but as the size of pre-train models grows, it can be time-consuming and computationally expensive. |
| Approach: | They propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. |
| Outcome: | The proposed framework saves 25% inference FLOPs while maintaining competitive downstream performance. |
TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Using pretrained language models, we propose an error-annotated dataset for text generation . we use carefully selected prompt words to guide GPT-2 to generate candidate sentences . |
| Approach: | They propose an error-annotated dataset with multiple benchmark tasks for text generation from pretrained language models. |
| Outcome: | The proposed dataset covers 24 types of errors according to common sense and linguistics. |
Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure. |
| Approach: | They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions. |
| Outcome: | The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses . |
Investigating African-American Vernacular English in Transformer-Based Text Generation (2020.emnlp-main)
Copied to clipboard
Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, William Yang Wang
| Challenge: | Recent work in Natural Language Generation (NLG) uses a Transformer-based language model to generate high-quality, coherent text when prompted by arbitrary input. |
| Approach: | They evaluate the performance of a Transformer-based model that generates high-quality, coherent text when prompted by arbitrary input. |
| Outcome: | The proposed model improves on AAVE and SAE text with pretrained sentiment classifiers. |
On Robustness of Finetuned Transformer-based NLP Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Transformer-based language models have been finetuned for a large number of tasks. |
| Approach: | They characterize changes between pretrained and finetuned models with CKA and STIR metrics. |
| Outcome: | The proposed models are more robust to perturbations than BERT and T5 on classification tasks and generation tasks. |
LPZero: Language Model Zero-cost Proxy Search from Zero (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing zero-cost (ZC) proxies rely on expert knowledge and incur significant trial-and-error costs. |
| Approach: | They propose a framework that automatically designs zero-cost (ZC) proxies for various tasks and incorporates genetic programming to find the optimal symbolic composition. |
| Outcome: | The proposed framework achieves higher ranking consistency than human-designed proxies on NLP tasks. |
Sequentially Controlled Text Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using GPT-2, long documents can ramble and do not follow human-like writing structure. |
| Approach: | They propose a controlled text generation task that generates documents with structure . they use a news article as a dataset to test different degrees of structural awareness . |
| Outcome: | The proposed task generates documents with a structure that is human-like, but long documents lack structure. |
Substance over Style: Document-Level Targeted Content Transfer (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models excel at writing from scratch, but many real-world scenarios require rewriting an entire document to fit a set of constraints. |
| Approach: | They propose a document-level targeted content transfer task that addresses the challenge of rewriting an entire document coherently by generating coherent and diverse rewrites that obey a constraint while remaining close to the original document. |
| Outcome: | The proposed model outperforms existing methods by generating coherent and diverse rewrites that obey the constraint while remaining close to the original document. |
Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework (2025.emnlp-main)
Copied to clipboard
Yuhang Chen, Zhen Tan, Ajay Kumar Jaiswal, Huaizhi Qu, Xinyu Zhao, Qi Lin, Yu Cheng, Andrew Kwong, Zhichao Cao, Tianlong Chen
| Challenge: | Bit-flip errors (BFEs) are hardware faults where individual bits in memory or processing units are unintentionally flipped. |
| Approach: | They propose a novel defense strategy to mitigate bit-flip errors (BFEs) they propose bfe protection and a self-correction mechanism to minimize performance degradation . |
| Outcome: | The proposed defense strategy minimizes performance degradation while significantly improving robustness against BFEs. |
Inquisitive Question Generation for High Level Text Comprehension (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing data-driven questions generate questions that fill gaps in knowledge . a dataset of 19K questions is used to generate meaningful questions . |
| Approach: | They propose a dataset of 19K questions that are elicited while a person is reading a document. |
| Outcome: | The proposed model generates reasonable questions, but the task is challenging. |
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation (2023.acl-long)
Copied to clipboard
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi
| Challenge: | Empirical results suggest that scale is not the only way to build commonsense capabilities. |
| Approach: | They propose a commonsense distillation framework that can achieve a competitive level of commonsensing without relying on the benefits of scale. |
| Outcome: | The proposed framework breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model’s own enhanced commonsense acquisition capabilities. |
Dependency-based Mixture Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models to incorporate syntactic structures into neural language models have relied heavily on elaborate components for a specific language model, which makes them unwieldy in practice to fit into other models. |
| Approach: | They propose a dependency-based mixture language model that incorporates syntactic structures into neural language models by mixing previous dependency modeling probabilities with self-attention. |
| Outcome: | The proposed method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks. |
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, AdamS demonstrates strong performance in various tasks . et al., 2023b): AdamS is efficient, efficient, and model-agnostic. |
| Approach: | They propose a model-agnostic alternative to Adam for large language model pretraining and post-training. |
| Outcome: | The proposed method matches memory footprint of SGD with momentum while delivering superior performance. |
Do BabyLMs Wanna Learn Wanna Contraction? On the Learnability without Language-Specific Bias (2026.findings-acl)
Copied to clipboard
| Challenge: | grammatical constraints on wanna contraction can be learned by artificial learners under cognitively plausible learning conditions. |
| Approach: | They evaluate 24 BabyLMs from 2024 BabyLM Challenge and four standard models . they find that only encoder-based BabyLM models can capture wanna contraction . |
| Outcome: | The proposed models show modest but meaningful sensitivity on large datasets and high-frequency wanna instances. |
Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language models can't predict the probability distribution over next words or masked words because of a single hidden state. |
| Approach: | They propose a multi-facet softmax to address the limitations of softmax . they propose LMs that can predict the next word distribution better as their sizes grow . |
| Outcome: | The proposed model achieves two-fold improvements in the perplexity of GPT-2 and BERT. |
Generating Major Types of Chinese Classical Poetry in a Uniformed Framework (2020.lrec-1)
Copied to clipboard
| Challenge: | Chinese classical poetry is one of the most valuable literary and cultural heritages of China . it has many particular characteristics in its language structure, ranging from form, sound to meaning . a proposed uniformed framework for generating major types of Chinese classical poems is proposed . |
| Approach: | They propose a uniformed framework for generating major types of Chinese classical poems . they use a form- stressed weighting method to strengthen the control to the form of the generated poems a proposed framework is incorporated into Jiuge, the most influential Chinese classical poetry generation system developed by Tsinghua University. |
| Outcome: | The proposed framework can generate Chinese classical poems of major types with high quality in form and content. |
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theories of human sentence processing can be divided into two broad categories: expectation-based theories and memory-based ones. |
| Approach: | They propose to integrate expectations and retrieval from working memory into a unified cognitive model that can capture syntactic and semantic interference effects observed in human experiments. |
| Outcome: | The proposed model captures syntactic and semantic interference effects observed in human experiments. |
Text-Free Prosody-Aware Generative Spoken Language Modeling (2022.acl-long)
Copied to clipboard
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu
| Challenge: | Experimental results show that generative spoken language models (LMs) are natural unsupervised multitask learners. |
| Approach: | They propose a prosody-aware generative spoken language model that uses discovered units to generate natural, meaningful, and coherent speech. |
| Outcome: | The proposed model can generate natural, meaningful, and coherent speech given a spoken prompt. |
Textual Data Augmentation for Efficient Active Learning on Tiny Datasets (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing active learning approaches for textual data are limited due to the complexity of language. |
| Approach: | They propose an approach where guided outputs of a language generation model can be enhanced through an active learning process. |
| Outcome: | The proposed approach achieves performance increases of 3% and 5% on TREC-6 and SST-2 datasets compared with NGDG, which does not optimize for a reward function. |
Stable Language Model Pre-training by Reducing Embedding Variability (2024.emnlp-main)
Copied to clipboard
| Challenge: | Stable pre-training is essential for achieving better-performing language models, but tracking pre-train stability is impractical due to high computational costs. |
| Approach: | They propose to use Token Embedding Variability as a proxy to estimate pre-training stability. |
| Outcome: | The proposed method improves stability and lowers perplexities even at deeper layer counts. |
Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for dialogue state tracking are based on Graph Attention Networks . if the relationship between slots and values is modelled explicitly, this can be improved . |
| Approach: | They propose a model architecture that augments GPT-2 with Graph Attention Networks to allow sequential prediction of slot values. |
| Outcome: | The proposed architecture improves performance against a strong GPT-2 baseline and with sparsely supervised training. |
Active Example Selection for In-Context Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning performance is unstable across samples of examples, suggesting the idiosyncrasies of how language models acquire information. |
| Approach: | They propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples and propose 'in-context learning' performance can be highly unstable across samples of examples, suggesting the idiosyncrasies of how language models acquire information. |
| Outcome: | The proposed model can perform tasks with examples with a 5.8% improvement on GPT-2 and GPT-3, but the improvement diminishes on larger models, suggesting emerging capabilities of large language models. |
SAMRank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT-2 (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for keyphrase extraction use contextualized embeddings to capture semantic relevance between words, sentences, and documents. |
| Approach: | They propose an unsupervised keyphrase extraction approach that uses only a self-attention map in a pre-trained language model to determine the importance of phrases. |
| Outcome: | The proposed approach outperforms embedding-based models on three keyphrase extraction datasets. |
Powerful Training-Free Membership Inference Against Fine-Tuned Autoregressive Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for auditing fine-tuned language models have limited detection rates . membership inference attacks aim to determine if a specific record was in a model's training set . |
| Approach: | They propose a membership inference attack that exploits memorization at error positions . EZ-MIA achieves 3.8 higher detection than previous state-of-the-art . |
| Outcome: | The proposed attack achieves 3.8 higher detection than previous state-of-the-art models . EZ-MIA achieves 8 higher detectability than prior work, requiring no model training . |
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text generation evaluation metrics are lacking in robustness analysis. |
| Approach: | They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization . |
| Outcome: | The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization. |
Perturbed examples reveal invariances shared by language models (2024.findings-acl)
Copied to clipboard
| Challenge: | Rapid growth in natural language processing (NLP) research has led to numerous new models outpacing our understanding of how they compare to established ones. |
| Approach: | They propose a framework to compare two NLP models by revealing their shared invariance to interpretable input perturbations targeting a specific linguistic capability. |
| Outcome: | The proposed framework can shed light on the types of invariances retained or emerging in new models. |
GOLEM: GOld Standard for Learning and Evaluation of Motifs (2024.lrec-main)
Copied to clipboard
| Challenge: | Motifs are distinctive, recurring, widely used idiom-like words or phrases, often originating from folklore, whose meaning are anchored in a narrative. |
| Approach: | They present a dataset annotated for motific information in English . it contains 26,078 motif candidates across 34 motif types from three cultural or national groups: Jewish, Irish, and Puerto Rican. |
| Outcome: | The first dataset annotated for motific information identifies 26,078 motif candidates across 34 motif types from three cultural or national groups: Jewish, Irish, and Puerto Rican. |
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights (2022.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Transformers incur O(T) complexity during per-token generation due to the self-attention mechanism. |
| Approach: | They propose a kernel-based method to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. |
| Outcome: | The proposed method outperforms prior methods and retains 99% of attention’s performance on WikiText-103 against more complex attention substitutes. |
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models cannot be directly employed to generate text under specified lexical constraints. |
| Approach: | They propose a method for insertion-based text generation that inserts tokens between existing tokens in a parallel manner. |
| Outcome: | The proposed method is intuitive and interpretable on Wikipedia and Yelp datasets. |
SORTIE: Dependency-Aware Symbolic Reasoning for Logical Data-to-text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on logical data-to-text generation rely on neural language models to generate the final table description, but they have difficulty working out key entities in the description. |
| Approach: | They propose a symbolic reasoning framework that reasons out each entity in the table description with a table-compatible programming language. |
| Outcome: | The proposed framework outperforms existing methods on three datasets and three backbones with an absolute improvement of 5.7%11.5% on SP-Acc. |
Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work suggests hierarchical organization where different layers specialize in capturing distinct levels of linguistic structure. |
| Approach: | They probe 25 models from BERT Base to Qwen2.5-7B focusing on linguistic properties: lexical identity and inflectional features. |
| Outcome: | The proposed model maintains inflectional features across layers while trading off lexical identity for compact, predictive representations. |
How Good Are LLMs at Out-of-Distribution Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is crucial for ensuring AI safety . large language models (LLMs) are becoming more prevalent due to their scale, pre-training objectives, and paradigms used for inference. |
| Approach: | They propose to use large language models to investigate out-of-distribution (OOD) detection in machine learning. |
| Outcome: | The proposed method outperforms other OOD detectors in zero-grad and fine-tuning scenarios. |
Advancing Parameter Efficiency in Fine-tuning via Representation Editing (2024.acl-long)
Copied to clipboard
Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang
| Challenge: | Parameter Efficient Fine-Tuning (PEFT) has gained significant attention for its ability to achieve competitive results while updating only a small subset of trainable parameters. |
| Approach: | They propose a new approach to fine-tuning neural models that scales and biases the representation produced at each layer. |
| Outcome: | The proposed approach reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning and by . 32 compared with LoRA. |
Generating Labeled Data for Relation Extraction: A Meta Learning Approach with Joint GPT-2 Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Relation Extraction (RE) is the task of identifying semantic relation between entities mentioned in text. |
| Approach: | They propose a framework to automatically generate labeled data for Relation Extraction . they propose 'reward function' to update pre-trained language model for RE . |
| Outcome: | The proposed framework generates labeled data for relation extraction using a pre-trained language model and a meta learning approach to improve the generated samples. |
IDEATE: Detecting AI-Generated Text Using Internal and External Factual Structures (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to detect AI-generated text rely on internal evidences, but external evidences are not considered. |
| Approach: | They propose a hierarchical graph network that utilizes internal and external factual structures to detect AI-generated text. |
| Outcome: | The proposed network outperforms current state-of-the-art methods on four datasets. |
Language acquisition: do children and language models follow similar learning stages? (2023.findings-acl)
Copied to clipboard
| Challenge: | During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master complex syntactic structures. |
| Approach: | They train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. |
| Outcome: | The proposed model exhibits similar learning trajectories to human children aged between 18 months and 6 years. |
Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions for word probability distributions are limited and the output softmax layer is inherently limited. |
| Approach: | They propose to use the output softmax layer to compute the word probability distribution instead of using pointer networks to break the bottleneck. |
| Outcome: | The proposed method improves factCC score by 2 points in CNN/DM and XSUM dataset, and MAUVE scores by 30% in bookSum paragraph-level dataset. |
Jump to Conclusions: Short-Cutting Transformers with Linear Transformations (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models create hidden representations of inputs at every layer, but only use final-layer representations for prediction. |
| Approach: | They propose a method for casting hidden representations as final representations, bypassing transformer computation in-between. |
| Outcome: | The proposed method produces more accurate predictions from hidden layers across various model scales, architectures, and data distributions. |
Identifying Noise in Human-Created Datasets using Training Dynamics from Generative Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing noise detection techniques for autoencoder models do not generalize to ArLMs due to differences in learning dynamics. |
| Approach: | They propose a method that leverages training dynamics to rank datapoints from easy-to-learn to hard-tolear . TDRanker achieves at least 2x faster denoising than previous techniques . |
| Outcome: | The proposed method demonstrates robustness across multiple model architectures and noise levels. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
Exploring Layer-wise Representations of English and Chinese Homonymy in Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | lexical ambiguity can arise due to the misunderstanding of its multiple senses. |
| Approach: | They propose to use part of speech to examine homonyms in Chinese and English . they find no universal layer depth excels in differentiating homnomial representations . |
| Outcome: | The proposed model improves contextualization of homonym representations in Chinese . the results challenge the simplistic understanding of their inner workings, the authors say . |
Membership and Memorization in LLM Knowledge Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Knowledge Distillation (KD) aim to mitigate the high computational demands of Large Language Models (LLMs). |
| Approach: | They characterize and investigate membership privacy risks inherent in six LLM KD techniques . they use instruction-tuning settings that span seven NLP tasks and three teacher model families and various size student models to examine the extent of privacy risks. |
| Outcome: | The proposed methods carry membership and memorization privacy risks from the teacher to students, but differ across different techniques. |
Enhancing Goal-oriented Proactive Dialogue Systems via Consistency Reflection and Correction (2025.acl-long)
Copied to clipboard
| Challenge: | Unlike traditional dialogue systems, goal-oriented proactive dialogue systems focus on achieving specific objectives by actively guiding and anticipating user needs. |
| Approach: | They propose a model-agnostic two-stage Consistency Reflection and Correction framework that allows the model to reflect on discrepancies between generated responses and dialogue contexts and suggest possible corrections. |
| Outcome: | The proposed framework significantly improves the consistency between generated responses and dialogue contexts on three datasets. |
On Zero-Shot Counterspeech Generation by LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used in numerous NLP tasks, including counterspeech generation. |
| Approach: | They propose three different prompting strategies for generating different types of counterspeech and propose a set of prompting techniques for counterspeak generation. |
| Outcome: | The proposed prompting strategies improve the performance of the models for counterspeech generation in two datasets, but with high toxicity with increase in model size. |
SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with factual inaccuracies, a critical issue in clinical NLP applications where errors could lead to serious consequences. |
| Approach: | They propose a pipeline that leverages >100B parameter GPT variants to act as synthetic experts to generate edit feedback without additional human annotations. |
| Outcome: | The proposed pipeline aims to improve the quality of clinical note summarizations by generating edit feedback without human annotations. |
Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Structured pruning is an effective technique for compressing pre-trained language models (PLMs), but it requires retraining, leading to additional computational overhead. |
| Approach: | They propose a task-specific pruning framework that prunes redundant modules of pre-trained language models before fine-tuning them. |
| Outcome: | The proposed pruning framework achieves higher performance on GLUE, SQUAD, WikiText-2, Wik-103, and PTB datasets while reducing the time required for fine-tuning. |
Quite Good, but Not Enough: Nationality Bias in Large Language Models - a Case Study of ChatGPT (2024.lrec-main)
Copied to clipboard
| Challenge: | Nationality is a key demographic element that enhances the performance of large language models, but it has received less scrutiny regarding inherent biases. |
| Approach: | They investigated nationality bias in ChatGPT, a large language model for text generation. |
| Outcome: | The proposed model generates 4,680 discourses about nationality in Chinese and English, with 195 countries, 4 temperature settings, and 3 prompt types. |
CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) achieve excellent performance through pretraining on extensive data. |
| Approach: | They propose an efficient selective layer intervention based on parameter-efficient fine-tuning methods to select the optimal steering layer to modulate LLM semantics. |
| Outcome: | The proposed approach is based on a model-agnostic framework and is safe to deploy. |
Machine-generated text detection prevents language model collapse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly prevalent across the web, resulting in a degenerative process whereby LLMs reinforce their own errors and reduce output diversity. |
| Approach: | They propose to use machine-generated text to reduce model collapse by up-sampling likely human content in training data. |
| Outcome: | The proposed approach prevents model collapse and improves performance compared to training on purely human data. |
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |
Constructing Interpretable Features from Compositional Neuron Groups (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing LLMs rely on dictionary learning with sparse autoencoders (SAEs) however, SAEs struggle in causal evaluations and lack intrinsic interpretability, as their learning is not explicitly tied to the computations of the model. |
| Approach: | They propose to decompose MLP activations with semi-nonnegative matrix factorization (SNMF) such that the learned features are mapped to their activating inputs, making them directly interpretable. |
| Outcome: | Experiments on Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs and a strong supervised baseline on causal steering while aligning with human-interpretable concepts. |