Papers with LLaMA
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
Koala: An Index for Quantifying Overlaps with Pre-training Corpora (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can be influenced by the frequency of overlap between pre-training corpora. |
| Approach: | They propose to search over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
| Outcome: | Koala is a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for fine-tuning and inference. |
| Approach: | They propose a Bayesian Knowledge Distillation framework for compact Large Language Models in resource-constrained fine-tuning scenarios that employs Logits Dual-Scaling, Knowledge Alignment Module, and Bayes Distillations Optimization. |
| Outcome: | The proposed framework outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. |
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantization-aware Training (QAT) is a popular technique for reducing memory usage and improving computational efficiency in large language models. |
| Approach: | They propose a weight-decomposed low-rank quantization-aware training approach that integrates QAT with a group-specific quantization magnitude adjustment. |
| Outcome: | The proposed method outperforms the state-of-the-art method on LLaMA and LLama2 models. |
SKICSE: Sentence Knowable Information Prompted by LLMs Improves Contrastive Sentence Embeddings (2024.naacl-short)
Copied to clipboard
| Challenge: | Experimental results show that our model outperforms the previous state-of-the-art model GloVe on STS tasks. |
| Approach: | They propose a simple and effective prompt template that is able to obtain the knowable information of input sentences from LLMs. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model PromptBERT on STS tasks. |
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs (2024.naacl-demo)
Copied to clipboard
| Challenge: | Recent advances in machine learning (ML) are attributed to large language models (LLMs), but their escalating memory requirements require developers to partition a large model to distribute it across multiple GPUs or TPUs. |
| Approach: | They propose a lightweight and user-friendly tool to automate distributed training and inference for LLMs and to simplify ML pipeline development. |
| Outcome: | The proposed tool automates distributed training and inference for LLMs, and simplifies ML pipeline development. |
Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for linguistic knowledge of Indigenous languages of the Americas focus on high- and medium-resource languages with substantial digital presence. |
| Approach: | They propose a framework for probing large language models’ linguistic knowledge of Indigenous languages of the Americas using zero-shot prompting and few-shot probing. |
| Outcome: | The proposed framework evaluates models from five major families on 13 Indigenous languages including Bribri, Guarani, and Nahuatl. |
A Trip Towards Fairness: Bias and De-Biasing in Large Language Models (2024.starsem-1)
Copied to clipboard
| Challenge: | a little or a large bias in CtB-LLMs may cause huge harm . LLaMA and OPT families have an important bias in gender, race, religion, and profession. |
| Approach: | They propose to debiase three families of Very Large-Language Models with LORA to reduce bias by 4.12 points in the normalized stereotype score. |
| Outcome: | The proposed model reduces bias up to 4.12 points in the normalized stereotype score. |
Zero-Shot Strategies for Length-Controllable Summarization (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models struggle with precise length control, particularly in zero-shot settings. |
| Approach: | They propose to use length approximation, target adjustment, sample filtering and automated revisions to improve LLMs' length control capabilities. |
| Outcome: | The proposed methods improve length control in large language models while maintaining or enhancing summary quality without the need for model fine-tuning or architectural changes. |
Using Subtext to Enhance Generative IDRR (2025.acl-short)
Copied to clipboard
| Challenge: | Arguments contain subtexts, but they are connotative and need prompts to be recognized . a lightweight subtext generator is helpful when the prompt doesn't raise a complex CoT. |
| Approach: | They leverage LLaMA to generate subtexts for argument pairs and verify their effectiveness . they construct a baseline IDRR using the decoder-only backbone LLama . |
| Outcome: | The proposed approach achieves higher F1 scores on two benchmarks than previous models. |
Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? (2025.coling-main)
Copied to clipboard
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, Yongfeng Zhang
| Challenge: | Large language models have shown remarkable performances across a wide range of tasks, but mechanisms by which they encode tasks of varying complexity remain poorly understood. |
| Approach: | They propose to explore the possibility that LLMs process concepts in different layers . they propose to categorize concepts based on their level of abstraction . |
| Outcome: | The proposed model can process complex concepts in shallow layers, the authors show . the proposed model could be used to prob complex tasks in shallow ones . |
IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages (2026.eacl-industry)
Copied to clipboard
| Challenge: | Indic Jailbreak Robustness (IJR) is a judge-free benchmark for adversarial safety across 12 languages. |
| Approach: | They propose a judge-free benchmark for adversarial safety across 12 languages . they find contracts inflate refusals but do not stop jailbreaks . |
| Outcome: | The proposed benchmarks cover 45,216 prompts in JSON and Free tracks. |
Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. |
| Approach: | They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning. |
| Outcome: | The proposed framework achieves comparable results to existing models on three language understanding benchmarks. |
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations. |
| Approach: | They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context. |
| Outcome: | The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks. |
Reconfidencing LLMs from the Grouping Loss Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to calibrate confidence scores for large language models often overlook biases towards certain groups, such as specific nationalities. |
| Approach: | They propose a method to calibrate confidence scores of Large Language Models by considering different groups, a process they call reconfidencing. |
| Outcome: | The proposed method mitigates biases against minority groups, the authors show . they show that the proposed method is more reliable than existing methods . |
Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation (2026.acl-long)
Copied to clipboard
Xi Xiao, Chenrui Ma, Yunbei Zhang, Chen Liu, Zhuxuanzi Wang, Yanshu Li, Lin Zhao, Guosheng Hu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a key parameter-efficient fine-tuning method . however, its effectiveness is hampered by semantic drift and structural incoherence . |
| Approach: | They propose a low-rank Adaptation framework that tackles semantic drift and structural incoherence by pruning task-irrelevant directions. |
| Outcome: | Experiments on large language models, vision models, and vision models show that the proposed framework outperforms LoRA and advanced dynamic rank allocation and sparsity-based methods. |
L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models. |
| Approach: | They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy. |
| Outcome: | The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy. |
Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023.emnlp-main)
Copied to clipboard
| Challenge: | asymmetric outliers in transformer language models are a challenge for post-training quantization . we propose a framework for outlier suppression that can be seamlessly migrated into subsequent modules . |
| Approach: | They propose a framework for post-training quantization that includes the channel-wise shifting and scaling for concentration. |
| Outcome: | The proposed framework can be migrated into subsequent modules while maintaining equivalence. |
PibE-MPP: A Play-it-by-Ear Masking Performance Plug-in for LLMs (2026.findings-acl)
Copied to clipboard
Mengwei Wang, Simin Niu, Xun Liang, Yuefeng Ma, Sensen Zhang, Jiawei Yang, Shichao Song, Hanyu Wang, Huayi Lai
| Challenge: | Random masking is a widely adopted classic baseline in large language models (LLMs). |
| Approach: | They propose a play-it-by-ear masking performance plug-in which enables LLMs to adaptively select masking target combinations for each task. |
| Outcome: | The proposed performance plug-in retains the advantages and mitigates the drawbacks of random masking in large language models. |
Are Large Language Models Economically Viable for Industry Deployment? (2026.acl-industry)
Copied to clipboard
Abdullah Mohammad, Sushant Kumar Ray, Pushkar Arora, Rafiq Ali, Ebad Shabbir, Gautam Siddharth Kashyap, Jiechao Gao, Usman Naseem
| Challenge: | Generative AI is increasingly deployed in healthcare, financial analytics, and conversational automation. |
| Approach: | They propose a framework that evaluates large language models across their full lifecycle on legacy GPUs. |
| Outcome: | The proposed framework evaluates LLMs across their full lifecycle on legacy GPUs. |
HalluDetect: Detecting, Mitigating, and Benchmarking Hallucinations in Conversational Systems in the Legal Domain (2025.emnlp-industry)
Copied to clipboard
Spandan Anaokar, Shrey Ganatra, Swapnil Bhattacharyya, Harshvivek Kashid, Shruthi N Nair, Reshma Sekhar, Siddharth Manohar, Rahul Hemrajani, Pushpak Bhattacharyya
| Challenge: | Large Language Models (LLMs) are widely used in industry but still produce hallucinations, limiting their reliability in critical applications. |
| Approach: | They propose to reduce hallucinations in consumer grievance chatbots by reducing their token accuracy by 0.4159 per turn. |
| Outcome: | The proposed system achieves an F1 score of 68.92% outperforming baseline detectors by 22.47% while maintaining the highest token accuracy. |
RuCCoD: Towards Automated ICD Coding in Russian (2025.emnlp-main)
Copied to clipboard
Alexandr Nesterov, Andrey Sakhovskiy, Ivan Sviridov, Airat Valiev, Vladimir Makharev, Petr Anokhin, Galina Zubkova, Elena Tutubalina
| Challenge: | a new dataset for clinical coding in Russian is available for download . human coders must navigate a wide array of medical terminology and time pressures . |
| Approach: | They present a new dataset for ICD coding in Russian, a language with limited biomedical resources. |
| Outcome: | The proposed model improves accuracy on an in-house EHR dataset from 2017 to 2021. |
GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design (2025.findings-naacl)
Copied to clipboard
| Challenge: | Text-Attributed Graphs (TAGs) are a powerful tool for understanding complex systems and relationships. |
| Approach: | They propose a benchmark to evaluate large language models for graph-structured data using prompts. |
| Outcome: | The proposed benchmark outperforms state-of-the-art graph LLMs and graph neural networks on graph learning tasks without training. |
Where CoT Reasoning Commits: Entropy Traces Identify Interpretable Attention Heads (2026.findings-acl)
Copied to clipboard
| Challenge: | a growing body of work suggests a disconnect between the generated rationale and the model's actual choice. |
| Approach: | They propose a mechanism-aware framework that interprets the evolving "choice state" of attention heads during CoT generation . they identify a set of intervention targets and perform Selective Head Fine-Tuning . |
| Outcome: | The proposed framework interprets the "choice state" of attention heads during CoT generation . it detects two functional behaviors: Steadfast Heads and Wavering Heads . |
Toward Inclusive Language Models: Sparsity-Driven Calibration for Systematic and Interpretable Mitigation of Social Biases in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new method to mitigate stereotypical bias in large language models is needed . inherent biases from training on vast Internet datasets can amplify harmful stereotypes . |
| Approach: | They propose a method to identify stereotypical bias in decoder-only transformer models . they apply a localization mechanism that correlates internal activations with a new Context Influence score . |
| Outcome: | The proposed method reduces stereotypical biases on BBQ, StereoSet, and CrowS-Pairs while improving reasoning performance on MMLU by 10%. |
LLMDet: A Third Party Large Language Models Generated Text Detection Tool (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing detection tools rely on access to LLMs and can only distinguish between machine-generated and human-authored text. |
| Approach: | They propose a model-specific, secure, efficient, and extendable detection tool that can source text from specific LLMs. |
| Outcome: | The proposed tool can source text from specific LLMs, such as GPT-2, OPT, LLaMA, and others. |
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024.acl-long)
Copied to clipboard
Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks, but their application to information retrieval tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. |
| Approach: | They propose to use instruction tuning to enhance LLMs' proficiency in IR tasks by combining a dataset with manually written templates to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions. |
| Outcome: | The proposed model can be used to perform query understanding, document understanding, and query-document relationship understanding tasks. |
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training (2025.emnlp-industry)
Copied to clipboard
Maryam Dialameh, Rezaul Karim, Hossein Rajabzadeh, Omar Mohamed Awad, Boxing Chen, Hyock Ju Kwon, Walid Ahmed, Yang Liu
| Challenge: | ECHO-LLaMA transforms LLa MA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. |
| Approach: | They propose an efficient LLaMA architecture that transforms LLama models into shared KV caching across certain layers, reducing computational complexity while maintaining or improving language performance. |
| Outcome: | ECHO-LLaMA achieves up to 77% higher token-per-second throughput during training, up to 16% higher Model FLOPs Utilization (MFU) and up to 14% lower loss when trained on an equal number of tokens. |
GRAFT: A Graph-based Flow-aware Agentic Framework for Document-level Machine Translation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing Document-level machine translation systems struggle to handle discourse-level phenomena such as pronoun resolution, lexical cohesion, and ellipsis. |
| Approach: | They propose a graph-based document-level machine translation framework that leverages Large Language Models to model translation flow and discourse structure. |
| Outcome: | The proposed framework outperforms commercial and closed systems in eight languages and six domains. |
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaption (LoRA) is a low-level pruning method that can be expensive and slow to deploy. |
| Approach: | They propose a low-rank adaption pruning framework that provides an accurate structured pruned model in a memory-efficient manner. |
| Outcome: | The proposed pruning framework reduces perplexity and memory usage by 52.6% on LLaMA and T5 models while reducing memory usage. |
Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) |
| Approach: | They propose to use LLMs to probe their behavior using controlled experiments. |
| Outcome: | The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do. |
Evaluating AI for Finance: Is AI Credible at Assessing Investment Risk Appetite? (2025.emnlp-industry)
Copied to clipboard
Divij Chawla, Ashita Bhutada, Duc Anh Do, Abhinav Raghunathan, Vinod Sp, Cathy Guo, Dar Win Liew, Prannaya Gupta, Rishabh Bhardwaj, Rajat Bhardwaj, Soujanya Poria
| Challenge: | Our analysis was conducted on proprietary systems and open-weight models . FINRISKEVAL analyzed 1,720 profiles spanning a broad spectrum of possible risk categories . |
| Approach: | They evaluated proprietary AI systems and open-weight models to assess investment risk appetite using carefully curated user profiles. |
| Outcome: | The proposed models exhibit significant variance when user attributes that should not influence risk computation are changed. |
Metaphor Understanding Challenge Dataset for LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Metaphor understanding is an essential task for large language models (LLMs). |
| Approach: | They propose to evaluate the metaphor understanding capabilities of large language models (LLMs) the metaphor understanding challenge dataset provides over 10k paraphrases and 1.5k instances of inapt paraphrase. |
| Outcome: | The metaphor understanding challenge dataset evaluates the performance of large language models on a range of NLU tasks. |
Benchmarking Hallucination in Large Language Models Based on Unanswerable Math Word Problem (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are highly effective in various natural language processing tasks, but can produce unreliable conjectures in ambiguous contexts, which is known as hallucination. |
| Approach: | They propose a method to evaluate LLM hallucination in Question Answering based on the unanswerable math word problem (UMWP) . they combine text similarity and mathematical expression detection to determine whether LLM considers the question unanswered. |
| Outcome: | The proposed method combines text similarity and mathematical expression detection to determine whether the LLM considers the question unanswerable. |
COMMIT: Code-Mixing English-Centric Large Language Model for Multilingual Instruction Tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | COMMIT improves the exact match score of low-resource language QA by 32x. |
| Approach: | They propose to specialize instruction tuning to deviate from English-centric instruction tuning . they propose to perform cross-lingual alignment to overcome data imbalance . |
| Outcome: | The proposed method improves the exact match score of low-resource language QA by 32x. |
Watermarking LLMs with Weight Quantization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are being deployed at an astonishing speed, exposing users to high risks. |
| Approach: | They propose a method that plants watermarks in quantization process of large language models without pre-defined triggers during inference. |
| Outcome: | The proposed method protects model weights without pre-defined triggers . it works when the model is used in the fp32 mode and remains hidden when the models are quantized to int8 . |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs (2026.acl-long)
Copied to clipboard
Xuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Haoyu Wang, Yujun Yan, Haifeng Chen, Zhengzhang Chen
| Challenge: | Large language models (LLMs) produce outdated or inaccurate content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. |
| Approach: | They propose a robust and scalable method that treats knowledge control as interventions within the model’s representation space. |
| Outcome: | The proposed method achieves fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. |
Program-of-Thought Reveals LLM Abstraction Ceilings (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models exhibit reasoning ability when supervised with chain-of-thought (CoT) traces. |
| Approach: | They evaluate large language models with CoT traces and fine-tune them with Program-of-Thought supervision. |
| Outcome: | The proposed model performance degrades sharply under numeric perturbations under isomorphic variants. |
Large Language Models are Temporal and Causal Reasoners for Video Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. |
| Approach: | They propose a framework that exploits linguistic shortcuts and mitigates 'linguistic bias' by flipping the source pair and target label to understand their complex relationships. |
| Outcome: | The proposed framework outperforms both LLMs-based and non-LLMs- based models on five challenging VideoQA benchmarks. |
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for misinformation detection are limited by data scarcity . existing methods fail to detect early-stage misinformation on emerging topics . |
| Approach: | They propose a meta learning based approach for domain adaptive few-shot misinformation detection that leverages limited target examples to provide feedback and guide the knowledge transfer from the source to the target domain. |
| Outcome: | The proposed method improves performance on real-world datasets with reduced parameters. |
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |
LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing (2025.acl-long)
Copied to clipboard
Dario Di Palma, Alessandro De Bellis, Giovanni Servedio, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia
| Challenge: | Large Language Models (LLMs) have become central to NLP, demonstrating their ability to adapt to various tasks through prompting techniques. |
| Approach: | They probe the hidden layers of Large Language Models to identify where sentiment features are most represented and to assess how this affects sentiment analysis. |
| Outcome: | The proposed approach enables sentiment tasks to be performed with memory requirements reduced by an average of 57%. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
Multilingual Prompting for Improving LLM Generation Diversity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models lack cultural representation and diversity in their generations . lack of demographic diversity can lead to unfair lack of exposure of artists . |
| Approach: | They propose a prompting method which generates several variations of a base prompt with added cultural and linguistic cues from several cultures, generates responses, and then combines the results. |
| Outcome: | The proposed method outperforms existing diversity-enhancing techniques . it can generate multiple variations of a base prompt with cultural cues from multiple cultures . |
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are useful interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. |
| Approach: | They evaluate the effectiveness of up-scaling and augmenting the LM with a passage retrieval database to reduce inconsistency. |
| Outcome: | The proposed models reduce inconsistency but retrieval augmentation is more efficient. |
The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | cipher decoding, riddles, code execution embedded into model prompts bypass safety safeguards of large language models (LLMs) . |
| Approach: | They introduce a novel class of adversarial jailbreak adversarials on large language models, termed Task-in-Prompt (TIP) attacks. |
| Outcome: | The proposed techniques circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2, and consistently generate restricted content . |
The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that fine-tuning large language models improves their translations, but it is unclear what is the impact on desirable LLM behaviors that are not present in neural machine translation models. |
| Approach: | They perform an extensive translation evaluation on LLaMA and Falcon models with model size ranging from 7 billion up to 65 billion parameters. |
| Outcome: | The proposed model produces less literal translations after fine-tuning on parallel data. |
LLaMA Pro: Progressive LLaMA with Block Expansion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that pre-trained LLMs are limited in certain domains, such as programming, mathematics, biomedical, or finance. |
| Approach: | They propose a new post-pretraining method with an expansion of Transformer blocks to tune the expanded blocks using only new corpus, efficiently and effectively improving the model’s knowledge while mitigating forgetting. |
| Outcome: | The proposed model outperforms existing models in programming and math and its instruction-following counterpart LLaMA Pro-8.3B in general tasks, programming, and mathematics. |
Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can answer many questions correctly, but can also hallucinate and give wrong answers. |
| Approach: | They propose a question-answering benchmark for Wikidata that uses SPARQL to ground large language models. |
| Outcome: | The proposed method outperforms the state-of-the-art for QALD-7 by 3.6% in F1 score. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
EOP-LLM: Energy Oriented Pruning for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Inference energy consumption has grown rapidly in large language models (LLMs) but existing methods focus on reducing FLOPs or latency rather than modeling or enforcing end-to-end inference energy constraints. |
| Approach: | They propose an energy-oriented dynamic pruning framework that enables LLM inference under explicit per-sequence energy budgets. |
| Outcome: | EOP-LLM outperforms state-of-the-art dynamic pruning baselines while adhering to per-sequence energy constraints. |
Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent large language models have demonstrated improved processing of complex language, but detecting logical fallacies remains a challenge. |
| Approach: | They propose a prompt formulation approach for logical fallacy detection that integrates contextual information into input text and queries for validity within the argument’s context. |
| Outcome: | The proposed approach improves over state-of-the-art models by 0.57 in F1-scores and 0.45 in fine-tuned models. |
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the promising potential of chat models, they are only accessible through restricted APIs, creating barriers for new research and progress in the field. |
| Approach: | They propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself. |
| Outcome: | The proposed pipeline generates a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself, simulating both user and AI responses. |
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | NVFP4 supports fine-grained block isolation, 4-bit quantization errors and mixed-precision approaches . ARCQuant boosts NVFO4 performance via Augmented Residual Channels . |
| Approach: | They propose a framework that boosts NVFP4 performance via Augmented Residual Channels. |
| Outcome: | ARCQuant boosts NVFP4 performance via Augmented Residual Channels . the proposed framework achieves state-of-the-art accuracy comparable to full-precision baselines compared to FP16 . |
Temporal Alignment of Time Sensitive Facts with Activation Engineering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models are trained on diverse and conflicting knowledge spanning multiple domains and time periods. |
| Approach: | They propose a method for temporally aligning large language models to improve factual recall without training. |
| Outcome: | The proposed method improves factual recall without training. |
Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. |
| Approach: | They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens. |
| Outcome: | The proposed method outperforms existing temporal language models on two datasets and three tasks. |
TripleFact: Defending Data Contamination in the Evaluation of LLM-driven Fake News Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for fake news detection are based on static datasets and closed-world assumptions that are inadvertently memorized during pre-training. |
| Approach: | They propose a framework to mitigate BDC risk while prioritizing real-world applicability by integrating three components to assess robustness against human-crafted misinformation. |
| Outcome: | The proposed framework mitigates BDC risk while prioritizing real-world applicability. |
DavIR: Data Selection via Implicit Reward for Large Language Models (2025.acl-long)
Copied to clipboard
Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang
| Challenge: | 6% of Alpaca dataset selected with DavIR can steer both LLaMA and Gemma models to produce superior performance compared to the same models trained on the full 52K dataset. |
| Approach: | They propose a model-based data selection method for post-training Large Language Models . they generalize Reducible Holdout Loss to core-set selection problem of causal language modeling . |
| Outcome: | The proposed method can steer both LLaMA and Gemma models to superior performance compared to the same models trained on the full 52K dataset. |
Is Compound Aspect-Based Sentiment Analysis Addressed by LLMs? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) aims to predict aspect-based elements from text . large language models (LLMs) have impressive abilities in handling human instructions . |
| Approach: | They propose a framework to evaluate LLMs' ability to handle complex ABSA tasks . they use constrained prompts to automatically organize the returned predictions . |
| Outcome: | The proposed framework outperforms supervised methods in some cases, but it is still lacking in other areas. |
Do Language Models Care about Text Quality? Evaluating Web-Crawled Corpora across 11 Languages (2024.lrec-main)
Copied to clipboard
Rik van Noord, Taja Kuzman, Peter Rupnik, Nikola Ljubešić, Miquel Esplà-Gomis, Gema Ramírez-Sánchez, Antonio Toral
| Challenge: | Large, curated, web-crawled corpora play a vital role in training language models . however, relatively little attention has been given to the quality of these corporata . |
| Approach: | They compare four of the currently most relevant large, web-crawled corpora across eleven lower-resourced European languages to evaluate their quality. |
| Outcome: | The CC100 corpus achieves the highest scores on the tests in 11 lower-resourced European languages. |
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent multilingual models support limited number of human languages due to lack of training data for low resource languages. |
| Approach: | They propose a multilingual multilingual LLM that scales to 100 languages . they use a human feedback dataset and a data set to perform multilingual instruction tuning . |
| Outcome: | The proposed model outperforms its peers on five multilingual benchmarks. |
Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extending the maximum context lengths of language models are lacking a strong baseline for in-context few-shot classification and on more challenging Chain-of-Thought reasoning, such as HotpotQA, deteriorate question miscomprehension and false inference. |
| Approach: | They propose to harness window-wise attention and positional embedding techniques to extend the maximum context lengths of language models. |
| Outcome: | The proposed method is able to extend the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. |
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |
Dodo: Dynamic Contextual Compression for Decoder-only LMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to NLP are sparsifying attention patterns or approximating the attention computation with kernel methods. |
| Approach: | They propose a method for dynamic contextual compression for decoder-only LMs. |
| Outcome: | The proposed method reduces the cost of self-attention to a fraction of typical time and space. |
Do Large Language Models Know What They Don’t Know? (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have vast knowledge that allows them to excel in various NLP tasks. |
| Approach: | They propose an automated method to detect uncertainty in the responses of large language models and a dataset to measure their self-knowledge. |
| Outcome: | The proposed method detects uncertainty in the responses of large language models and provides a novel measure of their self-knowledge. |
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning (2024.acl-long)
Copied to clipboard
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, Chang Zhou
| Challenge: | In math reasoning with large language models, fine-tuning data augmentation by query evolution and diverse reasoning paths is empirically verified effective. |
| Approach: | They propose to fine-tune data augmentation by query evolution and diverse reasoning paths. |
| Outcome: | The proposed model achieves new state-of-the-art on GSM8K and MATH. |
It Is Not Only the Negative that Deserves Attention! Understanding, Generation & Evaluation of (Positive) Moderation (2025.naacl-long)
Copied to clipboard
| Challenge: | Moderation is essential for maintaining and improving the quality of online discussions. |
| Approach: | They annotate a dataset on 13 modes of discussion and use it to generate positive moderation. |
| Outcome: | The proposed model shows that professional moderation generates higher ratings than professional moderated moderation, but prefers professional moderate in pairwise comparison. |
Learning to Substitute Words with Model-based Score Ranking (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that the proposed approach outperforms both masked language models and large language models. |
| Approach: | They propose a model-based scoring approach to quantify sentence quality . they propose 'loss function' that optimizes alignment between model predictions and sentence scores . |
| Outcome: | The proposed approach outperforms masked language models and large language models in the quantitative analysis of word substitutions. |
IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages (2024.acl-long)
Copied to clipboard
| Challenge: | IndicGenBench is the largest benchmark for evaluating large language models on user-facing generation tasks across a diverse set of 29 Indic languages . |
| Approach: | They evaluate large language models on user-facing generation tasks across 29 languages . they use human curation to provide multi-way parallel evaluation data for many under-represented languages a github repository . |
| Outcome: | IndicGenBench is the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set of 29 Indic languages covering 13 scripts and 4 language families. |
Cross-Lingual Knowledge Editing in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Knowledge editing is a promising technique to adapt large language models to new knowledge without retraining from scratch. |
| Approach: | They propose to use a multilingual dataset to translate a large-scale cross-lingual synthetic dataset from English to Chinese and then to evaluate their performance in Chinese. |
| Outcome: | The proposed method can change model performance on several special cases without retraining from scratch. |
Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Foundation Models (FMs) have demonstrated success in a wide range of applications, but their optimization often requires access to sensitive data. |
| Approach: | They propose a framework that combines FMs and Federated Learning to enable privacy-preserving and collaborative learning across multiple end-users. |
| Outcome: | The proposed framework combines benefits of FMs and Federated Learning (FL) it enables privacy-preserving and collaborative learning across multiple end-users. |
LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit exceptional translation capabilities in high-resource language tasks, yet their effectiveness in low-resourced languages is suboptimal. |
| Approach: | They conduct extensive multilingual continual pre-training on the LLaMA series models and develop LLiMAX for translation support across more than 100 languages. |
| Outcome: | The proposed model achieves higher translation performance than existing open-source models and performs on-par with specialized translation model on the Flores-101 benchmark. |
MAIN: Mutual Alignment Is Necessary for instruction tuning (2025.emnlp-main)
Copied to clipboard
Fanyi Yang, Jianfeng Liu, Xin Zhang, Haoyu Liu, Xixin Cao, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Instruction tuning has enabled large language models to achieve remarkable performance, yet its success heavily depends on the availability of high-quality instruction-response pairs. |
| Approach: | They propose a mutual alignment framework which enforces coherence between instructions and responses through mutual constraints. |
| Outcome: | The proposed framework generalizes well across model architectures and sizes, achieving state-of-the-art performance on LLaMA, Mistral, and Qwen models across diverse benchmarks. |
Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | LLaMA achieves highest scores in both multilingual and English-only evaluations . foundation models inherit inaccuracies from training corpora, which are propagated downstream to the models that are fine-tuned from them. |
| Approach: | They propose to produce a 20-language factual association dataset and evaluate 5 models in a multilingual test and benchmark a diverse set of 24 models in an English-only test. |
| Outcome: | The proposed model achieves highest scores in multilingual and English-only evaluations . however, it has significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. |
How Good Are LLMs at Out-of-Distribution Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is crucial for ensuring AI safety . large language models (LLMs) are becoming more prevalent due to their scale, pre-training objectives, and paradigms used for inference. |
| Approach: | They propose to use large language models to investigate out-of-distribution (OOD) detection in machine learning. |
| Outcome: | The proposed method outperforms other OOD detectors in zero-grad and fine-tuning scenarios. |
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. |
| Approach: | They propose to use several LLMs to ensemble translation hypotheses . they use instruction tuning, quality-based reranking, and minimum Bayes risk (MBR) decoding to improve translation quality. |
| Outcome: | The proposed method improves translation quality and instruction tuning improves the quality of the output. |
On the Correspondence between the Squared Norm and Information Content in Text Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence of the correspondence between the squared norm of an embedding and the information content of the text it represents is lacking. |
| Approach: | They propose to derive two sufficient theoretical conditions for this correspondence to hold in embedding models. |
| Outcome: | The proposed embeddings exhibit a strong correspondence with the word embeddables and the subword token composition functions. |
1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank approximation compresses the model by retaining its essential structure with minimal information loss. |
| Approach: | They propose a method that leverages the strengths of pruning and low-rank approximation for LLMs. |
| Outcome: | The proposed methods surpass the existing methods on LLaMA and Qwen2.5 models. |
Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies (2026.findings-acl)
Copied to clipboard
Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Jun Gao, Qian Qiao, Shichao Weng, Weizhong Zhang, Cheng Jin
| Challenge: | Existing quantization-aware fine-tuning methods decouple weight precision and adapter capacity, overlooking that a layer’s ability to adapt is constrained by the information preserved in its frozen weights. |
| Approach: | They propose a framework that jointly optimizes per-layer quantization bit-width and LoRA rank. |
| Outcome: | Experiments on LLaMA and Qwen models show that the proposed framework matches or approaches 16-bit baselines while using substantially less memory. |
Lightweight Haar Wavelet Subband Pruning for LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive performance but require computational and memory resources. |
| Approach: | They propose a post-training framework that uses a Haar wavelet transform to prune weights. |
| Outcome: | The proposed pruning framework reduces pruning time and computational costs by removing less important weights while preserving model architecture. |
Token Prediction as Implicit Classification to Identify LLM-Generated Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach for identifying large language models (LLMs) involved in text generation is proposed . instead of adding an additional classification layer, we reframe the classification task as a next-token prediction task . |
| Approach: | They propose a novel approach for identifying large language models involved in text generation . instead of adding an additional classification layer, they reframe the task as a next-token prediction task . |
| Outcome: | The proposed method performs exceptionally well in the text classification task . it can distinguish distinctive writing styles among various LLMs even without an explicit classifier. |
Neutral Is Not Unbiased: Evaluating Implicit and Intersectional Identity Bias in LLMs Through Structured Narrative Scenarios (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models often reproduce societal biases, yet most evaluations overlook how such biase evolve across nuanced contexts or intersecting identities. |
| Approach: | They propose a scenario-based evaluation framework built on 100 narrative tasks . they use critical discourse analysis and quantitative linguistic metrics to analyze LLMs . |
| Outcome: | The proposed evaluation framework provides ethically coherent and socially plausible settings for probing model behavior. |
ULTRABENCH: Benchmarking LLMs under Extreme Fine-grained Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks evaluate models on only a few attributes, typically fewer than five . a new benchmark evaluates large language models under dense, multi-attribute constraints . |
| Approach: | They propose a benchmark that evaluates large language models under dense, multi-attribute constraints. |
| Outcome: | The proposed benchmark evaluates large language models under dense, multi-attribute constraints. |
FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate original text using pre-trained language models are problematic as they are trained on corpora constructed by human authors. |
| Approach: | They propose a unique “self-plagiarism” contrastive decoding strategy that modifies prompts in LLMs to develop an amateur model and a professional model. |
| Outcome: | The proposed method enables the development of an amateur model and a professional model while maintaining its standard language model status. |
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown strong abilities to understand natural language, but how these factors affect the models’ language perception is unclear. |
| Approach: | They compare the self-attention of several existing large language models in different sizes to assess the effect of scaling and instruction tuning on language perception. |
| Outcome: | The proposed models are closer to non-native speakers than native speakers in attention, suggesting a sub-optimal language perception of all models. |
Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages (2025.acl-long)
Copied to clipboard
| Challenge: | XLM-R and mBART have advanced multilingualism in NLP, but low-resource languages such as Tibetan, Uyghur, Kazakh, and Mongolian are underserved. |
| Approach: | They propose a framework for adapting multilingual encoders to text generation in extremely low-resource languages by reusing the weights between the encoder and the decoder. |
| Outcome: | The proposed framework performs better on various downstream tasks even when compared with much larger models. |
RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models (2025.acl-long)
Copied to clipboard
Hieu Tran, Zonghai Yao, Zhichao Yang, Junda Wang, Yifan Zhang, Shuo Han, null Feiyun Ouyang, Hong Yu
| Challenge: | Existing work aims to improve reasoning accuracy and factual integrity across large language models for knowledge-intensive tasks such as medical and commonsense reasoning. |
| Approach: | They propose a versatile extension to the mutual reasoning framework (rStar) that enhances reasoning accuracy and factual integrity across large language models. |
| Outcome: | The proposed extension to the mutual reasoning framework improves reasoning accuracy and factual integrity across large language models for complex, knowledge-intensive tasks. |
Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proficient in natural language processing tasks, but their deployment is limited by extensive parameter sizes and computational demands. |
| Approach: | They propose a method to enhance computational efficiency in large language models by 4-bit weight and 8-bit activation quantization. |
| Outcome: | The proposed techniques significantly boost task accuracies to levels comparable with full-precision models. |
Scaling Laws for Linear Complexity Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws for large language models are unclear, but they are useful for scalability. |
| Approach: | They propose scaling laws for linear complexity language models to establish a foundation for their scalability. |
| Outcome: | The proposed models demonstrate superior linguistic proficiency and knowledge retention. |
A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge. |
| Approach: | They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs . |
| Outcome: | The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets. |
Choosing a Model, Shaping a Future: Comparing LLM Perspectives on Sustainability and its Relationship with AI (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Claude, DeepSeek, GPT, LLaMA, and Mistral conceptualize sustainability and its relationship with AI. |
| Approach: | They investigated how five state-of-the-art LLMs conceptualize sustainability and its relationship with AI. |
| Outcome: | The proposed models differed significantly between models, revealing significant biases and biase . Claude, DeepSeek, GPT, LLaMA, and Mistral conceptualize sustainability and its relationship with AI. |
Text-to-TrajVis: Enabling Trajectory Data Visualizations from Natural Language Questions (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for this task are limited and there is no suitable one available. |
| Approach: | They propose a new visualization language called Trajectory Visualization Language (TVL) to facilitate querying trajectory data and generating visualizations. |
| Outcome: | The proposed language can be used to query and generate trajectory data and generate visualizations with large language models. |
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs (2025.emnlp-main)
Copied to clipboard
Minxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong, Zijia Lin, Hui Chen, Wei Zhou, Jungong Han, Guiguang Ding, Wenwu Ou, Di Zhang, Kun Gai, Songlin Hu
| Challenge: | Existing sparsification methods like pruning can lose model knowledge through parameter removal. |
| Approach: | They propose a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. |
| Outcome: | The proposed approach achieves superior performance across language modeling and downstream tasks under equivalent computational constraints. |
Whether LLMs Know If They Know: Identifying Knowledge Boundaries via Debiased Historical In-Context Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for active retrieval (AR) rely on training classification models or using the confidence of the model’s answer to determine knowledge boundaries. |
| Approach: | They propose a method to identify knowledge boundaries in active retrieval by retrieving historical queries as high-confidence in-context examples. |
| Outcome: | Experiments on four QA benchmarks show that DH-ICL achieves performance comparable to full retrieval on LLaMA with only half the number of retrievals, without any additional training. |
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback (2023.findings-emnlp)
Copied to clipboard
Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, Zhaopeng Tu
| Challenge: | Large language models (LLMs) like ChatGPT are only accessible through restricted APIs, which creates barriers to new research and advancements in the field. |
| Approach: | They propose a framework to enhance and regulate the translation abilities during chat . they reformulate translation data into the instruction-following style and introduce a "Hint" field . |
| Outcome: | The proposed framework enhances and regulates the translation abilities during chat . it reformulates translation data into the instruction-following style and introduces a "Hint" field . |
Tuna: Instruction Tuning using Feedback from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | LLms like LLaMA have shown to be cost-effective for generating better responses . however, the instruction-tuned model has only seen one response per instruction . |
| Approach: | They propose to fine tune an instruction-tuned LLM using probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. |
| Outcome: | The proposed model improves on Super Natural Instructions, LMentry and Vicuna QA. |
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook (2026.acl-long)
Copied to clipboard
Hao Gu, Lujun Li, Hao Wang, Lei Wang, Zheyu Wang, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo
| Challenge: | Recent sparsity-aware binarization approaches can achieve sub-1-bit compression, but they face performance degradation, mask-management overhead, and limited hardware compatibility. |
| Approach: | They propose a binary quantization framework that leverages binary pattern clustering and weight transformation to overcome performance degradation and mask-management overhead. |
| Outcome: | The proposed framework achieves state-of-the-art compression (1.11–0.7 bits) it maintains high performance with only a 3.1% accuracy drop in zero-shot benchmarks while delivering a 1.6 speedup over FP16. |
No Need for Large-Scale Search: Exploring Large Language Models in Complex Knowledge Base Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Knowledge Base Question Answering (KBQA) systems are a key research area in the field of natural language processing and information retrieval (IR). |
| Approach: | They propose to use large language models to convert natural language questions to structured knowledge representations by using a three-step fine-tune strategy to implement the KBQA system. |
| Outcome: | The proposed method achieves state-of-the-art performance across three datasets with a 79.9% F1 score. |
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages (2025.acl-long)
Copied to clipboard
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, Duygu Ataman
| Challenge: | preparing native language MMLU benchmarks is costly and limits representativeness of evaluation datasets. |
| Approach: | They propose to use a Turkic language MMLU benchmark to assess massive multitask language understanding capabilities. |
| Outcome: | The proposed benchmarks are based on a Turkic language morphosyntactic and cultural benchmark . the benchmarks evaluate a diverse range of open and proprietary multilingual large language models . |
Rethinking Table Instruction Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have overlooked the impact of hyperparameters on table understanding abilities . authors show that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. |
| Approach: | They propose a hyperparameter-based instruction-tuned model for table-related tasks that improves out-of-domain table understanding ability and general capabilities. |
| Outcome: | The proposed model outperforms existing models on table-related tasks while maintaining strong out-of-domain generalization and general capabilities. |
Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning (2025.acl-long)
Copied to clipboard
Chenxi Huang, Shaotian Yan, Liang Xie, Binbin Lin, Sinan Fan, Yue Xin, Deng Cai, Chen Shen, Jieping Ye
| Challenge: | Representation Fine-tuning (ReFT) is a proposed method for improving parameter efficiency . however, it yields suboptimal performance, as fixed-position representations have uncertain impact on outputs . |
| Approach: | They propose a method that fine-tunes critical representations in a low-rank linear subspace while freezing the base model. |
| Outcome: | The proposed method improves accuracy of LLaMA-2-7B and ReFT by 18.2 and 3.8 on GSM8K. |
USDC: A Dataset of ̲User ̲Stance and ̲Dogmatism in Long ̲Conversations (2025.findings-acl)
Copied to clipboard
| Challenge: | Previously, studies on stance and dogmatism in user conversations have focused on training models using annotated datasets at the post level, treating each post as independent and randomly sampling posts from conversation threads. |
| Approach: | They build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC. |
| Outcome: | The proposed dataset analyzes user opinion fluctuations in 764 long multi-user Reddit conversation threads. |
Towards Understanding the Robustness of Sparse Autoencoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to optimization-based jailbreak attacks that exploit internal gradient structure. |
| Approach: | They propose to integrate pretrained Sparse Autoencoders into transformer residual streams at inference time without modifying model weights or blocking gradients. |
| Outcome: | The proposed model reduces jailbreak success rate by 5x compared to baseline models . compared with models with weak white-box attacks, the proposed model is more robust . |
Task-agnostic Distillation of Encoder-Decoder Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing distillation methods that focus on encoder-only LMs fail to handle the distillation of encoder decoder LM. |
| Approach: | They propose a method that finetunes pretrained language models (LMs) they propose 'MiniEnD' that allows for task-agnostic distillation of LMs. |
| Outcome: | The proposed distillation method is generally effective and competitive compared to other alternatives. |
GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. |
| Approach: | They propose a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. |
| Outcome: | Experiments show that GeLoRA outperforms adaptive LoRA methods by up to +1.0% . |
Extracting Linguistic Information from Large Language Models: Syntactic Relations and Derivational Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we study their morphosyntactic competence and generalization capabilities. |
| Approach: | They propose to use morphosyntactic tasks to study their linguistic knowledge and generalization capabilities to extract different types of morphological structure for typologically diverse languages. |
| Outcome: | The proposed models outperform GPT-4o and LLaMA 3.3-70B in all diagnostic tasks, but show little evidence of abstract morphological rule learning. |
Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models (2026.acl-long)
Copied to clipboard
Chao Xue, Yao Wang, Mengqiao Liu, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Chenyao Lu, Lei Jiang, Yu Lu, Haibo Shi, Shuang Liang, Minlong Peng, Flora D. Salim
| Challenge: | Incomplete learning is widespread and heterogeneous in large language models . authors identify five recurrent sources of incomplete learning: missing prerequisite knowledge, conflicts between SFT supervision and pre-training knowledge, internal inconsistencies within SFT data, left-side forgetting during sequential fine-tuning, and insufficient optimization for rare or complex patterns. |
| Approach: | They propose a diagnostic-first framework that maps incomplete learning to causes . they identify five recurrent sources of incomplete learning: missing prerequisite knowledge, conflicts between supervision and pre-training knowledge, internal inconsistencies, left-side forgetting during sequential fine-tuning, and insufficient optimization for rare or complex patterns. |
| Outcome: | The proposed framework maps incomplete learning to causes using observable training and inference signals. |
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing alignment benchmarks focus on sentence embeddings, but prior research has shown that neural models tend to induce a non-smooth representation space, which impact of semantic alignment evaluation on low-resource languages. |
| Approach: | They propose a novel cross-lingual alignment evaluation method based on the consistency of parallel sentences to assess model alignment. |
| Outcome: | The proposed method achieves a correlation of 0.9556 with downstream tasks performance and 0.8524 with transferability even with a small dataset. |
Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient (2025.acl-long)
Copied to clipboard
| Challenge: | Recent pruning methods rely on heuristically hand-crafted metrics, leading to suboptimal performance. |
| Approach: | They propose a method that optimizes pruning masks by minimizing back-propagation . they learn an underlying Bernoulli distribution to sample binary pruning mask samples . |
| Outcome: | The proposed method is able to support global and heterogeneous pruning without back-propagation. |
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has identified MLP modules in early layers as key contributors to factual recall. |
| Approach: | They conduct a comprehensive evaluation of factual recall across several models . they find that Qwen-based models behave differently from previous patterns . |
| Outcome: | The results show that attention modules in the earliest layers contribute more to factual recall than MLP modules. |
Iterative Multilingual Spectral Attribute Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are unable to exploit this opportunity because they operate on individual languages. |
| Approach: | They propose to iterate multilingual spectral attribute error (IMSAE) to mitigate joint bias subspaces across multiple languages through iterative SVD-based truncation. |
| Outcome: | The proposed method outperforms monolingual and cross-lingual approaches while maintaining model utility. |
FISTAPruner: Layer-wise Post-training Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods require inefficient retraining for billion-scale LLMs or rely on heuristicically designed metrics to determine pruning masks, leading to performance degradation. |
| Approach: | They propose a convex optimization model that induces sparsity in large language models by leveraging FISTA. |
| Outcome: | The proposed method can remove 50% of model parameters while retaining 98.6% and 95.6% of the zero-shot performance. |
It’s Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems (2025.acl-long)
Copied to clipboard
| Challenge: | idioms are defined as words with a figurative meaning not deducible from their individual components. |
| Approach: | They compare idiom translation as compared to conventional news translation in two languages . they compare MT and SLT systems with MT, Large Language Models and cascaded alternatives . |
| Outcome: | The proposed systems show better handling of idioms than standard news translation systems. |
ModelCitizens: Representing Community Voices in Online Safety (2025.emnlp-main)
Copied to clipboard
Ashima Suvarna, Christina A Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel
| Challenge: | Existing toxic language detection models are trained on annotations that collapse diverse perspectives into a single ground truth. |
| Approach: | They propose to augment social media posts with conversational scenarios to reflect the impact of conversational context on toxicity. |
| Outcome: | The proposed model outperforms existing models on social media with conversational scenarios. |
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
PictoEduca: Building a Dataset for Spanish Text-to-Pictogram Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | PictoEduca is the first large-scale Spanish text-to-pictogram dataset for augmentative and alternative communication. |
| Approach: | They present PictoEduca, a large-scale Spanish text-to-pictogram dataset for augmentative and alternative communication. |
| Outcome: | The proposed dataset combines automatic annotation with targeted expert correction, supporting scalable and high-quality corpus construction. |
GSM-Noise: Exploring and Enhancing Large Language Models’ Reasoning under Noisy Inputs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models struggle when dealing with complex, ill-formed, or noisy inputs . open-source models are less robust, while closed-source ones are more robust . |
| Approach: | They propose to use GSM-Noise to refine inputs before engaging in in-depth analysis to improve LLM robustness under noisy conditions. |
| Outcome: | The proposed model can achieve consistent performance gains under noisy conditions with prompt engineering, supervised finetuning, and reinforcement learning. |
Learning to Control Summaries with Score Ranking (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in summarization focus on improving summary quality across multiple dimensions, but they overlook the challenge of controlling summary generation with respect to individual dimensions. |
| Approach: | They propose a loss function that aligns model outputs with fine-grained, model-based evaluation scores to enable both improvement in summary quality and dimension-specific control. |
| Outcome: | The proposed method improves the overall quality of summaries while maintaining strong control over individual quality dimensions. |
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |
Fast and Accurate Fisher-Guided Quantization via Efficient Kronecker Factorization (2026.acl-long)
Copied to clipboard
| Challenge: | Quantization has shown strong results in preserving model quality under compression, but under aggressive bit-width reductions, even quantization may require additional information to prevent performance degradation. |
| Approach: | They propose a Kronecker-factored approximation that captures second-order curvature information, captured by the Hessian, to achieve a 10 speedup over prior approaches. |
| Outcome: | The proposed method significantly accelerates the most expensive component in second-order quantization – Hessian parameterization . it achieves up to a 10 speedup over prior approaches. |
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction (2026.findings-acl)
Copied to clipboard
| Challenge: | Muon’s orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. |
| Approach: | They propose to preserve Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tail weight spectra. |
| Outcome: | The proposed algorithm suppresses the emergence of heavy-tailed weight spectra and over-emphasizes training along noise-dominated directions. |
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models are slow and lack of performance. |
| Approach: | They propose a Zeroth-Order optimization framework that uses forward passes to fine-tune Large Language Models. |
| Outcome: | The proposed framework achieves 1.7 to 3.0 wall-clock acceleration on LLaMA and OPT models. |