Diagnosing LLMs via Information Spectrum Analysis: Tail Behavior and the Effects of Side Information (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models exhibit non-stationary generation because of variability in output distributions . authors propose a framework that treats LLMs as general sources without stationarity or ergodicity . |
| Approach: | They propose a diagnostic framework that treats large language models as general sources without stationarity, ergodicity, or the asymptotic equipartition property. |
| Outcome: | The proposed framework treats large language models as general sources without stationarity, ergodicity, or the asymptotic equipartition property. |
Similar Papers
Detecting LLM Hallucination Through Layer-wise Information Deficiency: Analysis of Ambiguous Prompts and Unanswerable Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate confident yet inaccurate responses, introducing significant risks for deployment in safety-critical domains. |
| Approach: | They propose a method to detect model hallucination by systematic analysis of information flow across model layers. |
| Outcome: | The proposed approach improves model reliability by immediately integrating with universal LLMs without additional training or architectural modifications. |
Bridging Internal Consistency and External Alignment: A Causal and Dynamic Interpretability Framework for LLM Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability methods focus on internal and external aspects of the model . existing explanations often focus on surface correlations or static dependencies . |
| Approach: | They propose a causal and dynamic interpretability framework for Large Language Models . they characterize backdoor-adjusted causal effects of generated prefix and prompt . |
| Outcome: | The proposed framework provides a unified causal view of internal consistency and external alignment in LLM generation dynamics. |
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |
Reliability of Distribution Predictions by LLMs: Insights from Counterintuitive Pseudo-Distributions (2025.naacl-srw)
Copied to clipboard
Toma Suzuki, Ayuki Katayama, Seiji Gobara, Ryo Tsujimoto, Hibiki Nakatani, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Recent studies highlight the use of Large Language Models (LLMs) for predicting response distributions as a cost-effective survey method. |
| Approach: | They examine whether LLMs can rationally estimate distributions when presented with explanations that are against commonsense. |
| Outcome: | The proposed models can rationally estimate distributions when presented with explanations that are against commonsense, but smaller or less human-optimized models follow explanations uncritically, compared to larger models that resist counterintuitive explanations by leveraging their pretraining-acquired knowledge. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |
In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search (2024.emnlp-main)
Copied to clipboard
Huihan Li, Yuting Ning, Zeyi Liao, Siyuan Wang, Xiang Li, Ximing Lu, Wenting Zhao, Faeze Brahman, Yejin Choi, Xiang Ren
| Challenge: | Logic-Induced-Knowledge-Search (LINK) is a framework for generating factually-correct yet long-tail inferential knowledge. |
| Approach: | They introduce a framework to obtain factually-correct yet long-tail inferential statements using variable-wise prompting grounded on symbolic rules. |
| Outcome: | The proposed framework is able to obtain factually-correct yet long-tail inferential statements while ensuring factual correctness. |
Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models lack knowledge of nuanced, domain-specific details and are susceptible to hallucinations. |
| Approach: | They construct a benchmark that measures head, torso, and tail facts in terms of popularity. |
| Outcome: | The proposed model is based on 18K question-answer pairs regarding head, torso, and tail facts in terms of popularity. |
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs (2025.emnlp-main)
Copied to clipboard
Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing detection methods fail to account for **self-consistent error** . study identifies self-consistency errors and evaluates them . |
| Approach: | They propose a method that fuses hidden state evidence from an external verifier LLM to detect self-consistent errors. |
| Outcome: | The proposed method significantly enhances performance on self-consistent errors across three LLM families. |
SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to inference with Large Language Models (LLMs) are expensive and time-consuming. |
| Approach: | They propose a framework for accelerating large language model inference without additional training or modification to the original LLM. |
| Outcome: | The proposed framework outperforms state-of-the-art methods and achieves 4.08x speedups across benchmarks and LLM architectures. |