Papers with entropy
Generating Text from Language Models (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models. |
| Approach: | They will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
| Outcome: | This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
Part Represents Whole: Improving the Evaluation of Machine Translation System Using Entropy Enhanced Metrics (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing machine translation metrics have poor correlations with human assessments . entropy-based evaluations are often limited to a limited number of samples . |
| Approach: | They propose a fast and unsupervised approach to enhance machine translation metrics using entropy by introducing sentence-level difficulty. |
| Outcome: | The proposed method outperforms existing metrics on five sub-tracks in the WMT19 Metrics shared tasks. |
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting (2026.acl-short)
Copied to clipboard
| Challenge: | Excessive exploitation can cause the model to become overconfident in its suboptimal solutions, thereby limiting its capabilities to explore novel reasoning strategies. |
| Approach: | They propose a method that dynamically down-weights extreme token-level updates via a Gaussian kernel and reduces the instability caused by the trade-off. |
| Outcome: | The proposed method improves downstream performance across reasoning benchmarks and stabilizes entropy as training progresses. |
Multi-Scale Spectral Selection and Entropy-Guided Uncertainty Fusion for Multimodal Rumor Detection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal content detection fail to capture cross-modal semantic inconsistencies and ignore inherent noise in multimodal features. |
| Approach: | They propose a multimodal rumor detection method based on a frequency domain spectral selection method and entropy-guided uncertainty fusion method to capture cross-modal semantic inconsistencies. |
| Outcome: | The proposed method outperforms state-of-the-art methods in multimodal rumor detection . it shows stronger detection capability and robustness on multiple datasets . |
Systematic Generalization in Language Models Scales with Information Entropy (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for assessing compositional behavior are unclear on how to measure the difficulty of a systematic generalization problem. |
| Approach: | They propose a framework for measuring entropy in a sequence-to-sequence task and a method for measuring it. |
| Outcome: | The proposed framework scales with the entropy of the distribution of component parts in the training data. |
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate long one-sentence responses that are less effective because they overlook two crucial factors: intra-cluster similarity and inter-c cluster similarity. |
| Approach: | They propose a method that generalizes semantic entropy and uses token probabilities to quantify uncertainty in large language models. |
| Outcome: | The proposed method can be extended to white-box settings by incorporating token probabilities. |
Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token’s Nature (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods that use entropy as a discrete filter or post-hoc regulator are limited in their ability to optimize for reasoning tasks. |
| Approach: | They propose a token-aware algorithm that continuously adapts optimization dynamics based on token-level entropy throughout the entire training process. |
| Outcome: | Extensive experiments on mathematical reasoning, code, and logic tasks across multiple models demonstrate HAPO’s consistent superiority over DAPO. |
Reanalyzing the Most Probable Sentence Problem: A Case Study in Explicating the Role of Entropy in Algorithmic Complexity (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing descriptive complexity measures are ineffective at describing algorithms' behaviour, and can make an apparently tractable problem seem NP-complete. |
| Approach: | They propose to use statistical measures to give an updated analysis of the complexity of the NP-complete most probable sentence problem for pCFGs. |
| Outcome: | The proposed method can be applied to word sense disambiguation and inference tasks. |
The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions (L18-1)
Copied to clipboard
| Challenge: | a new corpus of task-oriented spontaneous dialogues is available for free . we study the structure of non task-orientated dialogues and how they evolve over time . |
| Approach: | They describe a corpus of 54 interactions between pairs of subjects interacting to find differences in two very similar scenes. |
| Outcome: | The proposed dataset contains 54 interactions between subjects to find differences in two very similar scenes. |
A Multitask Active Learning Framework for Natural Language Understanding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies have shown that relation information between intents and slots can improve the efficiency of active learning algorithms. |
| Approach: | They propose a multitask active learning framework that exploits relation information between sub-tasks provided by a joint model. |
| Outcome: | The proposed framework achieves competitive performance with less training data than baseline methods on all datasets. |
Generalizing Backpropagation for Gradient-Based Interpretability (2023.acl-long)
Copied to clipboard
| Challenge: | Several feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model’s output with respect to its inputs, but they reveal little about the inner workings of the model itself. |
| Approach: | They propose a generalized backpropagation algorithm that generalizes the gradient computation of a model to efficiently compute other interpretable statistics about the gradient graph of neural networks. |
| Outcome: | The proposed generalized algorithm can be used to compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. |
The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |
Uncertainty-Aware Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in a wide range of tasks, but generating factually accurate outputs remains a challenge. |
| Approach: | They propose a method that dynamically adjusts model contributions at each decoding step based on uncertainty. |
| Outcome: | The proposed method significantly improves factual accuracy and reliability over existing methods. |
HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for training reasoning-oriented large language models assume high-resource settings with abundant data. |
| Approach: | They propose a framework that integrates high-value general-domain data to promote more diverse exploration. |
| Outcome: | The proposed framework matches or surpasses RLVR trained with 32 target-domain samples using 32 target domain samples. |
ExpSeek: Self-Triggered Experience Seeking for Web Agents (2026.findings-acl)
Copied to clipboard
Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, Yongbin Li
| Challenge: | Existing methods for integrating experience into web agents are struggling to adapt to dynamically changing contextual observations during agent-environment interaction. |
| Approach: | They propose a model that shifts experience toward step-level proactive seeking by estimating step- level entropy thresholds and designing step-Level tailored experience content. |
| Outcome: | The proposed model achieves 9.3% and 7.5% performance improvements on Qwen3-8B and 32B models across four challenging web agent benchmarks. |
EDSD: Entropy-Driven Design for Faster Speculative Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding incur substantial training overhead to mitigate information misalignment between autoregressive draft model training and decoding. |
| Approach: | They propose an Entropy-Driven Speculative Decoding framework that uses entropy as a unified, interpretable signal for both draft model training and architectural design. |
| Outcome: | Experiments on seven large language models show that EDSD improves training efficiency by 24.8% and increases acceptance length by 4.0% compared to state-of-the-art methods. |