How do decoding algorithms distribute information in dialogue responses? (2023.findings-eacl)
Copied to clipboard
| Challenge: | Using different decoding algorithms, we find that human dialogue generation is beneficial for adherence to the Uniform Information Density principle. |
| Approach: | They investigate whether decoding algorithms implicitly follow the Uniform Information Density principle by distributing information evenly in utterances. |
| Outcome: | The proposed method encourages non-uniform responses, but under low/high surprisal conditions, resulting in poor quality responses. |
Similar Papers
Revisiting the Uniform Information Density Hypothesis (2021.emnlp-main)
Copied to clipboard
| Challenge: | The uniform information density hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. |
| Approach: | They propose to test the hypothesis by using reading time and acceptability data to examine the effect of surprisal on language comprehension and acceptabilities. |
| Outcome: | The proposed hypothesis makes predictions about language comprehension and linguistic acceptability . |
Revisiting Entropy Rate Constancy in Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence supports the uniform information density hypothesis . however, we re-evaluate the hypothesis with neural language models . |
| Approach: | They propose to use n-gram language models to argue that English documents exhibit entropy rate constancy . they re-evaluate the claims of Genzel and Charniak with neural language models . |
| Outcome: | The proposed hypothesis fails to support the proposed hypothesis with language models. |
Is Information Density Uniform when Utterances are Grounded on Perception and Discourse? (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on the distribution of information in visually grounded contexts have focused on text-only inputs. |
| Approach: | They propose to use multilingual vision-and-language models to estimate surprisal . they find grounding on perception increases uniformity across typologically diverse languages . |
| Outcome: | The proposed hypothesis is tested in visual-language models over 30 languages and 13 storytelling languages . the results show grounding on perception increases uniformity across languages compared to text-only settings . |
Revisiting the Uniform Information Density Hypothesis in LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study has highlighted the fragility of Chain-of-Thought reasoning . a hypothesis suggests that effective communication is achieved by maintaining a stable flow of information. |
| Approach: | They propose a framework to quantify uniformity of information flow at local and global levels . they propose entropy-based stepwise density metric to quantify this phenomenon . |
| Outcome: | The proposed framework outperforms alternative signals as predictors of reasoning quality. |
A Cognitive Regularizer for Language Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | a uniform information density hypothesis is used to explain certain linguistic phenomena . a regularizer that encodes the UID hypothesis can be used for language training . |
| Approach: | They propose to augment the canonical MLE objective with a regularizer that encodes UID . they find that regularization consistently improves perplexity in language models . |
| Outcome: | The proposed hypothesis can be operationalized as an inductive bias for language modeling. |
Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse (2024.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
| Approach: | They propose a functional pressure that speakers modulate information rate based on location within a hierarchically-structured model of discourse. |
| Outcome: | The proposed hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
Is Information Density Uniform in Task-Oriented Dialogues? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evidence for the uniform information density principle has been found at many levels of language production. |
| Approach: | They propose to use the Uniform Information Density principle to test whether and within which contextual units it holds in task-oriented dialogues. |
| Outcome: | The proposed method is able to reduce fluctuations in the density of the information transmitted. |
GPT-who: An Information Density-based Machine-Generated Text Detector (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate misinformation, memorized content, plagiarized content, toxic speech, and hallucinated content. |
| Approach: | They propose a statistical detector that uses UID to model the unique statistical signature of each LLM and human author for accurate detection. |
| Outcome: | The proposed method outperforms state-of-the-art detectors by over 20% across domains. |
IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Traditional data generation methods are labor-intensive, resource-demanding, and raise privacy concerns. |
| Approach: | They propose an automatic synthetic data generation approach and introduce the **I**mplicit **P**ersonalized **Dialog**ue benchmark along with a training dataset, covering 10 tasks and 12 user attribute types. |
| Outcome: | The proposed approach incorporates the **Implicit **P**ersonalized **Dialog**ue benchmark along with a training dataset, covering 10 tasks and 12 user attribute types. |
Leveraging Implicit Feedback from Deployment Data in Dialogue (2024.eacl-short)
Copied to clipboard
| Challenge: | Xu et al., 2023) and Bai ed., 2019) use crowdworkers to collect signals from natural dialogue episodes. |
| Approach: | They use the publicly released BlenderBot deployment data to extract signals from conversations to implicitly measure the quality of a machine-generated utterance. |
| Outcome: | The proposed model improves over baseline models, but some proxy signals can lead to undesirable generations. |