Probing LLM World Models: Enhancing Guesstimation with Wisdom of Crowds Decoding (2025.emnlp-main)
Copied to clipboard
Yun-Shiuan Chuang, Sameer Narendran, Nikunj Harlalka, Alexander Cheung, Sizhe Gao, Siddharth Suresh, Junjie Hu, Timothy T. Rogers
| Challenge: | a common real-world skill of guesstimation is underexplored in large language model research . a recent study suggests that LLMs encode a world model that supports approximate reasoning . |
| Approach: | They propose to decode a guesstimation dataset using MARBLES, FUTURE, and ELECPRED . they replicate WOC effects in human participants and find similar benefits . |
| Outcome: | The proposed model improves accuracy over greedy, self-consistency, and mean decoding in human participants. |
Similar Papers
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate fluent text with minimal task-specific supervision, but their ability to generate rationales for knowledge-intensive tasks (KITs) remains under-explored. |
| Approach: | They propose to generate retrieval-augmented rationalization of KIT model predictions via external knowledge guidance within a few-shot setting. |
| Outcome: | The proposed rationales were compared with crowd-sourced rationale models on factuality, sufficiency, and convincingness. |
Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on calibration and uncertainty quantification focuses on individual models, overlooking the potential of model diversity. |
| Approach: | They propose a method that uses Jensen-Shannon Divergence to identify and aggregate well-calibrated subsets of large language models (LLMs) to improve calibration. |
| Outcome: | The proposed method improves accuracy on binary prediction tasks compared to single-model and naive ensemble baselines. |
Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation (2025.coling-main)
Copied to clipboard
| Challenge: | LLMs are used to emulate sequential decision-making processes of humans . however, their ability to perform probabilistic sampling is limited . |
| Approach: | They propose to use large language models (LLMs) as agents to emulate the sequential decision-making processes of humans represented as Markov decision-makers (MDPs). |
| Outcome: | The proposed models can understand probabilities, but struggle with sampling precision . integrating coding tools can improve sampling precision, but this level of sampling precision still makes it difficult to simulate human behavior as agents. |
Decoupling Task-Solving and Output Formatting in LLM Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies suggest intertwining task and format instructions with strict formatting requirements can negatively impact LLMs' reasoning capabilities. |
| Approach: | They propose a decoding framework that explicitly decouples format adherence from problem solving. |
| Outcome: | Experiments show that Deco-G consistently gains over prompting and structured generation baselines, with guaranteed format compliance. |
Are Large Language Models (LLMs) Good Social Predictors? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can generate human-like responses, but it is unclear how well they work and where the plausible predictions derive from. |
| Approach: | They propose to use LLMs to generate human-like responses by mutability and accessibility of social inputs to perform a social prediction task. |
| Outcome: | The proposed model performs well in three realistic settings and a novel social prediction task. |
A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses. |
| Approach: | They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks . |
| Outcome: | The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem . |
Finetuning LLMs for Human Behavior Prediction in Social Science Experiments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to simulate social science experiments . finetuning LLMs directly on individual-level responses from past experiments improves accuracy . |
| Approach: | They propose to fine tune large language models directly on individual responses from past experiments to achieve multiple levels of generalization. |
| Outcome: | The proposed model outperforms GPT-4o in completely unseen studies by 36% . the proposed model reduces demographic parity difference by 10.6% compared to GPT-4) |
Beyond Inherent Cognition Biases in LLM-Based Event Forecasting: A Multi-Cognition Agentic Framework (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models exhibit human-like cognitive biases in event forecasting . a human-curated dataset reveals significant cognitive bias in LLMs . |
| Approach: | They propose a human-curated dataset to explore LLMs' cognitive biases . they leverage LLM participants to act as multi-cognition event participants . |
| Outcome: | The proposed framework alleviates cognitive biases in LLMs and offers diverse perspectives. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |