The Illusion of Randomness: How LLMs Fail to Emulate Stochastic Decision-Making in Rock-Paper-Scissors Games? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research indicates that large language models articulate the theoretical probability distributions associated with optimal strategic choices, but their actual decision-making diverges from these prescriptions. |
| Approach: | a systematic evaluation of 20 state-of-the-art LLMs reveals a cognitive bias gap . intrinsic biases inherited from pre-training corpora alone are insufficient to explain deviations . a semantic-free paradigm strips away intrinsic bias to isolate pure positional bias . |
| Outcome: | a systematic evaluation of 20 state-of-the-art LLMs shows that intrinsic biases are insufficient to explain deviations. |
Similar Papers
Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used for social science simulations . however, most evaluations focus on task optimality rather than variability and adaptation characteristic of human decision-making. |
| Approach: | They propose a process-oriented evaluation framework with progressive interventions to evaluate two economics tasks using large language models. |
| Outcome: | The proposed evaluation framework targets two economic tasks with progressive interventions. |
Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation (2025.coling-main)
Copied to clipboard
| Challenge: | LLMs are used to emulate sequential decision-making processes of humans . however, their ability to perform probabilistic sampling is limited . |
| Approach: | They propose to use large language models (LLMs) as agents to emulate the sequential decision-making processes of humans represented as Markov decision-makers (MDPs). |
| Outcome: | The proposed models can understand probabilities, but struggle with sampling precision . integrating coding tools can improve sampling precision, but this level of sampling precision still makes it difficult to simulate human behavior as agents. |
Systematic Biases in LLM Simulations of Debates (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies. |
| Approach: | They propose to use LLMs to simulate political debates on topics that are important aspects of people’s day-to-day lives and decision-making processes. |
| Outcome: | The proposed model can simulate political debates on topics that are important aspects of people’s day-to-day lives and decision-making processes. |
Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Using zero-shot or few-shot prompting, Large Language Models have been widely adopted in downstream applications. |
| Approach: | They propose to quantify the impact of option order and token usage on LLMs and propose mitigation strategies to enhance model performance. |
| Outcome: | The proposed mitigation strategies improve model performance and reduce the impact of token and order sensitivity on LLMs. |
Do You Get the Hint? Benchmarking LLMs on the Board Game Concept (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have achieved impressive progress on many benchmarks, yet they still have fundamental weaknesses. |
| Approach: | They introduce Concept, a word-guessing board game, as a benchmark for probing abductive reasoning. |
| Outcome: | The proposed game is easily solved by humans, but is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). |
Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized product recommenders, but their susceptibility to adversarial manipulations is difficult to detect. |
| Approach: | They propose to use large language models to investigate cognitive biases as adversarial strategies in product research using LLMs. |
| Outcome: | The proposed approach is the first to tap into human psychological principles, making such manipulations hard to detect. |
Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used in decision support systems and workflows . traditional computational paradigms for decision-making under uncertainty choose an option that maximizes expected utility or payoff . |
| Approach: | They compare large language models as decision support systems and agentic workflows . they find that LLMs cluster into reasoning models and conversational models . |
| Outcome: | The proposed models differ in their ability to perform tasks and their ability in a human-like way. |
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains (2026.findings-acl)
Copied to clipboard
Md. Faiyaz Abdullah Sayeedi, Subhey Sadi Rahman, Md. Mahbub Alam, Md. Adnanul Islam, Jannatul Ferdous Deepti, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Large Language Models (LLMs) have redefined Machine Translation, enabling context-aware and fluent translations across hundreds of languages and textual domains. |
| Approach: | They propose a framework and dataset to evaluate the translation quality and fairness of open-source LLMs. |
| Outcome: | The proposed framework and dataset evaluates translation quality and fairness of open-source LLMs. |
Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent surge in multilingual large language models (LLMs) and Retrieval Augmented Generation (RAG) has significantly expanded conversational search across varied linguistic and cultural demographics. |
| Approach: | They found that LLMs displayed systemic bias towards information in the same language as query language in document retrieval and answer generation. |
| Outcome: | The results highlight the linguistic divide within multilingual LLMs in information search systems. |
Decision Biases and Intent-Irony Decoupling in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive linguistic fluency, but it remains unclear whether they possess human-like Theory of Mind (ToM) or rely on statistical heuristics . a recent study examined the performance of LLMs against 300 human participants . |
| Approach: | a study establishes a framework for large language models that modulates contextual contrast, linguistic cues, and cognitive mechanisms. |
| Outcome: | a new evaluation framework compares ten state-of-the-art LLMs against 300 human participants . the framework systematically modulates contextual contrast, linguistic cues, and cognitive mechanisms . |