Investigating Value-Reasoning Reliability in Small Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | sLLMs have been widely deployed in practical applications, but little attention has been paid to their value-reasoning abilities, particularly in terms of reasoning reliability. |
| Approach: | They propose a systematic evaluation framework for assessing the Value-Reasoning Reliability of small Large Language models (sLLMs) . framework includes three core tasks: Repetition Consistency task, Interaction Stability task, and Open-ended Expression Consistencies task. |
| Outcome: | The proposed framework incorporates self-reported confidence scores to evaluate the model’s value reasoning reliability from two perspectives: the model's self awareness of its values, and its value-based decision-making. |
Similar Papers
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MLLMs have achieved significant breakthroughs in understanding across text and vision, but current models still face inconsistencies in reasoning outcomes. |
| Approach: | They propose to evaluate multimodal large language models using a multimodal knowledge reasoning dataset to examine the extent of consistency degradation. |
| Outcome: | The proposed evaluation tasks show that MLLMs are inefficient at integrating knowledge across modalities . |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |
SaGE: Evaluating Moral Consistency in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on Large Language Models (LLMs) have focused on accuracy but lack universally agreed-upon answers for moral scenarios. |
| Approach: | They propose a measure called Semantic Graph Entropy to measure a model's moral consistency grounded in "Rules of Thumb" they construct a moral Consistency Corpus (MCC) with 50K moral questions and the RoTs they followed to investigate LLM consistency on two popular datasets. |
| Outcome: | The proposed measure measures moral consistency on two popular datasets . |
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work in language modeling has led to effective SLMs with impressive performance levels across various benchmarks. |
| Approach: | They propose a benchmark that introduces process-level evaluation for commonsense reasoning tasks. |
| Outcome: | The proposed benchmarks show that large language models provide correct answers despite flawed reasoning processes in a substantial portion of cases. |
ThinkSLM: Towards Reasoning in Small Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning has long been viewed as an emergent property of large language models (LLMs), appearing at or above a certain scale (100B parameters). |
| Approach: | They propose a benchmark to evaluate the reasoning abilities of small language models (SLMs) using quantization, pruning, and distillation. |
| Outcome: | The proposed benchmark systematically evaluates reasoning abilities of small language models trained from scratch or derived from LLMs using quantization, pruning, and distillation. |
Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps (2025.findings-emnlp)
Copied to clipboard
| Challenge: | despite their extensive context window, long-context language models fail in some basic cases . a recent study shows that long-cot methods are not necessary for long-constituency tasks . |
| Approach: | a new study evaluates long-context language models with a large context window . the authors propose a method that can be well addressed with arbitrary reasoning steps . |
| Outcome: | The proposed methods are well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. |
Are Large Language Models Consistent over Value-laden Questions? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) appear to bias survey answers toward certain values . however, some argue that LLMs are inconsistent to simulate particular values - a recent study . |
| Approach: | They define value consistency as similarity of answers across paraphrases, related questions and multilingual translations of a question to English, Chinese, German, and Japanese. |
| Outcome: | The proposed model is consistent across paraphrases, use-cases, translations, and within a topic. |
Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process (2026.acl-long)
Copied to clipboard
Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano, Hiroya Takamura
| Challenge: | Large language models have been shown remarkable ability to understand given contexts. |
| Approach: | They propose a method to evaluate whether beliefs held by LLMs remain consistent . they propose to use multiple choice question answering format to assess belief consistency . |
| Outcome: | The proposed method evaluates the consistency of LLMs in a multiple-choice question answering format. |
Safety in Large Reasoning Models: A Survey (2025.findings-emnlp)
Copied to clipboard
Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, Jiaheng Zhang, Bryan Hooi
| Challenge: | Large Reasoning Models (LRMs) have a high level of advanced reasoning capabilities, but they are vulnerable and vulnerable. |
| Approach: | This paper presents the first comprehensive survey of Large Reasoning Models . it explores the new safety risks, attacks, and defense strategies specific to LRMs based on reasoning . |
| Outcome: | The proposed study examines the safety and security risks of large reasoning models. |