Papers with Self-Consistency
FaithfulPersona: Balancing Faithfulness and Personalization in Code Explanations through Self-Critique (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for generating faithful code explanations face challenges balancing faithfulness to the original code and personalization for diverse user needs. |
| Approach: | They propose a benchmark and method for generating faithful personalized code explanations using code samples and user profiles. |
| Outcome: | The proposed method achieves 3.7% improvement in Pass@5 compared to the strong baseline method, Self-Consistency, while maintaining high personalization with a 61.08% win rate in the LLM-as-a-Judge evaluation. |
The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for difficulty estimation rely on repeated response sampling, auxiliary models, or fine-tuning the target model itself. |
| Approach: | They propose a method that leverages only the hidden representations produced by large language models. |
| Outcome: | The proposed method outperforms baselines in difficulty estimation on textual and multimodal tasks and improves adaptive reasoning strategies with fewer generated tokens. |
Mirror-Consistency: Harnessing Inconsistency in Majority Voting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a widely-used decoding strategy that relies on the plurality voting rule, which focuses on the most frequent answer while overlooking all other minority responses. |
| Approach: | They propose to incorporate a ‘reflective mirror’ into the self-ensemble decoding process and enables LLMs to critically examine inconsistencies among multiple generations. |
| Outcome: | The proposed method incorporates a ‘reflective mirror’ into the self-ensemble decoding process and enables LLMs to critically examine inconsistencies among multiple generations. |
NeuroSym-Cal: Bridging the Reasoning-Execution Gap in Code Generation via Hierarchical Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing calibration methods rely on the assumption that consensus implies correctness . Existing methods fail under systematic errors, leading to miscalibrated high-confidence predictions. |
| Approach: | They propose a hierarchical calibration framework that measures confidence at two levels . they propose sensitivity analysis to measure local curvature of deductive process . |
| Outcome: | The proposed framework de-saturates overconfident errors and improves selective generation performance on OOD benchmarks. |
DPC: Training-Free Text-to-SQL Candidate Selection via Dual-Paradigm Consistency (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating SQL queries lack the ability to self-evaluate correctness without an execution oracle. |
| Approach: | They propose a framework that reformulates SQL selection from a probabilistic guessing task on hidden data into a deterministic verification task on visible data. |
| Outcome: | Experiments on BIRD and Spider show that the proposed method outperforms baselines. |
Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index (2025.coling-main)
Copied to clipboard
| Challenge: | Prompt engineering is a growing subdiscipline of natural language processing . a lack of appropriate consideration for the financial constraints of computationally burdensome methods can limit their adoption and impact. |
| Approach: | They propose a new metric that combines accuracy scores with token consumption to reflect different resource constraints. |
| Outcome: | The economic prompting index (EPI) measures the performance of 6 prompting techniques across 10 widely-used language models and 4 diverse datasets. |
Evaluating the Consistency of LLM Evaluators (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown potential as general evaluators with the benefits of speed and cost. |
| Approach: | They conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency on different scoring scales and criterion granularity with open-source and proprietary models. |
| Outcome: | The results show that strong proprietary models are not necessarily consistent evaluators, highlighting the importance of considering consistency in assessing the capability of LLM evalueators. |
Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Self-consistency improves reasoning reliability but incurs substantial inference cost . Adaptive self-consistent methods rely on count-based stopping rules that treat all responses equally . |
| Approach: | They propose a method that reframs adaptive sampling from response counting to evidence sufficiency by leveraging response-level confidence. |
| Outcome: | The proposed method reduces inference cost by up to 70% while preserving accuracy on GSM8K. |
Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that Test-Time Scaling (TTS) can improve reasoning performance without retraining the model. |
| Approach: | They propose a step-wise pruning strategy that identifies and removes redundant chains using inter-chain similarity at the thought level. |
| Outcome: | The proposed method reduces inference latency and KVC usage by up to 45% and 26% with R1-Distill while maintaining or improving accuracy. |