RCScore: Quantifying Response Consistency in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations of large language models rely on a single instruction template, overlooking models’ sensitivity to instruction style. |
| Approach: | They propose a multi-dimensional framework quantifying how instruction formulation affects model responses by transforming benchmark problems into multiple instruction styles. |
| Outcome: | The proposed framework reveals that instruction style can shift accuracy by 16.7% points. |
Similar Papers
What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have significantly improved productivity in a number of routine tasks. |
| Approach: | They propose two metrics for classification tasks, namely *sensitivity* and *consistency*, which are complementary to task performance. |
| Outcome: | The proposed metrics are complementary to task performance and can be used to guide prompt engineering and obtain LLMs that balance robustness and performance. |
SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that manually ensuring a consistent response style and maintaining high data quality can significantly improve the performance of fine-tuned Large Language Models (LLMs). |
| Approach: | They introduce a style-aware response ranking system that prioritizes instruction-response pairs based on their stylistic consistency. |
| Outcome: | The proposed model matches or surpasses models trained on the entire dataset in coding and open-ended question-answering benchmarks. |
Revisiting the Reliability of Language Models in Instruction-Following (2026.acl-long)
Copied to clipboard
| Challenge: | Several benchmarks have been proposed to measure instruction-following accuracy, but these scores do not translate to reliable services in real-world use. |
| Approach: | They propose a new metric reliable@k and develop an automated pipeline to generate cousin prompts. |
| Outcome: | The proposed model can be instantiated with cousin prompts and generates high-quality cousin prompt data. |
MAIN: Mutual Alignment Is Necessary for instruction tuning (2025.emnlp-main)
Copied to clipboard
Fanyi Yang, Jianfeng Liu, Xin Zhang, Haoyu Liu, Xixin Cao, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Instruction tuning has enabled large language models to achieve remarkable performance, yet its success heavily depends on the availability of high-quality instruction-response pairs. |
| Approach: | They propose a mutual alignment framework which enforces coherence between instructions and responses through mutual constraints. |
| Outcome: | The proposed framework generalizes well across model architectures and sizes, achieving state-of-the-art performance on LLaMA, Mistral, and Qwen models across diverse benchmarks. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on instruction-tuned Large Language Models has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. |
| Approach: | They perform a meta-evaluation of automatic methods and assess their reliability across a broad range of tasks. |
| Outcome: | The proposed method is unreliable in free-form generation tasks and cross-lingual scenarios. |
When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following (2025.findings-emnlp)
Copied to clipboard
Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Edison Marrese-Taylor, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
| Challenge: | a large number of languages are increasingly used to evaluate their ability to follow multiple instructions simultaneously. |
| Approach: | They propose two benchmarks to evaluate LLMs' ability to follow multiple instructions simultaneously . they use many instruction-following eval and style-aware Mostly Basic programming problems . |
| Outcome: | The proposed models predict performance on unseen instruction combinations and not used during training with 10% error. |
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. |
| Approach: | They propose a framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. |
| Outcome: | The proposed framework integrates response agreement and focal loss with binary cross-entropy to improve calibration from baselines. |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent development of code-LLMs has demonstrated remarkable performance across various software engineering applications. |
| Approach: | They propose a round-trip code execution reasoning task to test round- trip consistency . they use zero-shot prompting, supervised fine-tuning on execution traces and self-reflection mechanisms to evaluate models . |
| Outcome: | The proposed benchmarks show that LLMs struggle with round-trip consistency . the benchmarks lack the internal coherence required for trustworthy code reasoning . |